Information processing methods, information processing systems, terminal devices, and computer programs

Abstract

This invention provides an information processing method, information processing system, terminal device, and computer program that control the timing of sending work data to a server device according to its priority. [Solution] In the information processing system, the wearable device 10 acquires multiple work data related to the work performed by the worker, and the worker's mobile terminal 30 performs a process to control the timing of sending the acquired work data to the server device 20 according to the priority of the work data. The terminal device sends work data with relatively high priority immediately and sends work data with relatively low priority afterward.

Description

Technical Field

[0006] , , ,

[0005] ,

[0007] , ,

[0001] The present invention relates to an information processing method, an information processing system, a terminal device, and a computer program.

Background Art

[0002] Patent Document 1 discloses an inspection work guidance system including an audio information output unit that outputs inspection items in an inspection work as audio information, and an audio information input unit that inputs an inspection result from an inspector as audio information, and converts the audio information input through the audio information input unit into character information and transmits it to a server device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, although it is possible to transmit the work result to the server device as character information, it is not possible to control the transmission timing thereof.

[0005] An object of the present disclosure is to provide an information processing method, an information processing system, a terminal device, and a computer program capable of controlling the timing of transmission to a server device according to the priority of work data.

Means for Solving the Problems

[0006] An information processing method according to a first aspect of the present disclosure acquires a plurality of work data related to work performed by a worker, and controls the timing of transmitting the work data to a server device according to the priority of the acquired work data, and the processing is executed by the worker's terminal device.

[0007] The information processing method relating to the second aspect of this disclosure is an information processing method relating to the first aspect, wherein the terminal device immediately transmits work data with a relatively high priority and subsequently transmits work data with a relatively low priority.

[0008] The information processing method relating to the third aspect of this disclosure is an information processing method relating to the first or second aspect, wherein the work data includes work result data entered by the worker, imaging data obtained by imaging the work performed by the worker, acoustic data obtained by recording sounds at the work site, or equipment data obtained with respect to the equipment being worked on.

[0009] The information processing method relating to the fourth aspect of this disclosure is configured such that, in the information processing method relating to the third aspect, the priority of the work result data is set higher than the priority of the imaging data, the acoustic data, and the equipment data.

[0010] The information processing method relating to the fifth aspect of this disclosure, in which work recognition is performed immediately based on the imaging data in the information processing method relating to the third aspect, sets the priority of the imaging data higher than the priority of the work result data, the acoustic data, and the equipment data.

[0011] The information processing method relating to the sixth aspect of this disclosure, in which, in the information processing method relating to the third aspect, performs immediate acoustic recognition based on the acoustic data, sets the priority of the acoustic data higher than the priority of the work result data, the imaging data, and the equipment data.

[0012] The information processing method relating to the seventh aspect of this disclosure is an information processing method relating to any one of the second to sixth aspects, wherein the terminal device acquires work data for each of the multiple work procedures and sets the priority of the work data according to the work procedure.

[0013] The information processing method relating to the eighth aspect of this disclosure is an information processing method relating to any one of the third to seventh aspects, wherein the terminal device guides the worker through the work content for each work procedure, and triggers the guidance of the work content to start at least one of imaging the work at each work procedure and recording sounds at the work site.

[0014] The information processing method relating to the ninth aspect of this disclosure is an information processing method relating to any one of the third to eighth aspects, wherein the terminal device recognizes a work scene based on the image data and decides whether or not to transmit the image data according to the recognized work scene.

[0015] The information processing method relating to the tenth aspect of this disclosure is an information processing method relating to any one of the second to ninth aspects, wherein the terminal device detects the communication status and, according to the result of detecting the communication status, decides whether or not to transmit the work data with a relatively low priority.

[0016] The information processing method relating to the 11th aspect of this disclosure is an information processing method relating to any one of the 1st to 10th aspects, wherein the terminal device obtains information relating to the transmission status of the work data from the server device, and outputs the transmission status of the work data based on the obtained information.

[0017] The information processing system relating to the twelfth aspect of this disclosure comprises a worker's terminal device and a server device that is communicably connected to the terminal device, wherein the terminal device acquires a plurality of work data related to work performed by the worker and controls the timing of transmitting the work data to the server device according to the priority of the acquired work data.

[0018] The terminal device relating to the 13th aspect of this disclosure comprises at least one control unit, which acquires a plurality of work data related to work performed by an operator and controls the timing of transmitting the work data to a server device according to the priority of the acquired work data.

[0019] A computer program according to a 14th aspect of the present disclosure causes a computer to execute a process of acquiring a plurality of work data related to work performed by a worker and controlling the timing of transmitting the acquired work data to a server device according to the priority of the acquired work data.

Advantages of the Invention

[0020] According to the present disclosure, it is possible to control the timing of transmission to the server device according to the priority of the work data.

Brief Description of the Drawings

[0021] [Figure 1] It is an explanatory diagram for explaining an outline of a process executed by an information processing system according to Embodiment 1. [Figure 2] It is a block diagram showing an internal configuration of a wearable device. [Figure 3] It is a block diagram showing an internal configuration of a server device. [Figure 4] It is a conceptual diagram showing an example of a work procedure document database. [Figure 5] It is an explanatory diagram for explaining an example of voice guidance by a wearable device. [Figure 6] It is a flowchart showing a processing procedure of voice guidance executed by a wearable device according to Embodiment 1. [Figure 7] It is a flowchart showing a processing procedure of periodic upload executed by a wearable device according to Embodiment 1. [Figure 8] It is a conceptual diagram showing an example of a work history database. [Figure 9] It is a schematic diagram showing a confirmation screen of upload status. [Figure 10] It is an explanatory diagram for explaining an example of priority setting in Embodiment 2. [Figure 11] It is a flowchart showing a processing procedure of voice guidance executed by a wearable device according to Embodiment 2. [Figure 12]This flowchart shows the periodic upload processing procedure performed by the wearable device according to Embodiment 2. [Figure 13] This flowchart shows the procedure for processing performed by the wearable device according to Embodiment 3. [Modes for carrying out the invention]

[0022] The information processing system according to the embodiment will be described in detail below with reference to the drawings. (Embodiment 1) Figure 1 is an explanatory diagram illustrating the outline of the processing performed by the information processing system 1 according to Embodiment 1. The information processing system 1 according to Embodiment 1 comprises a wearable device 10 worn by a worker and a server device 20 that manages the work process. The wearable device 10 and the server device 20 are connected to each other so as to be able to communicate with each other via a communication network NW such as the Internet.

[0023] In this embodiment, the worker is, for example, a worker who performs installation, repair, and inspection work related to refrigeration and air conditioning equipment. Refrigeration and air conditioning equipment refers to equipment connected to refrigerant piping and constituting a refrigeration cycle. Refrigeration and air conditioning equipment includes equipment such as outdoor units, indoor units, ventilation systems, heat source equipment, and central control devices that constitute an air conditioning system, and equipment such as refrigerators, heat exchangers, and control devices that constitute a refrigeration system.

[0024] The wearable device 10 is a device having imaging, recording, audio output, and communication functions. The wearable device 10 is worn, for example, around the neck of a worker. Alternatively, the wearable device 10 may be worn on the worker's head, or on another body part such as the shoulder or arm. Furthermore, the wearable device 10 may be a goggle-type camera device, or any device with the above functions, such as a smartphone or action camera, may be used instead of the wearable device 10.

[0025] The wearable device 10 retrieves work procedure manuals from the server device 20 regarding the work to be performed by the worker. The work procedure manual is data that defines the work procedures for tasks such as installation, repair, and inspection, and consists of multiple work procedures. The work procedure manuals are assumed to be created in advance by the work manager or the like for each type of work and each type of equipment, and are registered in the work procedure manual database DB1 (see Figure 4). The wearable device 10 retrieves the necessary work procedure manuals from the server device 20 according to the type of work and the equipment being worked on for the work the worker intends to perform.

[0026] The wearable device 10 provides voice guidance according to the work procedure manual obtained from the server device 20. For example, the wearable device 10 can refer to the work procedure manual and provide voice instructions on what the worker should do. The wearable device 10 may also refer to the work procedure manual and provide voice inquiries about the work the worker has performed. The worker performs the work according to the voice guidance output from the wearable device 10, confirms the work performed, and responds to inquiries from the wearable device 10 by voice.

[0027] The wearable device 10 starts capturing images and recording audio in response to the output of voice guidance corresponding to each work procedure. Since the wearable device 10 is worn by the worker, the images obtained by the wearable device 10 are from the worker's perspective (i.e., first-person perspective). In this embodiment, the images obtained by the wearable device 10 are videos captured at an appropriate frame rate, but they may also be still images captured at appropriate time intervals. The images captured by the wearable device 10 include the work scene, the equipment being worked on, and various measuring instruments. The sounds recorded by the wearable device 10 include the worker's voice, sounds emitted from the equipment being worked on, ambient sounds, and any other sounds that occur at the work site.

[0028] The wearable device 10 terminates imaging and recording when a work procedure is completed. In this embodiment, imaging data and acoustic data are obtained for each work procedure. The imaging data and acoustic data for each work procedure are examples of work data in this disclosure.

[0029] The work data may include imaging data and acoustic data for each work procedure, as well as work result data entered by the worker. The work results are entered into the wearable device 10 as voice responses to inquiries from the wearable device 10. The work result data may be voice data or text data converted by speech recognition.

[0030] The work data may include imaging data and acoustic data for each work procedure, as well as equipment data obtained for the equipment being worked on. The equipment data includes measured values ​​such as current, voltage, electrical resistance, power, temperature, airflow, and wind speed measured for the equipment being worked on. The measured values ​​may be values ​​measured by measuring instruments or sensors installed in the equipment being worked on, or values ​​measured by an operator using measuring instruments or sensors. The equipment data may also include status data and error codes output from the equipment being worked on.

[0031] The wearable device 10 uploads the acquired work data to the server device 20 via the communication network NW. At this time, the wearable device 10 controls the timing of uploading the work data to the server device 20 according to the priority of the work data.

[0032] In Embodiment 1, the priority of work result data is set higher than the priority of imaging data, acoustic data, and equipment data. When work result data is input to the wearable device 10, it immediately uploads it to the server device 20, and when imaging data, acoustic data, and equipment data are acquired, it uploads them to the server device 20 retrospectively. For imaging data, acoustic data, and equipment data to be uploaded retrospectively, they should be uploaded to the server device 20 periodically when the radio wave intensity is above a threshold.

[0033] The server device 20 may determine whether the work performed by the worker is appropriate based on the work result data uploaded from the wearable device 10. If the server device 20 determines that the work performed by the worker is appropriate, it can give instructions to the wearable device 10 to prompt the worker to move on to the next work procedure. On the other hand, if the server device 20 determines that the work is not appropriate, it can give instructions to the wearable device 10 and have it output voice guidance prompting the worker to correct the work, thereby guiding the worker to perform the work appropriately.

[0034] In addition to the wearable device 10, the worker may also carry a mobile terminal 30 such as a smartphone. The mobile terminal 30 has an input unit such as a touch panel and a display unit such as an LCD display, and complements some of the functions that the wearable device 10 does not have. The mobile terminal 30 is configured to communicate with the wearable device 10 using short-range wireless communication such as Bluetooth® or ZigBee®. Furthermore, the mobile terminal 30 is configured to communicate with the server device 20 via a communication network NW.

[0035] Figure 2 is a block diagram showing the internal configuration of the wearable device 10. The wearable device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an imaging unit 14, an audio input unit 15, an audio output unit 16, a sensor unit 17, and the like.

[0036] The control unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and the like. The ROM in the control unit 11 stores control programs and the like that control the operation of each hardware component of the wearable device 10. The CPU in the control unit 11 reads and executes the control programs and the like stored in the ROM, and controls the operation of the hardware components, thereby making the entire device function as the terminal device of this disclosure. The RAM in the control unit 11 temporarily stores data used during the execution of various processes.

[0037] The control unit 11 may also include functions such as a clock for outputting date and time information, a timer for measuring the elapsed time from the time a measurement start instruction is given until a measurement end instruction is given, and a counter for counting numbers.

[0038] The storage unit 12 is equipped with an auxiliary storage device and stores various types of data, as well as computer programs executed by the control unit 11. The data stored in the storage unit 12 includes the above-mentioned imaging data, acoustic data, processing result data, and equipment data. The computer program (program product) stored in the storage unit 12 includes a transmission control program PG that causes the computer to execute a process to acquire multiple work data and control the timing of transmitting the acquired work data to the server device 20 according to the priority of the acquired work data.

[0039] A computer program including a transmission control program PG may be provided on a non-temporary recording medium RM on which the computer program is recorded in a readable format. The recording medium RM is a portable memory such as a CD-ROM, USB memory, SD card, microSD card, or CompactFlash®. The control unit 11 reads various computer programs from the recording medium RM using a reading device (not shown in the figure) and stores the read computer programs in the storage unit 12. The computer programs stored in the storage unit 12 may also be provided by communication. In this case, the control unit 11 acquires the computer programs by communication via the communication unit 13 and stores the acquired computer programs in the storage unit 12.

[0040] The transmission control program PG may be a single computer program or may consist of multiple computer programs. The transmission control program PG may be executed on a single computer or may be executed collaboratively by multiple computers (e.g., a wearable device 10 and a server device 20). Furthermore, the transmission control program PG may partially utilize existing libraries.

[0041] The communication unit 13 is equipped with a communication module for wireless communication with external devices such as the server device 20. The communication module uses a communication module for wireless communication using known mobile communication standards such as 3G, 4G, and 5G, or wireless LAN methods such as WiFi (registered trademark). The communication unit 13 communicates with external devices such as the server device 20 via the communication network NW, transmits necessary data such as work data, and receives appropriate data transmitted from the external devices.

[0042] The communication unit 13 may be equipped with a communication module for short-range wireless communication, such as Bluetooth® or ZigBee®, in order to communicate with a mobile terminal 30 such as a smartphone carried by the worker.

[0043] The imaging unit 14 includes an optical lens, an image sensor, a driver circuit, and the like. A wide-angle lens is preferably used as the optical lens. The image sensor is a CMOS (Complementary Metal Oxide Semiconductor), CCD (Charge-Coupled Device), etc., and generates electrical signals according to the intensity of light formed through the optical lens. The driver circuit includes a timing generator (TG), etc., and sequentially reads electrical signals from the image sensor in synchronization with the clock signal output from the TG to generate video data. The video data generated by the imaging unit 14 is sent to the control unit 11 and stored in the storage unit 12. The video data generated by the imaging unit 14 is also transmitted to the server device 20 via the communication unit 13.

[0044] The sound input unit 15 includes a microphone for collecting sound, a processing circuit for converting the collected sound into a digital signal (acoustic data), and the like. The acoustic data generated in the sound input unit 15 is sent to the control unit 11, where it undergoes appropriate processing such as noise reduction, and is then stored in the storage unit 12. The acoustic data processed by the control unit 11 is then transmitted to the server device 20 via the communication unit 13.

[0045] The sound output unit 16 is equipped with a speaker that outputs sound. The sound output unit 16 outputs sound based on acoustic data provided by the control unit 11.

[0046] The sensor unit 17 is equipped with non-contact sensors for detecting the worker's fingers and other body parts. The sensors in the sensor unit 17 include proximity sensors and gesture sensors. For example, the proximity sensor detects when the worker's fingers come within a predetermined range. For example, the gesture sensor detects the movement of the worker's fingers. The detection results from the sensor unit 17 are notified to the control unit 11. The control unit 11 may control the operation of the wearable device 10 based on the detection results from the sensor unit 17.

[0047] Figure 3 is a block diagram showing the internal configuration of the server device 20. The server device 20 is a dedicated or general-purpose server device and includes, for example, a control unit 21, a storage unit 22, a communication unit 23, an operation unit 24, and a display unit 25.

[0048] The control unit 21 includes a CPU, ROM, RAM, etc. The ROM in the control unit 21 stores control programs and the like that control the operation of each hardware component of the server device 20. The CPU in the control unit 21 reads and executes the control programs stored in the ROM and the computer programs described later stored in the memory unit 22, and by performing processes to control the operation of the hardware components, the entire device functions as the server device 20 of this disclosure. The RAM in the control unit 21 temporarily stores data used during the execution of various processes.

[0049] In this embodiment, the control unit 21 is configured to include a CPU, ROM, and RAM, but the configuration of the control unit 21 is not limited to the above. The control unit 21 may be one or more processing circuits, for example, including a GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), quantum processor, volatile or non-volatile memory, etc. The control unit 21 may also include functions such as a clock that outputs date and time information, a timer that measures the elapsed time from the time a measurement start instruction is given to the time a measurement end instruction is given, and a counter that counts numbers.

[0050] The storage unit 22 is equipped with storage devices such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The storage unit 22 stores various types of data and various computer programs executed by the control unit 21. The storage unit 22 also includes a work procedure database DB1 that stores work procedure manuals created for each type of work and each type of equipment, and a work history database DB2 that manages the history of work performed by workers. An overview of the work procedure database DB1 will be explained using Figure 4. An overview of the work history database DB2 will be explained using Figure 7.

[0051] The communication unit 23 includes a communication module for wireless communication with external devices such as the wearable device 10. The communication module may utilize known mobile communication standards such as 3G, 4G, and 5G, or wireless LAN methods such as Wi-Fi (registered trademark). The communication unit 23 communicates with external devices such as the wearable device 10 via a communication network NW, receiving data such as field-of-view images (video data) and transmitting appropriate data to the external device. The communication unit 23 may further include a communication module for short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark).

[0052] The operation unit 24 is equipped with operating devices such as a touch panel, keyboard, and switches, and accepts various inputs and operations from work managers and others. The control unit 21 acquires information input through the operation unit 24 and performs appropriate control based on the various operation information provided by the operation unit 24.

[0053] The display unit 25 is equipped with a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor, and displays information that should be notified to the work manager, etc., in response to instructions from the control unit 21.

[0054] The server device 20 may be a single computer, or it may be a computer system composed of multiple computers and peripheral devices. Furthermore, the server device 20 may be a virtualized virtual machine, or it may be a cloud service. Additionally, some of the functions of the server device 20 may be implemented through a combination of SaaS (Software as a Service).

[0055] Figure 4 is a conceptual diagram showing an example of the work procedure database DB1. The work procedure database DB1 is a database that stores work procedure data for each type of work and equipment. In the work procedure database, for example, the procedure number, type, category, and body text are stored in association. The procedure number represents the order of the procedure. The type represents the distinction between guidance and questions. If the type is a question, it indicates a procedure that requires a response from the worker. In Figure 4, the questions exemplified by procedure numbers "02" to "06" all indicate that they can be answered with YES or NO. Questions registered in the work procedure are not limited to those that can be answered with YES or NO; they may also include questions that expect answers through numerical input such as temperature or pressure, or questions that expect answers through text input such as error codes or free-form text. The category represents the work target, such as the outdoor unit, indoor unit, controller, or tank. The body text is a text representation of the voice guidance output from the wearable device 10, and describes the work that the worker should perform in that procedure.

[0056] Figure 4 shows a work procedure manual for a "maintenance inspection" type of work and a "room air conditioner AAA" model. The work procedure manual database DB1 stores such work procedures for each type of work and each model.

[0057] Figure 5 is an explanatory diagram illustrating an example of voice guidance provided by the wearable device 10. Prior to starting work, the worker downloads the necessary work procedure manual from the server device 20 and stores it in the storage unit 12 of the wearable device 10. Specifically, the worker accesses the work procedure manual database DB1 using a mobile terminal 30 and searches for the necessary work procedure manual from the work procedure manual database DB1 according to the type of work and the equipment being worked on. To download the retrieved work procedure manual, the worker sends a download request from the mobile terminal 30 to the server device 20. The wearable device 10 is specified as the download destination. In response to the download request from the mobile terminal 30, the server device 20 reads the corresponding work procedure manual from the work procedure manual database DB1 and sends the read work procedure to the wearable device 10. The wearable device 10 receives the work procedure manual sent from the server device 20 and stores the received work procedure manual in the storage unit 12.

[0058] Alternatively, the work supervisor may give instructions to the server device 20, and the server device 20 may send appropriate work procedures to the wearable device 10 according to the type of work and the equipment being worked on.

[0059] The wearable device 10 provides voice guidance by referring to the work procedure manual downloaded from the server device 20. For example, when providing voice guidance according to the work procedure manual illustrated in Figure 4, the wearable device 10 first reads the text associated with procedure number "01" and outputs the read text as voice guidance from the sound output unit 16. Existing technology is used for the text-to-speech conversion. Since procedure number "01" is of the guidance type, a response from the worker is not required, but in the example in Figure 5, the worker is shown to have responded to the voice guidance from the wearable device 10 by saying, "I pressed it."

[0060] The wearable device 10 determines whether the work procedure guided by the voice guidance for procedure number "01" has been completed. For example, the wearable device 10 determines that the work is complete when it receives device data indicating the completion of the work from the target equipment (in the case of procedure number "01", device data indicating that the start button has been pressed).

[0061] If the wearable device 10 determines that the procedure for procedure number "01" is complete, it moves on to the next procedure, procedure number "02". The wearable device 10 reads the text associated with procedure number "02" as before and outputs the read text as voice guidance from the sound output unit 16. Since the text for procedure number "02" is a question that can be answered with YES or NO, voice guidance such as "Please answer with YES or NO" may be added.

[0062] The wearable device 10 can determine whether or not the work procedure for procedure number "02" has been completed based on the worker's response (voice). If it determines that the work procedure has not been completed, the wearable device 10 may provide additional voice guidance. Figure 5 shows an example where, in response to the voice guidance "Have you cleaned the heat exchanger?", the worker answered "No," and the wearable device 10 then provided additional voice guidance, "Please clean the heat exchanger."

[0063] The wearable device 10 will sequentially provide voice guidance in accordance with the work procedures defined in the work procedure manual, in the same manner as described below.

[0064] In the example shown in Figure 5, the wearable device 10 is configured to determine whether each work procedure has been completed. However, the device data obtained from the work target equipment and the worker's responses may be sent to the server device 20, and the server device 20 may determine whether each work procedure has been completed. When the server device 20 determines that the Nth work procedure has been completed, it instructs the wearable device 10 to proceed to the N+1th work procedure. The wearable device 10 then follows the instructions from the server device 20 and provides voice guidance for the next work procedure according to the work procedure manual.

[0065] The following describes the processes performed by the wearable device 10. Figure 6 is a flowchart showing the voice guidance processing procedure performed by the wearable device 10 according to Embodiment 1. The necessary work procedure manual is stored in the storage unit 12 of the wearable device 10. The control unit 11 reads the transmission control program PG from the storage unit 12 and executes it, thereby performing the following processing.

[0066] The control unit 11 first sets the counter value to i=1 (step S101). The control unit 11 refers to the work procedure manual stored in the memory unit 12 and, triggered by the voice guidance for the i-th work procedure, gives instructions to the imaging unit 14 and the sound input unit 15 to start imaging and recording (step S102). The control unit 11 acquires imaging data obtained by imaging the worker's work from the imaging unit 14 and acquires acoustic data obtained by recording sounds at the work site from the sound input unit 15.

[0067] After starting imaging and recording, the control unit 11 outputs voice guidance related to the i-th work procedure from the sound output unit 16 (step S103). The voice guidance includes instructions to inform the worker of the work to be performed, or questions to confirm the work performed by the worker. The control unit 11 outputs one or more voice guidances for each work procedure. The control unit 11 may output only instructions as voice guidance, or only questions as voice guidance. For example, after outputting voice guidance to announce the start of the i-th work procedure, the control unit 11 may output a question as voice guidance to confirm the work performed by the worker. Depending on the answer received from the worker, the control unit 11 may output additional voice guidance. In this embodiment, since the voice guidance is output in step S103 after starting imaging and recording in step S102, the speech from the voice guidance can be recorded as audio data.

[0068] In this embodiment, the control unit 11 is configured to acquire both imaging data and audio data, but it may also be configured to acquire only one of the two. In addition to imaging data and audio data, the control unit 11 may also acquire equipment data obtained about the equipment being worked on.

[0069] The control unit 11 determines whether or not the work result has been entered (step S104). The control unit 11 outputs a question as voice guidance to confirm the work performed by the worker, and if the worker's answer to the voice guidance is input by voice through the sound input unit 15, it determines that the work result has been entered.

[0070] If the control unit 11 determines that the work result has been entered (S104: YES), it immediately uploads the work result data to the server device 20 (step S105). At this time, the control unit 11 only needs to convert the audio data corresponding to the worker's response into text data and upload the converted text data to the server device 20. Existing speech recognition technology is used for the conversion from audio data to text data. If the system is configured to accept only pre-registered words such as YES, NO, numbers, and error codes as the worker's response, the wearable device 10 can convert the audio data into text data with relatively high accuracy. If the control unit 11 determines that the work result has not been entered (S104: NO), it proceeds to the process in step S106.

[0071] The control unit 11 determines whether the i-th work procedure has been completed (step S106). For example, if the control unit 11 uploads the work result data to the server device 20 and then receives a response from the server device 20 indicating that the work result data has been received, it determines that the i-th work procedure has been completed. Alternatively, if the server device 20, which has received the work result data, instructs the control unit 11 to proceed to the next work procedure, the control unit 11 may determine that the i-th work procedure has been completed. If it determines that the i-th work procedure has not been completed (S106: NO), the control unit 11 returns the process to step S104.

[0072] If the control unit 11 determines that the i-th work procedure has been completed (S106: YES), it terminates the imaging and recording that was started in step S103, and stores the imaging data and acoustic data obtained during that time in the storage unit 12 as the imaging data and acoustic data for the i-th work procedure (step S107).

[0073] Next, the control unit 11 refers to the work procedure manual and determines whether all work procedures have been completed (step S108). If there are any work procedures that have not been completed (S108: NO), the control unit 11 increments the value of the counter by 1 (step S109) and returns to step S102. If it is determined that all work procedures have been completed (S108: YES), the control unit 11 terminates the process according to this flowchart.

[0074] In this flowchart, the control unit 11 of the wearable device 10 determines the end of the work procedure in steps S106 and S108. However, the control unit 21 of the server device 20 may determine the end of the work procedure based on the work result data uploaded immediately from the wearable device 10. In this case, the server device 20 should instruct the wearable device 10 to proceed to the next work procedure. The wearable device 10 will follow the instructions from the server device 20 and provide voice guidance for the next work procedure according to the work procedure manual.

[0075] Figure 7 is a flowchart showing the periodic upload processing procedure performed by the wearable device 10 according to Embodiment 1. The control unit 11 performs the periodic upload processing of work data in parallel with the processing of the voice guidance described above.

[0076] The control unit 11 determines whether or not it is time for a periodic upload (step S121). The timing of a periodic upload is defined in advance, for example, by the transmission control program PG. The timing of a periodic upload can be any pre-set timing and does not need to be strictly periodic.

[0077] If the control unit 11 determines that it is not time for a scheduled upload (S121: NO), it waits until it is time for a scheduled upload. If it determines that it is time for a scheduled upload (S121: YES), the control unit 11 sequentially uploads the imaging data and acoustic data stored in the storage unit 12 to the server device 20 for each work procedure (step S122). At this time, the control unit 11 may detect the communication status by the communication unit 13 and transmit the imaging data and acoustic data to the server device 20 only if the communication status is good (for example, if the radio wave strength is above a threshold). If the control unit 11 has acquired equipment data in addition to the imaging data and acoustic data, it may also upload the equipment data to the server device 20.

[0078] The control unit 11 determines whether the upload of imaging data and acoustic data has been completed for all work procedures (step S123). If there is imaging data or acoustic data that has not been uploaded (S123: NO), the control unit 11 returns to step S121. If it is determined that the upload of imaging data and acoustic data has been completed for all work procedures (S123: YES), the control unit 11 terminates the process according to this flowchart.

[0079] The server device 20 registers the work data uploaded from the wearable device 10 into the work history database DB2. Figure 8 is a conceptual diagram showing an example of the work history database DB2. The work history database DB2 stores information about the work performed by the worker, associating it with, for example, item ID, item name, date and time, worker, work result, imaging data, and acoustic data. The item ID is an identifier used to identify each work procedure shown in the work procedure manual. The item name is the name of each work procedure shown in the work procedure manual. The date and time is the work date and time. Both the work start date and time and the work end date and time may be registered in the date and time. The worker represents the name of the worker. The work result represents the result of the worker's answer to a question output as voice guidance. The work result is registered as YES or NO. The work result may also include numerical values, error codes, etc., answered by the worker. The imaging data is imaging data for each work procedure uploaded from the wearable device 10. The acoustic data is acoustic data for each work procedure uploaded from the wearable device 10.

[0080] In the work history database DB2 shown in Figure 8, work data such as work result data, imaging data, and acoustic data are registered. In addition to these, equipment data obtained from the equipment being worked on may also be registered.

[0081] In the work history database DB2 shown in Figure 8, the file names of the imaging data and acoustic data are shown in a simplified manner. However, the file names may be set in a way that indicates which work procedure the data pertains to. For example, the file name may include the procedure identifier, procedure identifier, start date and time, end date and time, and the number of times the work has been performed. In this case, the file name can be treated as metadata.

[0082] The worker can access the server device 20 using a wearable device 10 or a mobile terminal 30 to check the status of the work data upload. Figure 9 is a schematic diagram showing the upload status confirmation screen. The worker can use the wearable device 10 to obtain information regarding the work data upload status from the server device 20 and transfer the obtained information to the mobile terminal 30, thereby displaying a confirmation screen like the one shown in Figure 9 on the display of the mobile terminal 30. If the wearable device 10 has a display unit, the confirmation screen may be displayed on the display unit of the wearable device 10.

[0083] The confirmation screen in Figure 9 displays the item names of the work procedures, the start date and time of the work, and the work results, along with the upload status of the work data for the outdoor unit inspection work. The work results are displayed as text information. Here, "Actual:-" indicates that the upload of the work results for the corresponding work item has not yet been completed. The upload status of the image data and sound data is indicated by icons. In the example in Figure 9, items that have been uploaded are indicated by solid line icons, and items that have not been uploaded are indicated by dashed line icons.

[0084] The worker can check the work results and the upload status of the work data by looking at the confirmation screen displayed on the mobile terminal 30.

[0085] In Figure 9, the configuration is shown to display the upload status of work data on the screen of the mobile terminal 30. However, the server device 20 may refer to the work history database DB2, create a work report summarizing the work details for each worker, and output it as an electronic or paper document.

[0086] As described above, in Embodiment 1, since voice guidance is provided for each work procedure using the wearable device 10, the worker can confirm the work procedure without interrupting their work and input the work results by voice. With this configuration, it is possible to improve work efficiency and reduce work errors.

[0087] In Embodiment 1, the timing of uploading work data to the server device 20 can be controlled according to the priority of the work data. In Embodiment 1, since the priority of work result data is set high, when work result data is input, it is uploaded to the server device 20 immediately. When other work data, including imaging data, acoustic data, and equipment data, is input, it is uploaded to the server device 20 retrospectively. The server device 20 checks the work results that are uploaded immediately, determines whether the work was performed properly, and if it determines that it was not performed properly, it can guide the worker to perform the work properly by outputting voice guidance from the wearable device 10.

[0088] The server device 20 stores the work results, imaging data, acoustic data, etc., associated with each work procedure, so that the imaging data, acoustic data, etc. can be used as a guarantee and evidence of the work results.

[0089] In Embodiment 1, first-person perspective video (image data) and annotations for that data (work content for each work procedure) are automatically collected in the work history database DB2. The data collected in the work history database DB2 can be easily repurposed as training data for video analysis.

[0090] In Embodiment 1, the priority of the work result data was set higher than the priority of the imaging data, acoustic data, and equipment data. Alternatively, the priority of the imaging data may be set higher than the priority of the work result data, acoustic data, and equipment data. In this case, the wearable device 10 may immediately upload the imaging data input through the imaging unit 14 to the server device 20, for example, in units of frames lasting several seconds. The server device 20 can analyze the imaging data uploaded immediately from the wearable device 10 and perform work recognition or scene recognition. Existing methods are used for analyzing the imaging data and performing work recognition or scene recognition. When work recognition or scene recognition is completed, the server device 20 may notify the wearable device 10 of this. The remaining frames after work recognition or screen recognition is completed may be transmitted retrospectively by periodic uploads.

[0091] Furthermore, the priority of acoustic data may be set higher than that of work result data, imaging data, and equipment data. For example, in work procedures where the surrounding environment is likely to change significantly (such as procedures immediately after moving to or from an outdoor or indoor environment), the priority of acoustic data may be set higher. In this case, the wearable device 10 may immediately upload the acoustic data input through the sound input unit 15 to the server device 20, for example, every few seconds. The server device 20 can analyze the work data immediately uploaded from the wearable device 10 and perform acoustic recognition. Acoustic recognition may be speech recognition that recognizes the worker's voice, or it may be abnormal sound detection to detect abnormal noises emitted from the work equipment or work environment. Existing methods can be used for acoustic recognition such as speech recognition or abnormal sound detection. If the server device 20 detects an abnormal sound through acoustic recognition, it may instruct the wearable device 10 to output additional voice guidance. The server device 20 may also detect the type and level of noise through acoustic recognition and adjust the speech recognition model and preprocessing parameters.

[0092] (Embodiment 2) Embodiment 2 describes a configuration in which priority is set for each work procedure. The overall configuration of the information processing system 1, and the internal configurations of the wearable device 10 and the server device 20 are the same as in Embodiment 1, so their description will be omitted.

[0093] Figure 10 is an explanatory diagram illustrating an example of priority setting in Embodiment 2. In Embodiment 2, a priority is set for each work procedure. The priority for each work procedure is set, for example, by the work manager and registered in the work procedure database DB1. In the work procedure database DB1 shown in Figure 10, in addition to the procedure number, type, category, and body text mentioned above, a priority item is added. The priority represents the priority when uploading work data to the server device 20.

[0094] In the example shown in Figure 10, the work procedures numbered "01" to "03" are set to a low priority. In this case, the work data obtained from work procedures numbered "01" to "03" is uploaded to the server device 20 retrospectively. That is, this work data is uploaded to the server device 20 at regular intervals, not immediately.

[0095] On the other hand, the work procedures numbered "04" to "05" are set to "high" priority. In this case, the work data obtained from the work procedures numbered "04" to "05" is immediately uploaded to the server device 20. The imaging data and acoustic data should be uploaded sequentially at intervals of a few seconds.

[0096] For procedure number "06," the priority is set to "high," and the application condition is specified as "model AAA." In this case, the work data obtained by procedure number "06" will be uploaded to the server device 20 immediately only if the model of the target equipment is "AAA." If the model of the target equipment is anything other than "AAA," the work data obtained by procedure number "06" will be uploaded to the server device 20 retrospectively.

[0097] In the procedure for step number "06," the model of the equipment being worked on is specified as a condition for applying a "high" priority. However, instead of specifying the model of the equipment being worked on, a configuration specifying a measuring instrument may also be used. For example, by specifying an ammeter, voltmeter, electrical resistance meter, power meter, thermometer, anemometer, wind speed meter, etc., the priority of procedures for measuring specific physical quantities can be set relatively higher.

[0098] The wearable device 10 can refer to the priority of each work procedure defined in the work procedure manual obtained from the server device 20 and decide whether to upload the data obtained in each work procedure immediately or retrospectively.

[0099] Figure 11 is a flowchart showing the voice guidance processing procedure performed by the wearable device 10 according to Embodiment 2. The necessary work procedure manual is stored in the storage unit 12 of the wearable device 10. The control unit 11 reads the transmission control program PG from the storage unit 12 and executes it, thereby performing the following processing.

[0100] The control unit 11 first sets the counter value to i=1 (step S201). The control unit 11 refers to the work procedure manual stored in the memory unit 12 and, triggered by the voice guidance for the i-th work procedure, gives instructions to the imaging unit 14 and the sound input unit 15 to start imaging and recording (step S202). The control unit 11 acquires imaging data obtained by imaging the worker's work from the imaging unit 14 and acquires acoustic data obtained by recording sounds at the work site from the sound input unit 15.

[0101] After the control unit 11 starts imaging and recording, it outputs voice guidance related to the i-th work procedure from the sound output unit 16 (step S203). In this embodiment, since the voice guidance is output in step S203 after the imaging and recording are started in step S202, the speech from the voice guidance can be recorded as acoustic data.

[0102] The control unit 11 refers to the work procedure manual and determines whether the priority set for the i-th work procedure is "high" (step S204). If the control unit 11 determines that the priority set for the i-th work procedure is "high" (S204: YES), it immediately uploads the work data obtained from that work procedure (step S205). In other words, when work result data is input, the control unit 11 should immediately upload the work result data to the server device 20, and when imaging data, audio data, and equipment data are input, they should be immediately uploaded to the server device 20 within a few seconds.

[0103] The server device 20 may determine whether the work procedure was performed correctly based on the work data uploaded immediately, and may notify the wearable device 10 of the determination result. If the wearable device 10 is notified by the server device 20 that the work procedure was performed correctly, it may retrospectively upload subsequent work data.

[0104] If the control unit 11 determines that the priority set for the i-th work procedure is not "high" (S204: NO), it will skip the processing in step S205 and execute the processing from step S206 onwards.

[0105] The control unit 11 determines whether the i-th work procedure has been completed (step S206). For example, if the control unit 11 uploads the work result data to the server device 20 and then receives a response from the server device 20 indicating that the work result data has been received, it determines that the i-th work procedure has been completed. Alternatively, if the server device 20, which has received the work result data, instructs the control unit 11 to proceed to the next work procedure, the control unit 11 may determine that the i-th work procedure has been completed. If it determines that the i-th work procedure has not been completed (S206: NO), the control unit 11 returns the process to step S204.

[0106] If the control unit 11 determines that the i-th work procedure has been completed (S206: YES), it terminates the imaging and recording that was started in step S203, and stores the imaging data and acoustic data obtained during that time in the storage unit 12 as the imaging data and acoustic data for the i-th work procedure (step S207).

[0107] Next, the control unit 11 refers to the work procedure manual and determines whether all work procedures have been completed (step S208). If there are any work procedures that have not been completed (S208: NO), the control unit 11 increments the value of the counter by 1 (step S209) and returns to step S202. If it is determined that all work procedures have been completed (S208: YES), the control unit 11 terminates the process according to this flowchart.

[0108] Figure 12 is a flowchart showing the periodic upload processing procedure performed by the wearable device 10 according to Embodiment 2. The control unit 11 performs the periodic upload processing of work data in parallel with the processing of the voice guidance described above.

[0109] The control unit 11 determines whether or not it is time for a periodic upload (step S221). The timing of a periodic upload is defined in advance, for example, by the transmission control program PG. The timing of a periodic upload can be any pre-set timing and does not need to be strictly periodic.

[0110] If the control unit 11 determines that it is not time for a scheduled upload (S221: NO), it waits until it is time for a scheduled upload. If it determines that it is time for a scheduled upload (S221: YES), the control unit 11 sequentially uploads the work data obtained from the work procedures with low priority to the server device 20 (step S222). At this time, the control unit 11 may detect the communication status by the communication unit 13 and send the work data to the server device 20 only if the communication status is good (for example, if the radio wave strength is above a threshold).

[0111] The control unit 11 determines whether the upload of work data has been completed for all work procedures (step S223). If there is any imaging data or acoustic data that has not been uploaded (S223: NO), the control unit 11 returns to step S221. If it is determined that the upload of work data has been completed for all work procedures (S223: YES), the control unit 11 terminates the process according to this flowchart.

[0112] As described above, in Embodiment 2, since the priority level for each work procedure is set in advance in the work procedure manual, the wearable device 10 can refer to the work procedure manual and immediately upload work data with a relatively high priority to the server device 20, and upload work data with a relatively low priority to the server device 20 retrospectively.

[0113] (Embodiment 3) Embodiment 3 describes a configuration that recognizes a work scene and determines whether or not to transmit imaging data according to the recognized work scene. The overall configuration of the information processing system 1, and the internal configurations of the wearable device 10 and the server device 20 are the same as in Embodiment 1, so their description will be omitted.

[0114] Figure 13 is a flowchart showing the processing steps performed by the wearable device 10 according to Embodiment 3. The control unit 11, following the same procedure as in Embodiments 1 and 2, triggers voice guidance for each work procedure, gives instructions to the imaging unit 14 and the sound input unit 15, starts imaging and recording, and outputs the voice guidance from the sound output unit 16. The control unit 11 acquires imaging data obtained by imaging the worker's work from the imaging unit 14, and acquires acoustic data obtained by recording sounds at the work site from the sound input unit 15.

[0115] The control unit 11 performs the following processing each time it acquires imaging data from the imaging unit 14. The control unit 11 recognizes the work scene based on the acquired imaging data (step S301). Existing methods are used for scene recognition. For example, EgoVLPv2 based on Transformer, or a scene recognition model tuned from EgoVLPv2 can be used. The scene recognition model is configured, for example, to output a scene recognition result that includes either "work" or "non-work" in response to input of imaging data in predetermined units (for example, imaging data in 6-second units).

[0116] The control unit 11 determines whether or not to upload the image data based on the recognition result of the work scene (step S302). If the recognition result of the work scene is "work," such as installation work or inspection work of the equipment to be worked on, the control unit 11 decides to upload the image data. If the recognition result of the work scene is "non-work," such as taking a break or moving around, the control unit 11 decides not to upload the image data.

[0117] If it is determined that the image data should be uploaded (S302: YES), the control unit 11 uploads the image data (step S303). The timing of uploading the image data follows the timing described in Embodiment 1 or 2. That is, as described in Embodiment 1, the control unit 11 may upload the image data periodically at a subsequent timing. Alternatively, as described in Embodiment 2, the control unit 11 may upload the image data immediately or periodically according to the priority level set for each work procedure.

[0118] If it is determined in step S302 not to upload the imaging data (S302: NO), the control unit 11 terminates the processing according to this flowchart.

[0119] As described above, in Embodiment 3, since image data from non-working situations such as during breaks or while moving is not uploaded to the server device 20, the load on the wearable device 10 and the server device 20 can be reduced.

[0120] Furthermore, while the application example of the information processing system according to this embodiment is assumed to be a work scenario involving air conditioning equipment, it is of course applicable to a variety of work scenarios, not limited to air conditioning equipment, such as work scenarios involving elevators, or maintenance and inspection work scenarios in chemical plants and power supply facilities.

[0121] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended. [Explanation of Symbols]

[0122] 10 Wearable Devices 11 Control Unit 12 Storage section 13 Communications Department 14 Imaging Unit 15. Audio input section 16. Sound output section 17. Sensor Unit 20 Server Devices PG Transmission Control Program DB1 Work Procedure Database DB2 Work History Database

Claims

1. We acquire multiple work data related to the work performed by the worker. The timing of sending the acquired work data to the server device is controlled according to the priority of the acquired work data. An information processing method in which the processing is performed by the operator's terminal device.

2. The terminal device transmits work data with relatively high priority immediately and work data with relatively low priority afterward. The information processing method according to claim 1.

3. The aforementioned work data includes work result data entered by the worker, imaging data obtained by imaging the worker's work, acoustic data obtained by recording sounds at the work site, or equipment data obtained about the equipment being worked on. The information processing method according to claim 2.

4. The priority of the data representing the work results is set higher than the priority of the imaging data, the acoustic data, and the equipment data. The information processing method according to claim 3.

5. When performing immediate work recognition based on the aforementioned imaging data, the priority of the imaging data is set higher than the priority of the work result data, the acoustic data, and the equipment data. The information processing method according to claim 3.

6. When performing immediate acoustic recognition based on the aforementioned acoustic data, the priority of the acoustic data is set higher than the priority of the work result data, the imaging data, and the equipment data. The information processing method according to claim 3.

7. The terminal device acquires work data for each of the multiple work procedures. The priority of the aforementioned work data is set according to the work procedure. The information processing method according to claim 2.

8. The aforementioned terminal device is The worker will be given instructions on the work procedure for each step of the process. The guidance on the aforementioned work procedures will trigger the commencement of at least one of the following: imaging of the work at each work step and recording of sounds at the work site. The information processing method according to claim 3.

9. The aforementioned terminal device is Based on the aforementioned imaging data, the work scene is recognized, Depending on the recognized work scene, it is determined whether or not to transmit the image data. The information processing method according to claim 3.

10. The aforementioned terminal device is Detects the communication status, Based on the detection results of the aforementioned communication status, it is determined whether or not to transmit the work data with a relatively low priority. The information processing method according to claim 2.

11. The aforementioned terminal device is From the server device, obtain information relating to the transmission status of the work data. Based on the acquired information, output the transmission status of the aforementioned work data. The information processing method according to claim 1.

12. The worker's terminal device, A server device that is connected to the aforementioned terminal device in a manner that enables communication. Equipped with, The aforementioned terminal device is Multiple work data related to the work performed by the aforementioned worker are acquired, The timing of transmitting the acquired work data to the server device is controlled according to the priority of the acquired work data. Information processing system.

13. It comprises at least one control unit, The control unit, We acquire multiple work data related to the work performed by the worker. The timing of sending the acquired work data to the server device is controlled according to the priority of the acquired work data. Terminal device.

14. We acquire multiple work data related to the work performed by the worker. The timing of sending the acquired work data to the server device is controlled according to the priority of the acquired work data. A computer program that causes a computer to perform a process.

Images