Computer program, information processing method, and information processing system

Abstract

Provided are a computer program, an information processing method, and an information processing system.　The present invention causes at least one computer to execute a process of acquiring a work video obtained by imaging work performed by a worker, detecting, on the basis of the acquired work video, a work scene in which restoration to an original state is carried out, and determining that the work is completed when the work scene has been detected.

Description

Computer Program, Information Processing Method, and Information Processing System

[0001] The present invention relates to a computer program, an information processing method, and an information processing system.

[0002] Patent Document 1 discloses a technique for determining start and end frames of work from captured frames by image analysis, inspecting a target group of frames after work completion, and notifying if there is a defect.

[0003] Japanese Patent Application Laid-Open No. 2019-191117

[0004] In Patent Document 1, image analysis is performed on the captured frames, and when the work determination result of the immediately preceding frame is different, it is determined that one work has ended, and it is not possible to determine the end of work according to the detection result of the work scene.

[0005] An object of the present disclosure is to provide a computer program, an information processing method, and an information processing system that can determine the end of work according to the detection result of a work scene.

[0006] The computer program according to the first aspect of the present disclosure acquires a work video obtained by imaging work by a worker, detects a work scene to be restored to its original state based on the acquired work video, and when the work scene is detected, causes at least one computer to execute a process of determining that the work has ended.

[0007] The computer program according to the second aspect of the present disclosure, in the computer program according to the first aspect, when it is determined that the work has ended, causes the computer to execute a process of transmitting the work video to the outside.

[0008] The computer program according to the third aspect of the present disclosure, in the computer program according to the first or second aspect, determines the presence or absence of work omission for each work content based on the work video, and when it is determined that there is work omission, causes the computer to execute a process of notifying the work omission.

[0009] The computer program relating to the fourth aspect of this disclosure is a computer program relating to any one of the first to third aspects, in which the work scene is a scene in which assembly work, tool cleanup, equipment operation check, or protective covering removal is being carried out.

[0010] The computer program relating to the fifth aspect of this disclosure is a computer program relating to any one of the first to fourth aspects, wherein the work video is a video obtained by capturing images of repair or inspection work on equipment.

[0011] The computer program relating to the sixth aspect of this disclosure is a computer program relating to any one of the first to fifth aspects, wherein the work video is a video captured by a wearable terminal worn by the worker.

[0012] The computer program relating to the seventh aspect of this disclosure will notify the computer program relating to any one of the first to sixth aspects to terminate imaging when it detects the aforementioned work scene.

[0013] The computer program relating to the eighth aspect of this disclosure outputs a notification inquiring whether or not the restoration work has been performed if the computer program relating to any one of the first to seventh aspects does not detect the work scene and an instruction to terminate imaging is given.

[0014] A computer program relating to the ninth aspect of this disclosure, in a computer program relating to any one of the first to eighth aspects, identifies a plurality of work items performed by the worker based on the acquired work video, obtains a list of a plurality of work items to be performed in relation to the work, and determines whether the work has been completed by comparing the identified plurality of work items with the plurality of work items included in the acquired list.

[0015] The computer program relating to the tenth aspect of this disclosure compares the specified work items with the work items included in the acquired list in no particular order, in the computer program relating to the ninth aspect.

[0016] The computer program relating to the 11th aspect of this disclosure determines whether there are any missing tasks by comparing the work items in the computer program relating to the 9th aspect, and if it is determined that there are missing tasks, it notifies the computer of the missing tasks.

[0017] The information processing method relating to the twelfth aspect of this disclosure includes acquiring a work video obtained by imaging the work performed by an worker, detecting a work scene to be restored to its original state based on the acquired work video, and, if such a work scene is detected, performing a process by at least one computer to determine that the work has been completed.

[0018] The information processing system relating to the 13th aspect of this disclosure comprises at least one processing unit, which acquires a work video obtained by imaging the work performed by an operator, detects a work scene to be restored to its original state based on the acquired work video, and determines that the work has been completed when such a work scene is detected.

[0019] According to this disclosure, the completion of work can be determined based on the detection results of the work scene.

[0020] This is an explanatory diagram illustrating the outline of the processing performed by the information processing system according to Embodiment 1. This is a block diagram illustrating the internal configuration of a wearable device. This is a block diagram illustrating the internal configuration of an analysis server. This is an explanatory diagram illustrating the method for generating a work classification model. This is an explanatory diagram illustrating the method for annotating work videos. This is an explanatory diagram illustrating a scene recognition method using a work classification model. This is a flowchart illustrating the procedure for processing performed by the analysis server according to Embodiment 1. This is a conceptual diagram showing an example of a work item list. This is a flowchart illustrating the procedure for processing performed by the analysis server according to Embodiment 2. This is a flowchart illustrating the procedure for processing performed by a wearable device according to Embodiment 3.

[0021] The information processing system according to the embodiment will be described in detail below with reference to the drawings. (Embodiment 1) Figure 1 is an explanatory diagram illustrating the outline of the processing performed by the information processing system 1 according to Embodiment 1. The information processing system 1 is a system for acquiring work videos obtained by imaging the work performed by a worker and for analyzing the acquired work videos. In this embodiment, the worker is a worker who performs repair work, inspection work, etc., on equipment such as air conditioning equipment. At the end of the work, such a worker performs assembly work, puts away tools, checks the operation of the equipment, removes protective coverings, etc.

[0022] In this embodiment, a wearable device 10 equipped with imaging capabilities is attached to the worker in order to capture images of the worker's work. In one example, the wearable device 10 is attached to the worker's neck. Alternatively, the wearable device 10 may be attached to the worker's head, or to another body part such as the shoulder or arm. Furthermore, the wearable device 10 may be a goggle-type camera device, or any device equipped with imaging capabilities, such as a smartphone or action camera, may be used instead of the wearable device 10.

[0023] The wearable device 10 captures images of the worker's work and generates a video (work video). Alternatively, the wearable device 10 may generate multiple still images obtained by capturing images at predetermined time intervals.

[0024] In this embodiment, the work video is a video from the worker's point of view (first-person perspective video), but it does not mean that the video perfectly reproduces the worker's field of view. That is, the range (angle of view) captured by the wearable device 10 does not need to strictly match the range that the worker sees, nor does the direction of imaging of the wearable device 10 need to strictly match the direction of the worker's gaze. The wearable device 10 only needs to be worn on the worker so that it is generally facing forward, so that it can capture at least a part of the worker's field of view.

[0025] The information processing system 1 includes an analysis server 20 that analyzes work videos obtained from the wearable device 10. The analysis server 20 is connected to the wearable device 10 via a communication network NW, such as the Internet. The analysis server 20 acquires work videos from the wearable device 10 by communicating with the wearable device 10 via the communication network NW. In this embodiment, the analysis server 20 acquires work videos from the wearable device 10 in real time. The analysis server 20 may acquire work videos frame by frame, or it may acquire work videos frame by frame at predetermined time intervals.

[0026] The path by which the analysis server 20 acquires work videos is not limited to a path that directly communicates with the wearable device 10. For example, work videos captured by the wearable device 10 may be transmitted to the analysis server 20 via another terminal device (e.g., the worker's smartphone).

[0027] The analysis server 20 performs a process to distinguish and recognize multiple types of work scenes based on the acquired work video. The analysis server 20 can classify the worker's work using the work classification model MD1 (see Figure 4), which will be described later, and distinguish and recognize multiple types of work scenes based on the classification results. In this embodiment, the analysis server 20 divides the work video into predetermined time units (for example, 6-second units) and performs scene recognition for each work video in each time unit. The work scenes recognized by the analysis server 20 include work scenes that restore the original state, such as assembly work, putting away tools, checking the operation of equipment, and removing protective coverings.

[0028] If the analysis server 20 detects a work scene that requires restoration to the original state as a result of scene recognition, it determines that the worker's work has been completed. If it determines that the worker's work has been completed, the analysis server 20 may issue an instruction to the wearable device 10 to stop imaging. The wearable device 10 stops capturing work video in response to the instruction from the analysis server 20. Since the worker does not need to perform a work completion operation, unnecessary video uploads due to forgetting to perform the operation are eliminated.

[0029] When the analysis server 20 determines that the worker has completed their work, it may determine whether any work has been missed based on the work video acquired during the work, and if it determines that work has been missed, it may notify the worker. The notification to the worker may be made via the wearable device 10 or via another terminal device.

[0030] Figure 2 is a block diagram showing the internal configuration of the wearable device 10. The wearable device 10 includes a processing unit 11, a storage unit 12, a communication unit 13, an imaging unit 14, an audio input unit 15, an audio output unit 16, a sensor unit 17, an operation unit 18, and the like.

[0031] The processing unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and the like. The ROM in the processing unit 11 stores control programs and the like that control the operation of each hardware component of the wearable device 10. The CPU in the processing unit 11 reads and executes the control programs and the like stored in the ROM, and controls the operation of each hardware component, thereby making the entire device function as the wearable device 10 of this disclosure. The RAM in the processing unit 11 temporarily stores data used during the execution of various processes.

[0032] The storage unit 12 is equipped with an auxiliary storage device and stores work videos and the like generated by the imaging unit 14. The storage unit 12 may also have an application program installed that is executed by the processing unit 11. The application program may be pre-installed or installed after use begins.

[0033] The communication unit 13 is equipped with a communication module for wireless communication with external devices such as the analysis server 20. The communication module may be a communication module for wireless communication using known mobile communication standards such as 3G, 4G, and 5G, or wireless LAN methods such as Wi-Fi (registered trademark). The communication unit 13 communicates with external devices such as the analysis server 20 via a communication network NW, transmits necessary data such as work videos, and receives appropriate data transmitted from external devices. The communication unit 13 may also be equipped with a communication module for short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark) to communicate with terminals such as smartphones carried by workers.

[0034] The imaging unit 14 includes an optical lens, an image sensor, a driver circuit, and the like. A wide-angle lens is preferably used as the optical lens. The image sensor is a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge-Coupled Device), etc., and generates electrical signals according to the intensity of the light imaged through the optical lens. The driver circuit includes a timing generator (TG), etc., and sequentially reads electrical signals from the image sensor in synchronization with the clock signal output from the TG, generating video data. The video data generated by the imaging unit 14 is sent to the processing unit 11 and stored in the storage unit 12. Alternatively, the video data generated by the imaging unit 14 is transmitted to the analysis server 20 via the communication unit 13.

[0035] The sound input unit 15 includes a microphone for collecting sound, a processing circuit for converting the collected sound into a digital signal (acoustic data), and the like. The acoustic data generated in the sound input unit 15 is sent to the processing unit 11, where appropriate processing such as noise reduction is performed. The acoustic data generated in the sound input unit 15 is also stored in the storage unit 12 or transmitted to the analysis server 20 via the communication unit 13.

[0036] The sound output unit 16 is equipped with a speaker that outputs sound. The sound output unit 16 outputs sound based on acoustic data provided by the processing unit 11.

[0037] The sensor unit 17 is equipped with non-contact sensors for detecting the hands and fingers of an operator. The sensors in the sensor unit 17 include proximity sensors and gesture sensors. For example, the proximity sensor detects when an operator's hands and fingers come within a predetermined range. For example, the gesture sensor detects the movement of an operator's hands and fingers. The detection results from the sensor unit 17 are notified to the processing unit 11. Based on the detection results from the sensor unit 17, the processing unit 11 may give an instruction to start imaging or an instruction to stop imaging to the imaging unit 14.

[0038] The control unit 18 is equipped with various operation buttons, operation switches, etc., and receives operations from the operator. Operation information corresponding to the operation of the control unit 18 is input to the processing unit 11. The processing unit 11 performs appropriate processing based on the operation information input from the control unit 18. The operator may also give an imaging start command or an imaging end command to the wearable device 10 by operating the control unit 18.

[0039] Figure 3 is a block diagram showing the internal configuration of the analysis server 20. The analysis server 20 is a dedicated or general-purpose server device and includes, for example, a processing unit 21, a storage unit 22, a communication unit 23, an operation unit 24, and a display unit 25.

[0040] The processing unit 21 includes a CPU, ROM, RAM, etc. The ROM in the processing unit 21 stores control programs and the like that control the operation of each hardware component of the analysis server 20. The CPU in the processing unit 21 reads and executes the control programs stored in the ROM and the computer programs described later stored in the storage unit 22, and by executing processes that control the operation of the hardware components, the entire device functions as the analysis server 20 of this disclosure. The RAM in the processing unit 21 temporarily stores data used during the execution of various processes.

[0041] In the embodiment, the processing unit 21 is configured to include a CPU, a ROM, and a RAM. However, the configuration of the processing unit 21 is not limited to the above. The processing unit 21 may be, for example, one or more processing circuits including a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, a volatile or non-volatile memory, and the like. Further, the processing unit 21 may have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when a measurement start instruction is given until a measurement end instruction is given, and a counter that counts numbers.

[0042] The storage unit 22 includes a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). In the storage unit 22, various computer programs executed by the processing unit 21 and various data acquired through the communication unit 23 are stored.

[0043] The computer program (program product) stored in the storage unit 22 includes an analysis processing program PG1 for analyzing a work video. The analysis processing program PG1 is a computer program for causing a computer to execute a process of acquiring a work video obtained by imaging the work of an operator, detecting a work scene to be restored to its original state based on the acquired work video, and determining that the work by the operator has ended when the work scene is detected.

[0044] The analysis processing program PG1 may be a single computer program or a program group composed of a plurality of computer programs. Further, the analysis processing program PG1 may be executed by a single computer or may be executed in cooperation by a plurality of computers (for example, the wearable device 10 and the analysis server 20).

[0045] The computer program, including the analysis processing program PG1, is provided on a non-temporary recording medium RM on which the computer program is recorded in a readable format. The recording medium RM is a portable memory such as a CD-ROM, USB memory, SD card, microSD card, or CompactFlash®. The processing unit 21 reads various computer programs from the recording medium RM using a reading device (not shown in the figure) and stores the read computer programs in the storage unit 22. The computer programs stored in the storage unit 22 may also be provided via communication. In this case, the processing unit 21 acquires the computer programs via communication through the communication unit 23 and stores the acquired computer programs in the storage unit 22.

[0046] Furthermore, the memory unit 22 stores the work classification model MD1 used in the analysis processing program PG1 described above. The work classification model MD1 is a trained model that has been trained to output information about the actions of the photographer (i.e., the worker himself) in response to the input of a work video. The work classification model MD1 is described by its definition information. The definition information of the work classification model MD1 includes information about the layers that make up the model, information about the nodes that make up each layer, and parameters such as weights and biases between nodes. These parameters are obtained by training using a dataset that includes work videos and annotation data showing the actions of the photographer as shown in the work videos as training data. In this embodiment, it is assumed that the trained work classification model MD1 is stored in the memory unit 22.

[0047] The communication unit 23 includes a communication module for wireless communication with an external device such as the wearable device 10. As the communication module, a communication module for wireless communication using a known mobile communication standard such as 3G, 4G, 5G or a wireless LAN method such as WiFi (registered trademark) is used. The communication unit 23 communicates with an external device such as the wearable device 10 via the communication network NW, receives data such as a work video, and transmits appropriate data to be transmitted to the external device. The communication unit 23 may further include a communication module for short-range wireless communication such as Bluetooth (registered trademark) and ZigBee (registered trademark).

[0048] The operation unit 24 includes operation devices such as a touch panel, a keyboard, and a switch, and receives various inputs and operations by a work manager or the like. The processing unit 21 acquires the information input through the operation unit 24 and performs appropriate control based on various operation information given from the operation unit 24.

[0049] The display unit 25 includes a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor, and displays information to be notified to a work manager or the like in response to an instruction from the processing unit 21.

[0050] The analysis server 20 may be a single computer, or may be a computer system constituted by a plurality of computers, peripheral devices, or the like. Further, the analysis server 20 may be a virtual machine in which the entity is virtualized, or may be a cloud.

[0051] The following describes the work classification model MD1 used by the analysis server 20. Figure 4 is an explanatory diagram illustrating the method for generating the work classification model MD1. In this embodiment, a model tuned from EgoVLPv2 based on Transformer is used as the work classification model MD1. In EgoVLP (Egocentric Video-Language pre-training), the VLP model MD is constructed by pre-training using a dataset containing first-person perspective video clips and annotations (text narration) for those video clips as training data. EgoClip, created from Ego4D, is used as the training data for pre-training. Ego4D records the actions (annotations) of the camera wearer in the video and timestamps for approximately 10,000 videos captured from a first-person perspective. By extracting videos based on timestamps, a dataset (EgoClip) DS consisting of numerous pairs of video clips and text is obtained.

[0052] The VLP model MD comprises a video encoder En1 that extracts features from an input video clip and a text encoder En2 that extracts features from input text. EgoVLP trains video encoder En1 and text encoder En2 using contrastive learning with EgoNCE (NCE: Noise Contrastive Estimation) as the loss function. In contrastive learning, a self-supervised learning mechanism that compares data is used to learn features such that similar data are placed close together and different data are placed far apart.

[0053] The second-generation EgoVLPv2 features a gating mechanism that enables / disables cross-attention fusion between the video encoder En1 and the text encoder En2, allowing for flexible switching between dual encoder and fusion encoder modes. Furthermore, compared to stacking dedicated conversion layers and shared encoders, it has the advantage of requiring fewer fusion parameters, less GPU memory, computational resources, and less training time.

[0054] In this embodiment, a work classification model MD1 is generated by tuning the existing model EgoVLPv2 using a dataset that includes work videos captured at the work site using a wearable device 10 and annotations added to the work videos as training data.

[0055] In this embodiment, a method for generating the task classification model MD1 using EgoVLPv2 has been described. However, the method is not limited to EgoVLPv2; other learning models such as VideoMaev2 and CAST may be used, or learning models such as LSTM (Long Shor-Term Memory) and 3D-CNN (Convolutional Neural Network) may be used as the base model. The training data used when generating the learning model is not limited to the above example; it should be designed appropriately depending on the type of learning model used. Furthermore, the task classification model MD1 may be generated not only by tuning the base model, but also by training a model with initial parameters set from scratch.

[0056] Furthermore, although this embodiment shows a work classification model MD1 configured to output work classification results in response to the input of a work video, the work classification model MD1 may also be configured to take input of audio (sound data) recorded along with the work video, or text generated by speech recognition, and to output work classification results in response to the input of the work video and audio (or the work video and text).

[0057] The work classification model MD1 may be generated on the analysis server 20 or on an external server. The generated work classification model MD1 may be stored in the storage unit 22 of the analysis server 20 or in the storage unit of an external server accessible from the analysis server 20.

[0058] Figure 5 is an explanatory diagram illustrating the annotation method for work videos. Figure 5 shows an example where labels are assigned every two seconds. In this embodiment, accuracy of labels less than two seconds is not required, and the minimum unit for assigning labels as annotations is set to two seconds. In the example in Figure 5, if the first two-second interval is a scene of work A, the work video for that interval is labeled "Work A". If the next two-second interval includes scenes of both work A and work B, it is determined which scene accounts for the majority, and if it is determined that scenes of work B account for the majority, the label "Work B" is assigned. Similarly, labels corresponding to the worker's work content are assigned for every two-second interval. In this embodiment, labels corresponding to the work content include "Work Preparation", "Visual Inspection", "Measuring Instrument Inspection", "Disassembly", "Repair / Replacement", "Assembly Work", "Tool Cleanup", "Equipment Operation Check", and "Removal of Protective Coverings".

[0059] In this embodiment, in order to tune EgoVLPv2, training data was prepared by assigning labels to work videos every two seconds using the method illustrated in Figure 5. Due to the specifications of the base model, only six-second videos can be input as training data. Depending on how the video is cut, it is possible that two or more labels may be assigned to a six-second video, but in this embodiment, in order to assign one label to a six-second video, if two or more labels are included, the process was performed to round them down to the majority of the labels.

[0060] Through the above processing, a dataset is obtained that includes numerous videos (6-second videos) extracted from a series of work videos, and labels (text) assigned to each video. The work classification model MD1 according to this embodiment is generated by tuning a trained EgoVLPv2 using the obtained dataset as training data. Due to the switching capability of EgoVLPv2, the work classification model MD1 can be used for various tasks that require dual encoders and fusion encoders. The analysis server 20 according to this embodiment performs the task of recognizing work performed by an operator using the work classification model MD1. The analysis server 20 recognizes the work scene based on the results of work classification using the work classification model MD1.

[0061] Figure 6 is an explanatory diagram illustrating the scene recognition method using the work classification model MD1. In the operational phase after the work classification model MD1 is generated, the worker wears a wearable device 10, and the wearable device 10 captures images of the work performed by the worker. The analysis server 20 acquires the work video captured by the wearable device 10 in real time and inputs the work video in 6-second units (180 frames for 30fps video) into the work classification model MD1, thereby executing a work classification task by the work classification model MD1 for each 6-second unit of work video. As a result of executing the work classification task, the analysis server 20 acquires classification results including "work preparation," "visual confirmation," "instrument confirmation," "disassembly," "repair / replacement," "assembly work," "tool cleanup," "equipment operation confirmation," and "removal of protective coverings."

[0062] Based on the results of the work classification task, the analysis server 20 distinguishes and recognizes two types of scenes in a 6-second work video: work continuation scenes, which show the work in progress, and restoration scenes, which show the work being restored to its original state. Specifically, if the worker's work classified by the work classification task falls under any of the following categories: "work preparation," "visual inspection," "instrument inspection," "disassembly," or "repair / replacement," the analysis server 20 recognizes the scene in the work video as a work continuation scene. Conversely, if the worker's work classified by the work classification task falls under "assembly work," "tool cleanup," "equipment operation check," or "removal of protective coverings," the analysis server 20 recognizes the scene in the work video as a restoration scene.

[0063] The following describes the processes performed by the analysis server 20. Figure 7 is a flowchart illustrating the procedure for the processes performed by the analysis server 20 according to Embodiment 1. A worker who begins work at a work site attaches the wearable device 10 to their body. The wearable device 10 automatically starts imaging at an appropriate timing after being attached to the worker. Alternatively, the wearable device 10 starts imaging at a timing instructed by the worker or the analysis server 20. The wearable device 10 transmits the work video captured by the imaging unit 14 to the analysis server 20 via the communication unit 13. In this embodiment, the wearable device 10 transmits the work video to the analysis server 20 in real time during work.

[0064] The processing unit 21 of the analysis server 20 reads the analysis processing program PG1 from the storage unit 22 and executes it, and performs the analysis processing according to the following procedure.

[0065] The processing unit 21 acquires the work video transmitted from the wearable device 10 from the communication unit 23 (step S101). The processing unit 21 is assumed to be sequentially acquiring the work video that is continuously transmitted from the wearable device 10.

[0066] The processing unit 21 inputs the acquired work video into the work classification model MD1 in predetermined units (for example, 6-second units) and performs calculations (work classification tasks) using the work classification model MD1 (step S102). The processing unit 21 obtains the work classification result as a result of the calculations performed by the work classification model MD1 (step S103).

[0067] The processing unit 21 recognizes the work scene based on the classification result of the work classification model MD1 (step S104). In this embodiment, it is sufficient to distinguish between work continuation scenes and restoration scenes. The processing unit 21 recognizes a work continuation scene if the classification result of the work classification model MD1 is any of "work preparation," "visual inspection," "instrument inspection," "disassembly," and "repair / replacement," and recognizes a restoration scene if the classification result of the work classification model MD1 is any of "assembly work," "tool cleanup," "equipment operation check," and "removal of protective coverings."

[0068] The processing unit 21 determines whether the work scene recognized in step S104 is a scene for restoring the original state (step S105). If it is determined that it is not a scene for restoring the original state (S105: NO), the processing unit 21 returns to step S102 because the worker is continuing their work.

[0069] If the system determines that it is time to restore the device to its original state (S105: YES), the processing unit 21 determines that the worker has finished their work (step S106) and instructs the wearable device 10 to stop capturing the work video (step S107). The instruction to stop capturing the work video is transmitted from the communication unit 23 to the wearable device 10. Upon receiving the instruction to stop capturing from the analysis server 20, the wearable device 10 terminates the imaging by the imaging unit 14. The analysis server 20 may also instruct the wearable device 10 to notify the user that the imaging has ended, in addition to instructing the wearable device 10 to terminate the imaging. In this case, the information that the imaging has ended is output as audio from the sound output unit 16 and communicated to the worker. The analysis server 20 may also send a system termination instruction to the wearable device 10 instead of an imaging termination instruction.

[0070] As described above, the analysis server 20 according to Embodiment 1 determines that the worker has finished their work when it detects a scene that requires restoration to the original state based on the work video acquired from the wearable device 10. When the analysis server 20 determines that the worker has finished their work, it can give the wearable device 10 an instruction to stop imaging, thereby eliminating unnecessary video uploads due to forgetting to operate the device.

[0071] (Embodiment 2) Embodiment 2 describes a configuration in which a list of work items to be performed by the worker is prepared in advance, and whether or not the worker has completed their work is determined by comparing it with the work item list.

[0072] Figure 8 is a conceptual diagram showing an example of a work item list. A work item list is a list of work items to be performed by a worker, arranged in the order of work. For example, the work order, work content, and confirmation flag are recorded in association with each item. Here, the confirmation flag indicates whether or not the worker has performed the work for each work item. Figure 8 shows an example of a work list that starts with work preparation, proceeds in the order of visual inspection, dismantling work, repair / replacement work, and ends with assembly work. Also, by referring to the confirmation flag column, it can be seen that the work from work preparation to repair / replacement work was performed by the worker.

[0073] The work item list is created, for example, by the work manager's terminal. Before the worker starts the work, the analysis server 20 retrieves the work item list from the work manager's terminal and stores it in the storage unit 22.

[0074] After obtaining the work item list, the analysis server 20 performs the following processing. Figure 9 is a flowchart illustrating the procedure of processing performed by the analysis server 20 according to Embodiment 2. Following the same procedure as in Embodiment 1, the worker's work is captured by the wearable device 10, and the work video is transmitted to the analysis server 20.

[0075] The processing unit 21 acquires the work video transmitted from the wearable device 10 from the communication unit 23 (step S201). The processing unit 21 is assumed to be sequentially acquiring the work video that is continuously transmitted from the wearable device 10.

[0076] The processing unit 21 inputs the acquired work video into the work classification model MD1 in predetermined units (for example, 6-second units) and performs calculations (work classification task) using the work classification model MD1 (step S202). The processing unit 21 obtains the work classification result as a result of the calculations performed by the work classification model MD1 (step S203). Based on the classification result from the work classification model MD1, the processing unit 21 identifies the work performed by the worker and checks the confirmation flag in the work item list (step S204).

[0077] The processing unit 21 checks the confirmation flag column in the work item list to identify multiple work items performed by the worker, and determines whether the worker has completed their work by comparing them with the work items included in the work item list (step S205). In this embodiment, the processing unit 21 determines that the worker has completed their work if it can confirm from the confirmation flag that the work to restore the original state (assembly work in the work item list of Figure 8) has been performed by the worker.

[0078] If it is determined that the work is not yet complete (S205: NO), the processing unit 21 returns the process to step S202.

[0079] If the processing unit determines that the work is complete (S205: YES), it determines whether there are any missing tasks (step S206). If the processing unit determines that there are any missing tasks in the work item list even though it has determined that the work is complete, it determines that there are missing tasks.

[0080] If it is determined that there is a missing task (S206: YES), the processing unit 21 notifies the worker of the missing task (step S207). The processing unit 21 transmits information that there is a missing task to the wearable device 10 via the communication unit 23. At this time, the processing unit 21 may also transmit information about the missing task item to the wearable device 10. Based on the information transmitted from the analysis server 20, the wearable device 10 notifies the worker of the missing task by voice, for example, from the sound output unit 16.

[0081] If it is determined that there are no omissions in the process (S206: NO), the processing unit 21 terminates the process according to this flowchart.

[0082] If the work is being performed in a predetermined order, the processing unit 21 only needs to compare the work items with the work item list in the order defined in the work item list when comparing them in step S205. In this case, the processing unit 21 can determine whether any work has been missed each time it compares the work items. On the other hand, if the work is being performed in an order different from the predetermined order, the processing unit 21 may compare the work items with the work item list in any order when comparing them in step S205. In this case, the processing unit 21 only needs to determine whether any work has been missed after it has determined that the work has been completed.

[0083] As described above, in Embodiment 2, the completion of work and whether or not any work has been missed can be determined using a pre-prepared list of work items.

[0084] (Embodiment 3) Embodiment 3 describes a configuration in which the wearable device 10 makes a determination of the completion of work.

[0085] The memory unit 12 of the wearable device 10 stores the analysis processing program PG1 and the work classification model MD1 described in Embodiment 1. The processing unit 11 of the wearable device 10 reads the analysis processing program PG1 from the memory unit 12 and executes it, thereby performing the following processing.

[0086] Figure 10 is a flowchart illustrating the procedure of processing performed by the wearable device 10 according to Embodiment 3. In the same procedure as in Embodiment 1, the work performed by the worker is captured by the wearable device 10. The processing unit 11 of the wearable device 10 acquires the work video from the imaging unit 14 (step S301). The processing unit 11 is assumed to be sequentially acquiring the work video that is continuously output from the imaging unit 14.

[0087] The processing unit 11 inputs the acquired work video into the work classification model MD1 in predetermined units (for example, 6-second units) and performs calculations (work classification tasks) using the work classification model MD1 (step S302). The processing unit 11 obtains the work classification result as a result of the calculations performed by the work classification model MD1 (step S303).

[0088] The processing unit 11 recognizes the work situation based on the classification result of the work classification model MD1 (step S304). The processing unit 11 recognizes the work as a continuation situation if the classification result of the work classification model MD1 is any of "work preparation," "visual inspection," "instrument inspection," "disassembly," or "repair / replacement," and recognizes the work as a restoration situation if it is any of "assembly work," "tool cleanup," "equipment operation check," or "removal of protective coverings."

[0089] The processing unit 11 determines whether the work scene recognized in step S304 is a scene for restoring the original state (step S305). If it is determined that it is not a scene for restoring the original state (S305: NO), the processing unit 11 returns to step S302 because the worker is continuing their work.

[0090] If it is determined that the situation is one where the equipment needs to be restored to its original state (S305: YES), the processing unit 21 determines that the worker has finished their work (step S306) and terminates imaging by the imaging unit 14 (step S307). At this time, the processing unit 11 may notify the worker of the termination of imaging by outputting an audio message from the sound output unit 16. Alternatively, if the processing unit 11 receives an instruction to terminate imaging through the operation unit 18 without detecting a situation where the equipment needs to be restored to its original state in step 305, it may output an audio message from the sound output unit 16 asking whether the work to restore the equipment to its original state has been performed.

[0091] After imaging is complete, the processing unit 11 uploads the work video captured from the start to the end of the work to the server device (step S308). In Embodiment 3, since no analysis is performed by the analysis server 20, there is no need to upload the work video to the analysis server 20; it is sufficient to upload the work video to a predetermined server device that stores it as evidence.

[0092] As described above, the wearable device 10 according to Embodiment 3 determines that the worker has finished their work when it detects a scene that requires restoration to the original state based on the work video captured by the imaging unit 14. When the wearable device 10 determines that the worker has finished their work, it can upload all the work videos acquired from the start to the end of the work to the server device.

[0093] Furthermore, while the application example of the information processing system according to this embodiment is assumed to be a work scenario involving air conditioning equipment, it is of course applicable to a variety of work scenarios, not limited to air conditioning equipment, such as work scenarios involving elevators, or maintenance and inspection work scenarios in chemical plants and power supply facilities.

[0094] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended.

[0095] 10 Wearable device 20 Analysis server 21 Processing unit 22 Memory unit 23 Communication unit 24 Operation unit 25 Display unit PG1 Analysis processing program MD1 Work classification model

Claims

1. A computer program that causes at least one computer to execute a process to acquire work video obtained by imaging the work performed by a worker, to detect work scenes that require restoration to the original state based on the acquired work video, and to determine that the work is completed when such work scenes are detected.

2. The computer program according to claim 1, which causes the computer to perform a process of transmitting the work video to an external source when it determines that the work has been completed.

3. The computer program according to claim 1, which determines whether there are any missing tasks for each task based on the aforementioned work video, and, if it determines that there are missing tasks, causes the computer to execute a process to notify the computer of the missing tasks.

4. The computer program according to claim 1, wherein the work scene is a scene in which assembly work, tool cleanup, equipment operation check, or removal of protective coverings are being carried out.

5. The computer program according to claim 1, wherein the work video is a video obtained by capturing images of repair or inspection work on equipment.

6. The computer program according to claim 1, wherein the work video is a video captured by a wearable terminal worn by the worker.

7. The computer program according to claim 1, which causes the computer to execute a process to notify the computer to terminate imaging when the aforementioned work scene is detected.

8. The computer program according to claim 1, which, when the work scene is not detected and an instruction to terminate imaging is given, causes the computer to perform a process to output a notification asking whether or not the work to restore the original state has been performed.

9. A computer program according to claim 1, which causes a computer to perform a process of determining whether the work has been completed by identifying a plurality of work items performed by the worker based on the acquired work video, obtaining a list of a plurality of work items to be performed in relation to the work, and comparing the identified plurality of work items with the plurality of work items included in the acquired list.

10. The computer program according to claim 9, which causes the computer to perform a process of matching the specified work items with the acquired list in any order.

11. The computer program according to claim 9, which determines whether there are any missing tasks by comparing the work items, and if it is determined that there are missing tasks, causes the computer to execute a process to notify the computer of the missing tasks.

12. An information processing method that, by imaging the work performed by an worker, acquires a work video obtained from that video, detects a work scene that needs to be restored to its original state based on the acquired work video, and, if such a work scene is detected, determines that the work has been completed, and performs this process using at least one computer.

13. An information processing system comprising at least one processing unit, wherein the processing unit acquires a work video obtained by imaging the work performed by an operator, detects a work scene to be restored to its original state based on the acquired work video, and determines that the work has been completed when the work scene is detected.

Images