PROGRAM, INFORMATION PROCESSING APPARATUS, AND METHOD

Abstract

The system captures images of the worker's work status, analyzes the captured images (video), generates display data for displaying the worker's work status, and outputs the results. [Solution] The server 20 of the work analysis system 1 has the following functions: a step (S101) of acquiring video data capturing a situation in which a worker is performing a specified task; a step (S102) of analyzing the acquired video data and determining the start and end times of a work step that indicates a process when the specified task performed by the worker captured in the acquired video data is divided into multiple processes; a step (S103) of dividing the acquired video data into work steps identified by the start and end times based on the results of the determination, and generating display data for displaying the work status of each worker for each work step and for each worker; and a step (S105) of outputting the generated display data.

Description

[Technical field]

[0001] The present disclosure relates to a program, an information processing device, and a method. [Background technology]

[0002] In order to improve the quality of products in factories and the work efficiency of workers, the working conditions of workers are filmed and the filmed videos are analyzed. In addition, AI (artificial intelligence) technology is sometimes applied to this analysis.

[0003] Patent Document 1 discloses a technique for a work support system that supports workers working on site, in which the posture of a worker is analyzed from a captured image of the worker, and information according to the worker's condition is provided. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-144776 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, in conventional work analysis, captured images (videos) were analyzed to detect abnormalities in work (omissions of tasks, procedural errors, fraud, etc.). However, even if there was no abnormality in the work, a person would still have to look and judge what areas needed improvement when making improvements to the work process, which took time and resulted in inconsistent judgments.

[0006] Therefore, this disclosure describes a technology that makes it possible to capture images of a worker's work status, analyze the captured images (videos), generate display data for displaying the worker's work status, and output the results in order to collect data for process improvement of work. [Means for solving the problem]

[0007] According to an embodiment of the present disclosure, there is provided a program for analyzing a work status of a worker, the program being executed by a computer including a processor and a memory. The program causes the processor to execute the steps of acquiring video data capturing a situation in which a worker is performing a predetermined work, analyzing the acquired video data, and determining start times and end times of work steps that indicate a process when the predetermined work performed by the worker captured in the acquired video data is divided into a plurality of processes, dividing the acquired video data into work steps identified by start times and end times based on the determination results, generating display data for displaying the work status of each worker for each work step and for each worker, and outputting the generated display data. Effect of the Invention

[0008] According to the present disclosure, video data of a worker performing a specific task is divided into task steps identified by the start and end times of the task process and output. This makes it possible to generate display data for displaying the task status and output the results. This makes it possible to obtain basic data for considering what improvements can be made to the task process. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an overall configuration of a work analysis system 1 according to an embodiment of the present disclosure. [Diagram 2] 2 is a block diagram showing a functional configuration of the terminal device 10 of FIG. 1. [Diagram 3] 2 is a block diagram showing a functional configuration of a server 20 in FIG. 1. [Figure 4] FIG. 4 is a diagram showing an example of the data structure of a video database 2021 in FIG. 3. [Diagram 5] 4 is a diagram showing an example of the data structure of a display database 2022 in FIG. 3. [Figure 6] 10 is a flowchart showing an example of a flow of display data output processing performed by the work analysis system 1. [Figure 7] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 8] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 9] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 10] FIG. 11 is a block diagram showing a functional configuration of a server 20 according to a second embodiment of the present disclosure. [Figure 11] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 12] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 13] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 14] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 15] FIG. 2 is a diagram showing an example of a screen of display data displayed on the terminal device 10. [Figure 16] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. In all the drawings explaining the embodiments, the same reference numerals are given to the common components, and the repeated explanation is omitted. Note that the following embodiments do not unduly limit the contents of the present disclosure described in the claims. In addition, not all of the components shown in the embodiments are essential components of the present disclosure. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.

[0011] In the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). The at least one processor may be a single-core or multi-core.

[0012] Furthermore, the at least one processor may be a processor in the broad sense, such as a hardware circuit (for example, a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) that performs part or all of the processing.

[0013] In the following explanation, information that produces an output for an input may be described using an expression such as "xxx database," but this information may be data of any structure, or may be a learning model such as a neural network that produces an output for an input. Therefore, "xxx database" may be called "xxx information."

[0014] Furthermore, in the following description, the configuration of the tables that make up each database is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0015] In addition, in the following explanation, the processing may be described with the "program" as the subject, but since the program is executed by a processor to perform a specified processing step by appropriately using a memory unit and / or an interface unit, etc., the subject of the processing may be the processor (or a device such as a controller having the processor).

[0016] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] In the following description, identification numbers are used as identification information for various objects, but other types of identification information (for example, identifiers including alphabetic characters or symbols) may be used instead.

[0018] In addition, in the following description, when describing elements of the same type without distinguishing between them, reference signs (or common signs among the reference signs) may be used, and when describing elements of the same type with distinction between them, the identification numbers (or reference signs) of the elements may be used.

[0019] In the following description, the control lines and information lines are those that are considered necessary for the description, and not all control lines and information lines in the product are necessarily shown. All components may be connected to each other.

[0020] <Summary> The work analysis system according to the present disclosure will be described below. The work analysis system according to the present disclosure is a system used to analyze the work status of workers performing work such as manufacturing products at a work site such as a factory. This work analysis system captures the work status of workers at a work site such as a factory, acquires and analyzes the captured video data, and generates and outputs display data for displaying the work status for each work step and each worker. The work analysis system according to the present disclosure is a system provided by a so-called SaaS (Software as a Service) as a Web service, for example, by a cloud server, and is configured to be accessible by users through predetermined authentication.

[0021] Incidentally, in the above-mentioned worker work analysis method, photographed images (videos) are analyzed, the work time that constitutes each work unit is measured, and as a result, abnormalities in the work (omission of work, procedural errors, fraud, etc.) are detected. However, even if the photographed images (videos) are analyzed and no abnormalities are found in the work, a person still has to look at the photographed images to determine what improvements can be made to the process for that work. This makes it difficult for workers to intuitively understand how to correct the work, and human judgment is required to investigate the cause of long work times and the cause of the work being judged as abnormal by AI, which takes time and causes inconsistency in judgment.

[0022] Furthermore, when analyzing the work of workers in factories, etc., it is also necessary to analyze work on a time axis. However, because the objective is to shorten the time per work cycle (cycle time) and improve work quality, it is essential to extract the areas and time periods that have a particularly large impact on work delays, and traditional methods of analyzing worker work were not suitable.

[0023] Therefore, the work analysis system disclosed herein is configured to obtain video data capturing a situation in which a worker is performing a specified task, analyze the video data, determine the start time and end time of a task step that indicates a process when the specified task performed by the worker is divided into multiple processes, divide the video data into each task step identified by the start time and end time, generate display data, and output the display data.

[0024] In addition, the display data can be output in a variety of ways, such as by outputting it simultaneously with the display data of other workers, by outputting it superimposed on the display data of other workers, or by extracting and outputting the skeletal data of the worker.

[0025] With this configuration, by using the work analysis system according to the present disclosure, it is possible to obtain basic data for considering what improvements are necessary when making process improvements to work. In addition, when making process improvements to work, it is possible to present the user with training hints and recommendations for workers. This makes the results of the work analysis visible, making it possible to provide objective material for reflecting on subsequent process improvements and training.

[0026] <First embodiment> The work analysis system 1 according to an embodiment of the present disclosure will be described below. In the following description, for example, when the terminal device 10 accesses the server 20, the server 20 responds with information for generating a screen on the terminal device 10. The terminal device 10 generates and displays a screen based on the information received from the server 20.

[0027] <1 Overall configuration of work analysis system 1> FIG. 1 is a block diagram showing an overall configuration of a work analysis system 1 according to a first embodiment of the present disclosure. As shown in FIG. 1, the work analysis system 1 includes a plurality of terminal devices (terminal device 10A and terminal device 10B are shown in FIG. 1. Hereinafter, they may be collectively referred to as "terminal device 10") and a server 20. The terminal devices 10 and the server 20 are connected to each other via a network 80 so as to be able to communicate with each other. The network 80 is configured as a wired or wireless network. In this embodiment, the server 20 is a web server (including a cloud server) and exchanges information with the terminal device 10 through web pages. In addition, a web page browser for viewing web pages is installed on the terminal device 10, but a dedicated application for providing the services of the server 20 may also be installed and configured to be viewable through the dedicated application.

[0028] The terminal device 10 is a device operated by each user. Here, a user is a person who uses the terminal device 10 to analyze the work status of a worker, which is a function of the work analysis system 1. The terminal device 10 is realized by a stationary PC (Personal Computer), a laptop PC (notebook PC), or the like. In addition, the terminal device 10 may be, for example, a tablet compatible with a mobile communication system, a mobile terminal such as a smartphone, or the like.

[0029] The terminal device 10 is communicatively connected to the server 20 via a network 80. The terminal device 10 is connected to the network 80 by communicating with communication devices such as a wireless base station 81 compatible with communication standards such as 4G, 5G, and LTE (Long Term Evolution), and a wireless LAN router 82 compatible with wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. As shown as the terminal device 10B in FIG. 1, the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage unit 16, and a processor 19.

[0030] The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with an external device. The input device 13 is an input device (e.g., a keyboard, a touch panel, a touch pad, a pointing device such as a mouse, etc.) for receiving an input operation from a user. The output device 14 is an output device (a display, a speaker, etc.) for presenting information to a user. The memory 15 is for temporarily storing a program and data processed by the program, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for saving data, and is, for example, a flash memory or a HDD (Hard Disc Drive). The processor 19 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0031] The server 20 is a device that analyzes the work status of a worker at a work site such as a factory and outputs the analysis results. The server 20 acquires video data that captures the situation in which a worker is performing a specified task, and analyzes the video data. From the analysis results of the video data, the server 20 determines the start time and end time of a task step that indicates a process when the specified task performed by the worker is divided into multiple processes, and divides the video data into each task step identified by the start time and end time to generate display data. Furthermore, the server 20 outputs the generated display data.

[0032] The server 20 is a computer connected to a network 80. The server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29.

[0033] The communication IF 22 is an interface for inputting and outputting signals so that the server 20 can communicate with an external device. The input / output IF 23 functions as an interface with an input device for receiving an input operation from a user and an output device for presenting information to a user. The memory 25 is for temporarily storing programs and data processed by the programs, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for saving data, such as a flash memory or a HDD (Hard Disc Drive). The processor 29 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0034] <1.1 Configuration of the terminal device 10> FIG. 2 is a block diagram showing a functional configuration of the terminal device 10 constituting the work analysis system 1 of the first embodiment. As shown in FIG. 2, the terminal device 10 includes a plurality of antennas (antenna 111, antenna 112), wireless communication units (first wireless communication unit 121, second wireless communication unit 122) corresponding to each antenna, an operation reception unit 130 (including a keyboard 131 and a mouse 132), a display 140, a storage unit 150, and a control unit 160. The terminal device 10 also has functions and configurations (e.g., a battery for storing power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc.) that are not particularly shown in FIG. 2. As shown in FIG. 2, each block included in the terminal device 10 is electrically connected by a bus or the like.

[0035] The antenna 111 emits a signal generated by the terminal device 10 as a radio wave. The antenna 111 also receives a radio wave from space and provides the received signal to the first wireless communication unit 121.

[0036] The antenna 112 radiates a signal generated by the terminal device 10 as a radio wave. The antenna 112 also receives the radio wave from space and provides the received signal to the second radio communication unit 122.

[0037] The first wireless communication unit 121 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 111 so that the terminal device 10 can communicate with other wireless devices. The second wireless communication unit 122 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 112 so that the terminal device 10 can communicate with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, a Received Signal Strength Indicator (RSSI) calculation circuit, a Cyclic Redundancy Check (CRC) calculation circuit, a high-frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation / demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10, and provide the received signals to the control unit 160.

[0038] The operation reception unit 130 has a mechanism for receiving input operations from a user. Specifically, the operation reception unit 130 includes a keyboard 131 and a mouse 132. Note that the operation reception unit 130 may be configured as a touch screen that detects the user's contact position on the touch panel, for example, by using a capacitive touch panel.

[0039] The keyboard 131 accepts input operations by the user of the terminal device 10. The keyboard 131 is a device for inputting characters, and outputs input character information to the control unit 160 as an input signal.

[0040] The mouse 132 accepts input operations by the user of the terminal device 10. The mouse 132 is a pointing device for selecting an object displayed on the display 140, and outputs position information of an object selected on the screen and information indicating that a button has been pressed to the control unit 160 as input signals.

[0041] Display 140 displays data such as images, videos, and text under the control of control unit 160. Display 140 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0042] Storage unit 150 is configured with memory 15, such as a flash memory, and storage unit 16, and stores data and programs used by terminal device 10. In one aspect, storage unit 150 stores user information 151.

[0043] The user information 151 is information about a user who analyzes the work status of a worker, which is a function of the work analysis system 1 using the terminal device 10. The user information may include information for identifying the user (user ID), the user's name, address, and organizational information such as the company to which the user belongs.

[0044] The control unit 160 is constituted by, for example, the processor 19, and controls the operation of the terminal device 10 by reading a program stored in the storage unit 150 and executing instructions included in the program. The control unit 160 is, for example, an application that is pre-installed in the terminal device 10. The control unit 160 operates according to the program to fulfill the functions of an input operation reception unit 161, a transmission / reception unit 162, a data processing unit 163, and a notification control unit 164.

[0045] The input operation receiving unit 161 performs processing for receiving input operations by a user via an input device such as the keyboard 131 .

[0046] The transmission / reception unit 162 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol.

[0047] The data processing unit 163 performs a process of performing calculations on the data that the terminal device 10 accepts as input in accordance with a program, and outputs the calculation results to a memory or the like.

[0048] The notification control unit 164 performs a process of presenting information to the user. The notification control unit 164 performs a process of displaying a display image on the display 140, and the like.

[0049] <1.2 Functional configuration of server 20> 3 is a diagram showing the functional configuration of the server 20 constituting the work analysis system 1 of the embodiment 1. As shown in FIG. 3, the server 20 fulfills the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0050] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0051] The storage unit 202 stores data and programs used by the server 20. The storage unit 202 stores a video database 2021, a display database 2022, and the like.

[0052] The video database 2021 is a database for acquiring and storing video data of a worker performing a specific task in order to utilize the work analysis system 1. The video database 2021 is data including the results of determining the start time and end time of a task step that indicates a process when a specific task performed by a worker is divided into a plurality of processes. Details will be described later.

[0053] The display database 2022 is a database for registering and holding display data for displaying the work status of each worker, which is generated by dividing video data based on the results of determining the start time and end time of a work step using the work analysis system 1. Details will be described later.

[0054] The control unit 203 performs functions indicated by various modules, such as a receiving control module 2031, a transmitting control module 2032, a video data acquisition module 2033, a work step determination module 2034, a video data division module 2035, a display data generation module 2036, and a display data output module 2037, by the processor 29 of the server 20 performing processing according to a program.

[0055] The reception control module 2031 controls the process in which the server 20 receives a signal from an external device in accordance with a communication protocol.

[0056] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0057] The video data acquisition module 2033 controls a process of acquiring video data capturing a situation in which a worker is performing a predetermined task at a work site such as a factory. The video data acquisition module 2033 acquires video data captured by a camera capturing a situation in which a worker is performing a predetermined task, such as assembly work of a product at a work site such as a factory, or various processing tasks. The video data acquired by the video data acquisition module 2033 may be video data captured by a camera provided in the terminal device 10 or the server 20 (not shown), or may be video data captured by an external camera.

[0058] For example, when acquiring video data from a camera included in the terminal device 10 or an external camera, the video data acquisition module 2033 accepts the video data transmitted from the terminal device 10 or the external camera by receiving the video data via the communication unit 201. Also, when acquiring video data from a camera included in the server 20, the video data acquisition module 2033 accepts the video data as a function of the control unit 203. The format of the video data acquired by the video data acquisition module 2033 may be any data format, such as MP4, MOV, WMV, AVI, or AVCHD.

[0059] The video data acquisition module 2033 registers and stores the acquired video data in, for example, the video database 2021.

[0060] The work step determination module 2034 analyzes the video data acquired by the video data acquisition module 2033, and controls the process of determining the start time and end time of a work step indicating a process when a predetermined work by a worker captured in the acquired video data is divided into a plurality of processes. The plurality of work steps into which the work step determination module 2034 divides a predetermined work are specifically work steps such as taking out a tool, taking out an object, assembling parts, and placing the assembled object in a predetermined place in the assembly work of a product in a factory or the like.

[0061] The task step determination module 2034 analyzes the video data, and extracts feature values ​​from the video data for a specific task performed by a worker, for example. The task step determination module 2034 compares the feature values ​​with those defined for each of the multiple task steps to be classified, and determines whether the task performed by the worker corresponds to the defined task step. The task step determination module 2034 then determines the start time and end time of the task step for the video data determined to correspond to a specific task step. At this time, the task step determination module 2034 may determine that the extracted feature values ​​of the video data correspond to the defined task step if they are equal to or greater than a threshold value for the feature values ​​defined for each of the multiple task steps to be classified, or may calculate a degree of agreement by a specified calculation and determine that the extracted feature values ​​correspond to the defined task step if they are equal to or greater than a specified value.

[0062] The task step determination module 2034 may make a determination using a determination model that determines whether or not the task corresponds to a task step, which has been learned based on the video data acquired by the video data acquisition module 2033. The task step determination module 2034 may make a determination using a determination model that has been learned based on video data using any machine learning algorithm (supervised machine learning, etc.), a deep learning algorithm, a neural network (e.g., a deep neural network (DNN)) model, etc. The determination model used by the task step determination module 2034 does not need to be a single learning model, and may be realized by switching between multiple independent learning models for each task step.

[0063] The task step determination module 2034 registers and stores the start time and end time of the determined task step in, for example, the video database 2021.

[0064] The video data division module 2035 controls the process of dividing the video data acquired by the video data acquisition module 2033 into task steps identified by start times and end times based on the result of the determination by the task step determination module 2034. The video data division module 2035 divides the video data, for which the start times and end times of the task steps have been determined, into chapters at the determined start times and end times. Then, the video data division module 2035 links each piece of video data after chapter division to the name of the task step determined by the task step determination module 2034.

[0065] When dividing video data into chapters at the start time and end time, the video data division module 2035 may divide the video data at a time a predetermined time (e.g., about 5 seconds) earlier than the start time and a predetermined time (e.g., about 5 seconds) later than the end time. When dividing the video data into work steps, the video data division module 2035 may divide the video data so that the switching points of the work steps overlap by a predetermined time (e.g., about 5 seconds). This is to prevent scenes from being unnaturally cut out when dividing video data into chapters for each work step.

[0066] The video data division module 2035 registers and stores the divided video data in, for example, the display database 2022.

[0067] The display data generation module 2036 controls the process of generating display data for displaying the work status of each worker for each work step and for each worker from the video data divided for each work step. The display data generation module 2036 performs predetermined editing and processing on the video data divided for each work step, and generates display data to be used as basic data for considering improvement points when making process improvements to the filmed work.

[0068] For example, the display data generation module 2036 may set the name of the work step determined by the work step determination module 2034 as a title for the video data divided into work steps, or may set the name of the work step as a thumbnail image of the divided video data. Furthermore, the display data generation module 2036 may edit the video data divided into work steps so as to adjust the playback speed to match the video data of the reference work step. The video data of the reference work step is, for example, video data that is a sample of the work step, and may be video data of a work situation performed by a veteran worker that serves as a sample, or may be video data automatically generated by a generation AI or the like.

[0069] The display data generation module 2036 registers and stores the generated display data in, for example, the display database 2022.

[0070] The display data output module 2037 controls the process of outputting the display data generated by the display data generation module 2036. The display data output module 2037 outputs the display data and provides it to a user, for example, as basic data for process improvement of the photographed work step. The display data output module 2037 transmits the display data to be output to the terminal device 10 used by the user via the communication unit 201, and causes the display 140 to display the display data.

[0071] The display data output module 2037 may output display data for each work step for each worker as a list and display it on the display 140 of the terminal device 10. At this time, the display data may be output for each worker, or may be output for each of a plurality of workers (for example, for each specific department or group). The display data output module 2037 may accept a user's selection from the display data for each work step output as a list, and output (play) the display data related to the selected work step. When outputting (playing) the display data related to the selected work step, the display data output module 2037 may output (play) the display data of the selected worker and the display data of the other workers simultaneously, or may output (play) them superimposed on each other. Here, the display data of the other workers is display data on the work situation of a worker to be compared, and may be display data of the work situation of a veteran worker as a model, or may be display data automatically generated by a generation AI or the like.

[0072] <2 Data Structure> Fig. 4 is a diagram showing an example of the data structure of the video database 2021 in Fig. 3. Fig. 5 is a diagram showing an example of the data structure of the display database 2022 in Fig. 3.

[0073] As shown in FIG. 4, each record of the video database 2021 includes an item "Worker ID", an item "Worker", an item "Shooting date and time", an item "Step division time", and the like.

[0074] The item “worker ID” is information for identifying each worker who is performing a specific task and who is captured in video data and is the subject of analysis by the work analysis system 1.

[0075] The item "Worker" is the name, etc. of a worker who is performing a specific task that is captured in video data and is the subject of analysis by the work analysis system 1. In addition to the name of the worker, the item "Worker" may also store organizational information, such as the company to which the worker belongs.

[0076] The item “shooting date and time” is information that indicates the date and time when the video data that is the subject of analysis by the work analysis system 1 was shot.

[0077] The item "Step division time" is information indicating the results of determining the start time and end time of a work step for the video data that is the subject of analysis by the work analysis system 1, and specifically includes the items "Step ID", "Start time", and "End time", etc.

[0078] The item “step ID” is information for identifying each work step that is the result of judgment on video data that is the subject of analysis by the work analysis system 1.

[0079] The item “start time” is information indicating the start time of a task step that is the result of determination made on the video data that is the subject of analysis by the work analysis system 1.

[0080] The item “end time” is information indicating the end time of a task step that is the result of determination made on the video data that is the subject of analysis by the work analysis system 1.

[0081] The video data acquisition module 2033 of the server 20 adds a record to the video database 2021 as the video data is acquired. In addition, the task step determination module 2034 adds a record to the item "step division time" of the video database 2021 as the task step start time and end time of the video data are determined.

[0082] As shown in FIG. 5, each record of the display database 2022 includes an item "step ID", an item "step name", an item "display data details", and the like.

[0083] The item “Step ID” is information that identifies each work step that is the result determined for the video data that is the subject of analysis by the work analysis system 1, and corresponds to the item “Step ID” in the video database 2021.

[0084] The item "Step Name" is information indicating the name of the work step that is the result determined for the video data that is the subject of analysis by the work analysis system 1, and for example, a name such as "△○ process", which is the name of a specific work step, is stored.

[0085] The item "display data details" is information about the display data generated from video data that is the subject of analysis by the work analysis system 1, and specifically includes the items "worker ID" and "display data", etc.

[0086] The item “Worker ID” is information that identifies each worker performing a specific task who is captured in the video data that is the subject of analysis by the work analysis system 1, and corresponds to the item “Worker ID” in the video database 2021.

[0087] The item "display data" is display data generated from video data that is the subject of analysis by the work analysis system 1. The item "display data" may store not the display data itself, but the location of the server 20 where the display data is stored, or link information that allows access to other web servers, etc.

[0088] The video data division module 2035 of the server 20 adds a record to the display database 2022 as the video data is divided. Also, the display data generation module 2036 adds a record to the item “display data details” of the display database 2022 as the display data is generated.

[0089] <3 operations> Hereinafter, a display data output (method) by the work analysis system 1 according to the embodiment of the present disclosure will be described with reference to FIG.

[0090] FIG. 6 is a flowchart showing an example of the flow of the display data output process performed by the work analysis system 1.

[0091] In step S101, the video data acquisition module 2033 of the server 20 acquires video data capturing a situation in which a worker is performing a predetermined task at a work site such as a factory. In step S101, the video data captured by a camera included in the terminal device 10 or the server 20 may be acquired, or video data captured by an external camera may be acquired.

[0092] In step S102, the work step determination module 2034 of the server 20 analyzes the video data acquired in step S101, and determines the start and end times of work steps that indicate a process when a predetermined work by a worker captured in the acquired video data is divided into a plurality of processes. In step S102, for example, the video data is analyzed, and for a predetermined work by a worker, feature amounts of the video data are extracted and compared with feature amounts defined for each work step, and it is determined whether or not the work falls into a defined work step, and for video data determined to fall into a specific work step, the start and end times of the work step are determined.

[0093] In step S103, the video data division module 2035 of the server 20 divides the video data acquired in step S101 into work steps identified by start times and end times based on the result of the determination in step S102. In step S103, for the video data whose start times and end times of the work steps have been determined, chapter division is performed at the determined start times and end times, and each piece of divided video data is linked to the name of the work step.

[0094] In step S104, the display data generation module 2036 of the server 20 generates display data for displaying the work status of each worker for each work step and for each worker from the video data divided for each work step in step S103. In step S104, the video data divided for each work step is edited and processed as required to generate display data to be used as basic data for considering improvement points when making process improvements to the filmed work.

[0095] In step S105, the display data output module 2037 of the server 20 outputs the display data generated in step S104. In step S105, the display data to be output is transmitted to the terminal device 10 used by the user via the communication unit 201, and is displayed on the display 140.

[0096] As described above, the work analysis system 1 acquires video data of a worker performing a specific task at a work site such as a factory, analyzes the video data, and determines the start and end times of a task step that indicates a process when the specific task performed by the worker captured in the acquired video data is divided into multiple processes. Based on the determination result, the video data is divided into each task step identified by the start and end times to generate display data, and the display data is output. As a result, by using the work analysis system according to the present disclosure, it is possible to acquire display data that serves as basic data for considering what improvements can be made when improving the process of a task.

[0097] <4 Screen example> Hereinafter, examples of screens in the display data output process by the work analysis system 1 will be described with reference to FIGS.

[0098] Fig. 7 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 7 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0099] 7, a display data screen 1411 is displayed on the display 140 of the terminal device 10. This display data screen 1411 displays display data divided from video data captured of a predetermined task performed by a worker. The display data screen 1411 displays, for example, a worker 1412, and displays a work target 1414 and a tool 1415 placed on a workbench 1413 on which the worker 1412 performs the predetermined task. By referring to such a display data screen 1411, the user can consider what improvements are necessary when making process improvements to the task.

[0100] Fig. 8 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 8 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0101] As shown in FIG. 8, a display data screen 1421 is displayed on the display 140 of the terminal device 10. The display data screen 1421 displays a list display field 1422 of display data divided from video data captured of a predetermined task performed by a worker, and display data 1423 captured of the predetermined task performed by the worker. The display data list display field 1422 displays, for example, thumbnail images 1424 of display data for each task step as a list for each task step. The display data 1423 displays, for example, a worker, and displays a state in which the worker is performing a predetermined task. When the user selects one of the thumbnail images 1424 while the display data screen 1421 is displayed, the corresponding display data 1423 is output (played). By referring to such a display data screen 1421, the user can consider what improvements are necessary when making process improvements to the task.

[0102] Fig. 9 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 9 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0103] As shown in FIG. 9, a display data screen 1431 is displayed on the display 140 of the terminal device 10. In this display data screen 1431, a list display field 1432 of display data divided from video data captured of a predetermined task by a worker, display data 1433 captured of a predetermined task by one worker, and display data 1434 captured of a predetermined task by another worker are displayed. In the display data list display field 1432, for example, thumbnail images 1435 of display data for each task step are displayed as a list for each task step. In the display data 1433 and 1434, for example, a worker is displayed, and a state in which the worker is performing a predetermined task is displayed. When the user selects one of the thumbnail images 1435 while the display data screen 1431 is displayed, the display data 1433 of the corresponding one worker is output (played), and the display data 1434 of the corresponding other worker is output (played) at the same time. By referring to such a display data screen 1421, the user can consider what improvements can be made when making process improvements to the work.

[0104] 9 shows a configuration in which display data 1433 of a selected worker and display data 1434 of other workers are simultaneously output (played) from among thumbnail images 1435, but the present invention is not limited to this example. For example, the display data 1433 of a selected worker and the display data 1434 of other workers may be output (played) in a superimposed state.

[0105] <Summary> As described above, according to the present embodiment, video data of a worker performing a specific task at a work site such as a factory is acquired, the video data is analyzed, and the start and end times of task steps that indicate the task when the specific task performed by the worker captured in the acquired video data is divided into multiple steps are determined. Based on the determination result, the video data is divided into task steps identified by the start and end times, and display data is generated and output. In this way, by using the work analysis system according to the present disclosure, it is possible to acquire display data that serves as basic data for considering what improvements can be made when improving the process of a task.

[0106] Moreover, as display data, display data divided from video data captured of a specific task performed by a worker is output (played back). At this time, display data for each task step is displayed as a list, and the corresponding display data is displayed upon selection by the user. Also, display data captured of a specific task performed by one worker and display data captured of a specific task performed by another worker are displayed simultaneously or superimposed. This makes it possible to visually grasp the situation while comparing the work status of a worker with that of other workers when making process improvements to the task.

[0107] <Second embodiment> Other embodiments of the work analysis system 1 will be described below.

[0108] <1 Overall configuration of work analysis system 1> Fig. 10 is a diagram showing a functional configuration of the server 20 according to the second embodiment of the present disclosure. The overall configuration of the work analysis system 1 and the configuration of the terminal device 10 in the second embodiment are similar to those in the first embodiment, and therefore will not be described repeatedly. The configuration of the server 20 is similar to that in the first embodiment, except that it is newly provided with the function of a skeleton data extraction module 2038, as shown in Fig. 10. The function of the skeleton data extraction module 2038 in the second embodiment will be described below.

[0109] The skeletal data extraction module 2038 analyzes the video data acquired by the video data acquisition module 2033, estimates the positions of the worker's joints captured in the acquired video data, and controls the process of extracting the worker's skeletal data. The skeletal data extraction module 2038 analyzes the video data of a predetermined task performed by the worker, and, for example, identifies and extracts body parts of the worker's body movements using a known image recognition technology. The skeletal data extraction module 2038 connects the identified points indicating the positions of each body joint with a straight line, and uses the information obtained by the points and the line as skeletal information. The points are mainly arranged at the joints of the body, and are configured to change the angle with the straight line part according to the body movement. This configuration makes it possible to express the body movement with points and straight lines.

[0110] The display data generation module 2036 according to the second embodiment may generate display data for displaying the work status of each worker based on the skeleton data extracted by the skeleton data extraction module 2038. The display data generation module 2036 performs predetermined editing and processing on the video data divided into each work step, and generates display data by expressing the body movements of the worker using skeleton information obtained by points and lines.

[0111] Furthermore, the display data generation module 2036 may generate, as display data, data indicating characteristic amounts of the joint movements of the worker, contained in the skeletal data of the worker, based on the skeletal data extracted by the skeletal data extraction module 2038. The display data generation module 2036 generates, as display data, data indicating characteristic amounts of the joint movements of the worker, for example, data including one or more of the trajectory, coordinates, speed, moving distance, operating time, operating rate, and degree of influence on the required time of each joint of the worker. The display data generation module 2036 analyzes the skeletal data to obtain this data.

[0112] The display data output module 2037 according to the second embodiment may output display data based on skeletal data as the display data generated by the display data generation module 2036. The display data output module 2037 outputs, for example, the body movement of the worker as display data expressed by skeletal information obtained from points and lines. At this time, the display data output module 2037 may output, for example, display data indicating the movement of some of the joints of the worker.

[0113] The display data output module 2037 may output data indicating the characteristic amount of the joint movement of the worker, which is included in the skeleton data of the worker, as display data. The display data output module 2037 outputs, as the display data, data including one or more of the trajectory, coordinates, speed, moving distance, working time, working rate, and degree of influence on the required time of the work step, for each joint of the worker, as the data indicating the characteristic amount of the joint movement of the worker. The display data output module 2037 may output, as the display data indicating the characteristic amount of the joint movement of the worker, data indicating the characteristic amount of the joint movement of a specific joint of the worker in a form different from the data indicating the characteristic amount of the joint movement of the other joints. In this case, the data indicating the characteristic amount of the joint movement of the specific joint of the worker to be output may be data indicating the characteristic amount of the movement of a part that is affecting the work step movement of the worker (for example, causing a delay), or may be data indicating the characteristic amount of the movement of a part that has a large difference from the characteristic amount of the movement of a reference work step.

[0114] The display data output module 2037 may output display data showing the joint movements of the first worker based on the skeletal data of the first worker, and display data showing the joint movements of the other workers based on the skeletal data of the other workers. At this time, the display data output module 2037 may output (play) the skeletal data of the first worker and the skeletal data of the other workers simultaneously, or may output (play) them superimposed on each other. Here, the skeletal data of the other workers is skeletal data on the working situation of the worker to be compared, as described above, and may be skeletal data on the working situation of a veteran worker as a model, or may be skeletal data automatically generated by a generation AI or the like. At this time, the display data output module 2037 may output data showing the difference between data showing the feature amount of the joint movements based on the skeletal data of the first worker and data showing the feature amount of the joint movements based on the skeletal data of the other workers, and may output display data in which the difference deviates by a predetermined value or more. Here, the difference (showing the feature amount) of the joint movements is a concept including either a spatial difference or a temporal difference, or both. Specifically, the spatial difference is the difference in spatial position when comparing the movement based on the skeletal data of one worker with the movement based on the skeletal data of another worker, and is caused by, for example, a difference in the speed of the movement of the relevant part or a difference in the distance traveled. Also, the temporal difference is the difference on the time axis when comparing the movement based on the skeletal data of one worker with the movement based on the skeletal data of another worker, and is caused by, for example, a difference in the speed of the movement of the relevant part or a difference in working time.

[0115] <2 Data Structure> The data structure in the second embodiment is similar to that in the first embodiment, so it will not be described repeatedly.

[0116] <3 operations> The operation of the second embodiment is similar to that of the first embodiment, and therefore will not be described repeatedly.

[0117] <4 Screen example> Hereinafter, examples of screens in the display data output process by the work analysis system 1 will be described with reference to FIGS.

[0118] Fig. 11 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 11 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0119] As shown in FIG. 11, a display data screen 1441 is displayed on the display 140 of the terminal device 10. In this display data screen 1441, skeletal data extracted by dividing video data captured during a specific task performed by a worker is displayed as display data. For example, skeletal data 1442 of a worker is displayed on the display data screen 1441. The skeletal data 1442 represents the body of the worker as a skeleton, and is composed of points 1443 indicating the positions of the joints of the body of the worker and straight lines 1444 connecting the points 1443. In addition, for example, a trajectory 1445 of each joint of the worker is displayed on the display data screen 1441. Note that the work target object 1414 and the tool 1415 shown in FIG. 7 may also be displayed on the display data screen 1441 (not shown). By referring to such a display data screen 1441, a user can consider what improvements are required when making process improvements to a task.

[0120] Fig. 12 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 12 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0121] The diagram shown in FIG. 12 is a diagram showing a state in which skeleton data 1446 of another worker (for example, a reference worker, a model worker, etc.) is displayed superimposed on skeleton data 1442 of one worker displayed on the display data screen 1441 shown in FIG. 11. For example, the difference is shown by superimposing the skeleton data 1442 to show how the subject of the skeleton data 1442 differs from the skeleton data 1446 of the reference worker. In the example shown in FIG. 12, it is shown that a right arm part 1447 in the skeleton data 1442 of one worker and a right arm part 1448 in the skeleton data 1446 of another worker are moving differently. Such a difference between the right arm part 1447 and the right arm part 1448 is caused by either a difference in space or a difference in time, or both, as described above. By referring to such a display data screen 1441, a user can consider what improvements are necessary when making process improvements to a work.

[0122] Fig. 13 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 13 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0123] As shown in FIG. 13, a display data screen 1451 is displayed on the display 140 of the terminal device 10. On this display data screen 1451, a body part of the worker that meets a predetermined condition is displayed as display data, as data showing the characteristic amount of the joint movement of the worker based on skeletal data extracted from video data captured during a predetermined task performed by the worker. On the display data screen 1451, for example, a body schematic diagram 1452 showing the body of the worker is displayed, and a body part display 1453 showing the body part of the worker that meets the predetermined condition is displayed. The body part of the worker that meets the predetermined condition is, for example, a body part that has a large influence on the required time of a task step (above a predetermined value), and is calculated from the characteristic amount of the joint movement of the worker based on the skeletal data (which may be a spatial characteristic amount, a time axis characteristic amount, or both). Alternatively, the body part of the worker that meets the predetermined condition is, for example, a body part that has a large difference (above a predetermined value) from the movement of another worker to be compared, and is calculated from the characteristic amount of the joint movement of the worker based on the skeletal data. As described above, the difference in movement is a concept including either the difference in space or the difference in time, or both. By referring to such a display data screen 1451, the user can consider what improvements can be made to the work process.

[0124] Fig. 14 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 14 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0125] As shown in FIG. 14, a display data screen 1461 is displayed on the display 140 of the terminal device 10. On this display data screen 1461, a graph with a horizontal time axis showing the body parts of the worker that meet a predetermined condition is displayed as display data, as data showing the feature amount of the movement of the joints of the worker based on skeletal data extracted from video data captured during a predetermined task performed by the worker. On the display data screen 1461, for example, a graph 1462 of the body parts of the reference worker and a graph 1463 of the body parts of the target worker are displayed. The body parts of the worker that meet the predetermined condition are, for example, body parts of the worker whose speed, travel distance, working time, and working rate are large (above a predetermined value) for each joint of the worker, and are calculated from the feature amount of the movement of the joints of the worker based on the skeletal data. As shown in FIG. 14, the graph 1463 shows a portion 1464 showing a difference in the movement of the left arm of the worker, which is caused by either a spatial difference or a temporal difference from the reference movement, or both, as described above. By referring to such a display data screen 1461, the user can consider what improvements can be made when improving the work process.

[0126] Fig. 15 is a diagram showing an example of a screen of display data displayed on the terminal device 10. The example of the screen in Fig. 15 shows an example of a screen in a state in which display data for displaying the work status of each worker is displayed. This corresponds to step S105 in Fig. 6.

[0127] As shown in FIG. 15, a display data screen 1471 is displayed on the display 140 of the terminal device 10. On this display data screen 1471, a time chart of the time required for each work step is displayed as display data, as data showing the feature amount of the movement of the joints of the worker based on skeletal data extracted from video data captured of a specific work by the worker. On the display data screen 1471, for example, a time chart 1472 of the work steps of a reference worker and a time chart 1473 of the work steps of a target worker are displayed. The time charts 1472 and 1473 show a time axis in the horizontal direction, are divided into work steps by points 1474, and the points 1474 at each division are connected by a straight line 1475 to show a comparison between the work steps of the reference worker and the work steps of the target worker. By comparing the time chart 1472 and the time chart 1473, the difference in the movement on the time axis between the reference worker and the target worker can be grasped. Also, for example, if the worker does not perform a specific work step, the work step that should have been performed is displayed in the time chart 1472 as shown by the dashed line 1476, but is not displayed in the time chart 1473, and can be recognized by the dashed line 1476. By referring to such a display data screen 1471, the user can consider what improvements can be made when making process improvements to the work.

[0128] <Summary> As described above, according to this embodiment, video data of a worker performing a specific task at a work site such as a factory is analyzed, and the positions of the worker's joints captured in the acquired video data are estimated to extract skeletal data of the worker. Then, based on the skeletal data, display data for displaying the task status of each worker is generated and output. This makes it possible to visually grasp the task status of a worker while comparing it with other workers when making process improvements to the task.

[0129] <Basic computer hardware configuration> 16 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (interface). These are electrically connected to each other by a communication bus 921.

[0130] The processor 901 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0131] The main memory device 902 is for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0132] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0133] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network that is directly connected by a USB (Universal Serial Bus) cable or the like.

[0134] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration among multiple computers 90 and connecting them together via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0135] <Basic functional configuration of computer 90> A description will now be given of the functional configuration of a computer realized by the basic hardware configuration (FIG. 16) of a computer 90. The computer comprises at least the functional units of a control unit, a storage unit, and a communication unit.

[0136] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 connected to each other via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0137] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding the programs in the main storage device 902, and executing processes according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0138] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can secure a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with a program. Furthermore, the control unit can cause the processor 901 to execute processes of adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0139] The term database refers to a relational database, which is used to manage sets of data called masters and tables in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table or master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Usually, a column that serves as a primary key for uniquely identifying a record is set in each table and each master, but setting a primary key in a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in a specific table or master stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0140] In addition, the database and master in this disclosure may include any data structure (such as a list, a dictionary, an associative array, or an object) in which information is structurally defined. The data structure also includes data that can be considered as a data structure by combining data with a function, class, method, or the like written in any programming language.

[0141] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0142] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0143] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Java (registered trademark), and the like.

[0144] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0145] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general purpose processors, application specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the functions described. Processors include transistors and other circuits and are considered to be circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in a memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed in this specification or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0146] Although the embodiments of the present disclosure have been described above, they can be embodied in various other forms and can be implemented with various omissions, substitutions, and modifications. These embodiments and modifications, as well as omissions, substitutions, and modifications, are included in the technical scope of the claims and their equivalents.

[0147] <Additional Notes> The matters explained in the above embodiments will be supplemented below.

[0148] The notes shall be prepared at the time of filing. [Explanation of symbols]

[0149] 1: Work analysis system 10: Terminal device 10A: Terminal device 10B: Terminal device 13: Input device 14: Output device 15: Memory 16: Storage section 19: Processor 20: Server 25: Memory 26: Storage 29: Processor 80: Network 81: Wireless base station 82: Wireless LAN router 90: Computer 111: Antenna 112: Antenna 121: First wireless communication unit 122: Second wireless communication unit 130: Operation reception section 131: Keyboard 132: Mouse 140: Display 150: Storage section 151: User information 160: Control section 161: Input operation reception section 162: Transmitter / receiver 163: Data processing section 164: Notification control section 201: Communications Department 202: Storage section 203: Control section 901: Processor 902: Main memory 903 :Auxiliary storage device 921: Communication bus 2021: Video Database 2022: Display Database 2031: Receiving control module 2032: Transmission control module 2033: Video data acquisition module 2034: Work step judgment module 2035: Video data division module 2036: Display data generation module 2037: Display data output module 2038: Skeleton data extraction module

Claims

1. A program for analyzing a work situation of a worker when executed by a computer having a processor and a memory, The program causes the processor to: acquiring video data capturing a situation in which the worker is performing a predetermined task; a step of analyzing the acquired video data and determining start times and end times of work steps that indicate a process when the predetermined work performed by the worker captured in the acquired video data is divided into a plurality of processes; a step of dividing the acquired video data into work steps identified by start times and end times based on the result of the determination, and generating display data for displaying a work status of each worker for each work step and for each worker; analyzing the acquired video data, estimating positions of the worker's joints captured in the acquired video data, and extracting skeletal data of the worker; outputting the generated display data; A program that, in a step of outputting the generated display data, outputs data indicating a difference between the extracted data indicating characteristic amounts of joint movement of one of the workers based on the skeletal data of one of the workers and data indicating characteristic amounts of joint movement of another of the workers based on the skeletal data of the other worker.

2. A program for analyzing a work situation of a worker when executed by a computer having a processor and a memory, The program causes the processor to: acquiring video data capturing a situation in which the worker is performing a predetermined task; a step of analyzing the acquired video data and determining start times and end times of work steps that indicate a process when the predetermined work performed by the worker captured in the acquired video data is divided into a plurality of processes; a step of dividing the acquired video data into work steps identified by start times and end times based on the result of the determination, and generating display data for displaying a work status of each worker for each work step and for each worker; analyzing the acquired video data, estimating positions of the worker's joints captured in the acquired video data, and extracting skeletal data of the worker; outputting the generated display data; A program that, in a step of outputting the generated display data, outputs display data indicating parts of the worker's body where the movement of the worker's joints meets predetermined conditions based on the extracted skeletal data of the worker.

3. In the step of outputting the generated display data, the display data for each of the work steps is output as a list for each of the workers; receiving a selection from a user from the display data for each work step output as a list; 3. The program according to claim 1, further comprising: outputting display data relating to the selected work step.

4. 4. The program according to claim 3, wherein in the step of outputting the generated display data, the display data of one of the workers and the display data of the other workers related to the selected work step are output simultaneously.

5. 4. The program according to claim 3, wherein in the step of outputting the generated display data, the display data of one of the workers related to the selected work step and the display data of another of the workers are output in a superimposed manner.

6. 3. The program according to claim 1, wherein in the step of outputting the generated display data, data indicating feature amounts of movements of the joints of the worker, which are included in the extracted skeletal data of the worker, are output.

7. 7. The program according to claim 6, wherein in the step of outputting the generated display data, data indicating characteristics of the movement of the worker's joints includes one or more of the trajectory, coordinates, speed, movement distance, operating time, operating rate, and degree of impact on the required time of the work step for each of the worker's joints is output.

8. 7. The program according to claim 6, wherein in the step of outputting the generated display data, data indicating a characteristic amount of a movement of a predetermined joint of the worker is output in a form different from data indicating a characteristic amount of a movement of other joints.

9. 2. The program according to claim 1, wherein in the step of outputting the generated display data, display data showing the movement of the joints of one of the workers based on the skeletal data of the extracted one of the workers, and display data showing the movement of the joints of another of the workers based on the skeletal data of the other of the workers are output.

10. 2. The program according to claim 1, wherein in the step of outputting the generated display data, display data is output in which a difference between data indicating characteristic amounts of joint movement of one of the workers based on the extracted skeletal data of one of the workers and data indicating characteristic amounts of joint movement of another of the workers based on the skeletal data of the other workers deviates by a predetermined value or more.

11. 3. The program according to claim 1, wherein in the step of outputting the generated display data, display data showing a movement of a part of the joints of the worker included in the extracted skeletal data of the worker is output.

12. An information processing device that includes a control unit and a memory and analyzes a work situation of a worker, The control unit is acquiring video data capturing a situation in which the worker is performing a predetermined task; a step of analyzing the acquired video data and determining start times and end times of work steps that indicate a process when the predetermined work performed by the worker captured in the acquired video data is divided into a plurality of processes; a step of dividing the acquired video data into work steps identified by start times and end times based on the result of the determination, and generating display data for displaying a work status of each worker for each work step and for each worker; analyzing the acquired video data, estimating positions of the worker's joints captured in the acquired video data, and extracting skeletal data of the worker; outputting the generated display data; In a step of outputting the generated display data, the information processing device outputs data indicating a difference between the extracted data indicating characteristic amounts of joint movement of one of the workers based on the skeletal data of one of the workers and data indicating characteristic amounts of joint movement of another of the workers based on the skeletal data of the other workers.

13. An information processing device that includes a control unit and a memory and analyzes a work situation of a worker, The control unit is acquiring video data capturing a situation in which the worker is performing a predetermined task; a step of analyzing the acquired video data and determining start times and end times of work steps that indicate a process when the predetermined work performed by the worker captured in the acquired video data is divided into a plurality of processes; a step of dividing the acquired video data into work steps identified by start times and end times based on the result of the determination, and generating display data for displaying a work status of each worker for each work step and for each worker; analyzing the acquired video data, estimating positions of the worker's joints captured in the acquired video data, and extracting skeletal data of the worker; outputting the generated display data; An information processing device, in a step of outputting the generated display data, outputs display data indicating parts of the worker's body where the movement of the worker's joints meets predetermined conditions based on the extracted skeletal data of the worker.

14. A method for analyzing a work situation of a worker, the method being executed by a computer having a processor and a memory, the method comprising: The method further comprises the processor: acquiring video data capturing a situation in which the worker is performing a predetermined task; a step of analyzing the acquired video data and determining start times and end times of work steps that indicate a process when the predetermined work performed by the worker captured in the acquired video data is divided into a plurality of processes; a step of dividing the acquired video data into work steps identified by start times and end times based on the result of the determination, and generating display data for displaying a work status of each worker for each work step and for each worker; analyzing the acquired video data, estimating positions of the worker's joints captured in the acquired video data, and extracting skeletal data of the worker; outputting the generated display data; In the step of outputting the generated display data, the method outputs data indicating a difference between the extracted data indicating characteristic amounts of joint movement of one of the workers based on the skeletal data of one of the workers and data indicating characteristic amounts of joint movement of another of the workers based on the skeletal data of the other worker.

15. A method for analyzing a work situation of a worker, the method being executed by a computer having a processor and a memory, the method comprising: The method further comprises the processor: acquiring video data capturing a situation in which the worker is performing a predetermined task; a step of analyzing the acquired video data and determining start times and end times of work steps that indicate a process when the predetermined work performed by the worker captured in the acquired video data is divided into a plurality of processes; a step of dividing the acquired video data into work steps identified by start times and end times based on the result of the determination, and generating display data for displaying a work status of each worker for each work step and for each worker; analyzing the acquired video data, estimating positions of the worker's joints captured in the acquired video data, and extracting skeletal data of the worker; outputting the generated display data; A method in which, in a step of outputting the generated display data, display data is output showing parts of the worker's body where the movement of the joints of the worker meets predetermined conditions based on the extracted skeletal data of the worker.

Images