Program, information processing device, and method

Abstract

The system captures images (videos) of workers at work, analyzes the captured images to generate display data for showing the workers' work status, and outputs the results. [Solution] The server 20 of the work analysis system 1 performs the following functions: acquiring video data (S101) of a worker performing a predetermined task; analyzing the acquired video data and determining the start and end times of work steps that indicate a process when the predetermined task performed by the worker captured in the acquired video data is divided into multiple processes (S102); dividing the acquired video data into work steps identified by the start and end times determined by the results of the determination (S103); generating display data (S104) to display the work status of each worker for each work step and for each worker; and outputting the generated display data (S105).

Description

Technical Field

[0001] The present disclosure relates to a program, an information processing apparatus, and a method.

Background Art

[0002] For the purpose of improving the quality of products in factories and the work efficiency of workers, etc., the working conditions of workers are photographed and the photographed video is analyzed. Also, AI (artificial intelligence) technology is applied to this analysis.

[0003] Patent Document 1 discloses a technique for analyzing the posture of a worker from a photographed image of the worker and providing information according to the state in a work support system that supports workers working on site.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in conventional work analysis, photographed images (videos) are analyzed to detect work abnormalities (omissions, procedure errors, fraud, etc.). However, even when there are no abnormalities in the work, when making improvements to the process for the work, regarding what improvement points there are, people have been looking and making judgments, which has taken time and resulted in variations in judgments.

[0006] Therefore, in the present disclosure, in order to collect data for process improvement for work, the working conditions of workers are photographed, display data for displaying the working conditions of workers is generated by analyzing the photographed images (videos), and a technique that enables outputting the results will be described.

Means for Solving the Problems

[0007] According to one embodiment of the present disclosure, a program is provided for a computer having a processor and memory to run and analyze the work status of an employee. The program causes the processor to perform the following steps: acquire video data of an employee performing a predetermined task; analyze the acquired video data and determine the start and end times of work steps that indicate a process when the predetermined task performed by the employee captured in the acquired video data is divided into multiple processes; divide the acquired video data into work steps specified by the start and end times determined by the results of the determination, generate display data for each work step and for each employee to display the work status of each employee; and output the generated display data. [Effects of the Invention]

[0008] According to this disclosure, video data of a worker performing a predetermined task is divided and output for each work step, which is specified by the start and end times of the work process. This makes it possible to generate display data for showing the work status and output the results. This makes it possible to obtain basic data for considering what improvements can be made when improving the work process. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall configuration of the work analysis system 1 according to an embodiment of the present disclosure. [Figure 2] Figure 1 is a block diagram showing the functional configuration of the terminal device 10. [Figure 3] This block diagram shows the functional configuration of server 20 in Figure 1. [Figure 4] Figure 3 shows an example of the data structure of the video database 2021. [Figure 5] Figure 3 shows an example of the data structure of the display database 2022. [Figure 6] This flowchart shows an example of the flow of display data output processing performed by the work analysis system 1. [Figure 7] This figure shows an example of a screen displaying data on terminal device 10. [Figure 8] This figure shows an example of a screen displaying data on terminal device 10. [Figure 9] This figure shows an example of a screen displaying data on terminal device 10. [Figure 10] This is a block diagram showing the functional configuration of the server 20 according to Embodiment 2 of this disclosure. [Figure 11] This figure shows an example of a screen displaying data on terminal device 10. [Figure 12] This figure shows an example of a screen displaying data on terminal device 10. [Figure 13] This figure shows an example of a screen displaying data on terminal device 10. [Figure 14] This figure shows an example of a screen displaying data on terminal device 10. [Figure 15] This figure shows an example of a screen displaying data on terminal device 10. [Figure 16] A block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0011] Also, in the following description, a "processor" is one or more processors. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may also be other types of processors such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

[0012] Also, at least one processor may be a processor in a broad sense, such as a hardware circuit (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

[0013] Also, in the following description, expressions such as "xxx database" may be used to describe information from which an output is obtained for an input. This information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, "xxx database" can be referred to as "xxx information".

[0014] Also, in the following description, the configuration of the tables constituting each database is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0015] Also, in the following description, the "program" may be used as the subject to describe processing. However, since the program is executed by a processor to perform defined processing while appropriately using a storage unit and / or an interface unit, etc., the subject of the processing may be the processor (or a device such as a controller having that processor).

[0016] The program may be installed on a device such as a computer, or it may reside on a program distribution server or a computer-readable (e.g., non-temporary) recording medium. Furthermore, in the following description, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0017] Furthermore, in the following explanation, identification numbers are used as identification information for various objects, but other types of identification information (for example, identifiers that include letters or symbols) may also be used.

[0018] Furthermore, in the following explanations, when describing similar elements without distinction, a reference code (or a common code among reference codes) may be used, and when describing similar elements with distinction, the element's identification number (or reference code) may be used.

[0019] Furthermore, in the following explanation, only control lines and information lines deemed necessary for the explanation are shown, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0020] <Overview> The following describes the work analysis system related to this disclosure. The work analysis system related to this disclosure is a system used to analyze the work status of workers performing tasks such as product manufacturing at a work site such as a factory. This work analysis system films the work status of workers at a work site such as a factory, acquires the filmed video data, performs analysis, and generates and outputs display data to show the work status for each work step and for each worker. The work analysis system related to this disclosure is a system provided as a web service, such as SaaS (Software as a Service), via a cloud server, and is configured to allow users to access it through prescribed authentication.

[0021] Incidentally, the above-mentioned method for analyzing worker performance involves analyzing captured images (videos), measuring the work time which is an element of each work unit, and detecting work anomalies (missed tasks, procedural errors, fraud, etc.) as a result. However, even if the captured images (videos) show no anomalies in the work, it was still up to a human to judge what improvements could be made to the process for that particular task. As a result, it was difficult for workers to intuitively understand how to make corrections, and human judgment was required to investigate the reasons for long work times and why the AI ​​judged the work to be abnormal, which was time-consuming and led to inconsistencies in judgment.

[0022] Furthermore, while time-based work analysis is necessary in analyzing the work of workers in factories and other facilities, the objective is to shorten the time for each work cycle (cycle time) and improve work quality. Therefore, it is essential to identify the parts and time periods that have a particularly significant impact on work delays, and conventional methods for analyzing workers' work were not suitable for this purpose.

[0023] Therefore, the work analysis system described in this disclosure acquires video data of a worker performing a predetermined task, analyzes the video data to determine the start and end times of work steps that represent the processes when the predetermined task performed by the worker is divided into multiple processes, divides the video data for each work step identified by the start and end times to generate display data, and outputs the display data.

[0024] Furthermore, the display data is configured to be output in various ways, such as by outputting it simultaneously with the display data of other workers, overlaying it with the display data of other workers, or extracting and outputting the worker's skeletal data.

[0025] This configuration makes it possible to obtain basic data for considering what improvements can be made to work processes by using the work analysis system described in this disclosure. Furthermore, it is possible to provide users with educational hints and recommendations for workers when improving work processes. As a result, the results of the work analysis are visualized, making it possible to provide objective material for subsequent process improvements and training.

[0026] <First Embodiment> The following describes the work analysis system 1 according to an embodiment of this disclosure. In the following description, for example, when a terminal device 10 accesses a server 20, the server 20 responds with information for generating a screen on the terminal device 10. The terminal device 10 generates and displays a screen based on the information received from the server 20.

[0027] <1. Overall configuration of work analysis system 1> Figure 1 is a block diagram showing the overall configuration of the work analysis system 1 according to Embodiment 1 of this disclosure. As shown in Figure 1, the work analysis system 1 includes a plurality of terminal devices (in Figure 1, terminal devices 10A and 10B are shown; hereinafter, they may be collectively referred to as "terminal devices 10") and a server 20. The terminal devices 10 and the server 20 are connected to each other so as to be able to communicate with each other via a network 80. The network 80 is composed of a wired or wireless network. In this embodiment, the server 20 is a web server (including a cloud server) and exchanges information with the terminal devices 10 via web pages. In addition, the terminal devices 10 have a web page browser installed for viewing web pages, but a dedicated application for providing services from the server 20 may be installed and configured to be viewable by the dedicated application.

[0028] Terminal device 10 is a device operated by each user. Here, a user is a person who uses terminal device 10 to analyze the work status of workers, which is a function of the work analysis system 1. Terminal device 10 can be implemented as a stationary PC (Personal Computer), laptop PC (Note PC), etc. Alternatively, terminal device 10 may be a mobile device such as a tablet compatible with a mobile communication system or a smartphone.

[0029] The terminal device 10 is connected to the server 20 via the network 80 in a communicative manner. The terminal device 10 is connected to the network 80 by communicating with communication equipment such as a wireless base station 81 that supports communication standards such as 4G, 5G, and LTE (Long Term Evolution), and a wireless LAN router 82 that supports wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11. As shown as terminal device 10B in Figure 1, the terminal device 10 includes a communication interface 12, an input device 13, an output device 14, a memory 15, a storage unit 16, and a processor 19.

[0030] The communication interface 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with external devices. The input device 13 is an input device (for example, a keyboard, touch panel, touchpad, mouse, or other pointing device) for receiving input operations from the user. The output device 14 is an output device (display, speaker, etc.) for presenting information to the user. The memory 15 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for saving data, such as flash memory or an HDD (Hard Disk Drive). The processor 19 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0031] Server 20 is a device that analyzes the work status of workers in a factory or other work site and outputs the analysis results. Server 20 acquires video data of workers performing predetermined tasks and analyzes the video data. From the analysis results of the video data, Server 20 determines the start and end times of the work steps that represent the processes when the predetermined task performed by the worker is divided into multiple processes, and generates display data by dividing the video data into work steps identified by the start and end times. Furthermore, Server 20 outputs the generated display data.

[0032] Server 20 is a computer connected to network 80. Server 20 includes a communication interface 22, an input / output interface 23, memory 25, storage 26, and a processor 29.

[0033] Communication IF22 is an interface for inputting and outputting signals so that the server 20 can communicate with external devices. Input / Output IF23 functions as an interface to an input device for receiving input operations from the user and an output device for presenting information to the user. Memory 25 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM (Dynamic Random Access Memory). Storage 26 is a storage device for saving data, such as flash memory or HDD (Hard Disk Drive). Processor 29 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0034] <1.1 Configuration of terminal device 10> Figure 2 is a block diagram showing the functional configuration of the terminal device 10 that constitutes the work analysis system 1 of Embodiment 1. As shown in Figure 2, the terminal device 10 includes a plurality of antennas (antenna 111, antenna 112), wireless communication units corresponding to each antenna (first wireless communication unit 121, second wireless communication unit 122), an operation reception unit 130 (including a keyboard 131 and a mouse 132), a display 140, a storage unit 150, and a control unit 160. The terminal device 10 also has functions and configurations not specifically shown in Figure 2 (for example, a battery for maintaining power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc.). As shown in Figure 2, each block included in the terminal device 10 is electrically connected by a bus or the like.

[0035] Antenna 111 radiates signals emitted by terminal device 10 as radio waves. Antenna 111 also receives radio waves from space and provides the received signals to first wireless communication unit 121.

[0036] Antenna 112 radiates signals emitted by terminal device 10 as radio waves. Antenna 112 also receives radio waves from space and provides the received signals to second wireless communication unit 122.

[0037] The first wireless communication unit 121 performs modulation and demodulation processing, etc., for the terminal device 10 to transmit and receive signals via the antenna 111 in order to communicate with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing, etc., for the terminal device 10 to transmit and receive signals via the antenna 112 in order to communicate with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are a communication module that includes a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high-frequency circuit, etc. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation, demodulation, and frequency conversion of the wireless signals transmitted and received by the terminal device 10, and provide the received signal to the control unit 160.

[0038] The operation reception unit 130 has a mechanism for receiving user input operations. Specifically, the operation reception unit 130 includes a keyboard 131 and a mouse 132. The operation reception unit 130 may also be configured as a touchscreen that detects the user's contact position with the touch panel, for example, by using a capacitive touch panel.

[0039] The keyboard 131 accepts user input operations from the terminal device 10. The keyboard 131 is a device for character input and outputs the input character information as an input signal to the control unit 160.

[0040] The mouse 132 accepts user input operations from the terminal device 10. The mouse 132 is a pointing device for selecting objects displayed on the display 140, and outputs the selected position information on the screen and information indicating that a button is pressed as input signals to the control unit 160.

[0041] The display 140 displays data such as images, videos, and text in accordance with the control of the control unit 160. The display 140 is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0042] The storage unit 150 is composed of a memory 15, such as flash memory, and a storage unit 16, and stores data and programs used by the terminal device 10. In certain situations, the storage unit 150 stores user information 151.

[0043] User information 151 is information about a user who uses terminal device 10 to analyze the work status of workers, which is a function of work analysis system 1. User information may include information that identifies the user (user ID), the user's name, address, organizational information such as the company to which the user belongs, etc.

[0044] The control unit 160 is composed of, for example, a processor 19, and controls the operation of the terminal device 10 by reading a program stored in the memory unit 150 and executing instructions contained in the program. The control unit 160 is, for example, an application that is pre-installed on the terminal device 10. By operating according to the program, the control unit 160 performs the functions of an input operation receiving unit 161, a transmitting / receiving unit 162, a data processing unit 163, and a notification control unit 164.

[0045] The input operation reception unit 161 processes input operations from the user to an input device such as a keyboard 131.

[0046] The transmitting / receiving unit 162 performs processing to enable the terminal device 10 to send and receive data with an external device such as a server 20 in accordance with a communication protocol.

[0047] The data processing unit 163 performs calculations on the data received as input by the terminal device 10 according to the program and outputs the calculation results to memory or the like.

[0048] The notification control unit 164 performs processing to present information to the user. The notification control unit 164 also performs processing to display the display image on the display 140, etc.

[0049] <1.2 Functional Configuration of Server 20> Figure 3 shows the functional configuration of the server 20 that constitutes the work analysis system 1 of Embodiment 1. As shown in Figure 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0050] The communications unit 201 performs processing to enable the server 20 to communicate with external devices.

[0051] The storage unit 202 stores data and programs used by the server 20. The storage unit 202 also stores the video database 2021 and the display database 2022, etc.

[0052] The video database 2021 is a database for acquiring and storing video data of workers performing predetermined tasks, for use with the work analysis system 1. The video database 2021 contains data including the start and end times of work steps, which represent the processes when a predetermined task performed by a worker is divided into multiple stages. Further details will be described later.

[0053] Display Database 2022 is a database for registering and storing display data that shows the work status of each worker, which is generated by dividing video data based on the start and end times of work steps determined using the work analysis system 1. Further details will be described later.

[0054] The control unit 203 performs the functions shown in the various modules, namely the receive control module 2031, the transmit control module 2032, the video data acquisition module 2033, the work step determination module 2034, the video data splitting module 2035, the display data generation module 2036, and the display data output module 2037, as the server 20's processor 29 processes according to the program.

[0055] The receive control module 2031 controls the process by which the server 20 receives signals from external devices according to a communication protocol.

[0056] The transmission control module 2032 controls the process by which the server 20 transmits signals to external devices according to a communication protocol.

[0057] The video data acquisition module 2033 controls the process of acquiring video data of workers performing predetermined tasks at a factory or other work site. The video data acquisition module 2033 acquires video data captured by a camera of workers performing predetermined tasks, such as assembling manufactured goods or performing various processing tasks at a factory or other work site. The video data acquired by the video data acquisition module 2033 may be video data captured by a camera equipped on the terminal device 10 or server 20 (not shown in the diagram), or video data captured by an external camera.

[0058] The video data acquisition module 2033, for example, when acquiring video data from a camera on the terminal device 10 or an external camera, receives the video data transmitted from the terminal device 10 or the external camera via the communication unit 201. Furthermore, when acquiring video data from a camera on the server 20, for example, the video data acquisition module 2033 receives the video data as a function of the control unit 203. The video data acquired by the video data acquisition module 2033 can be in any data format, such as MP4, MOV, WMV, AVI, or AVCHD.

[0059] The video data acquisition module 2033 stores the acquired video data, for example, in the video database 2021.

[0060] The work step determination module 2034 analyzes the video data acquired by the video data acquisition module 2033 and controls the process of determining the start and end times of work steps that represent the processes when a predetermined task performed by a worker, as captured in the acquired video data, is divided into multiple processes. Specifically, the multiple work steps into which the predetermined task is divided by the work step determination module 2034 are work steps in the assembly of manufactured goods in a factory, such as taking out tools, taking out objects, assembling parts, and placing the assembled object in a predetermined location.

[0061] The work step determination module 2034 analyzes the video data and, for example, extracts features from the video data for a predetermined task performed by a worker. The work step determination module 2034 compares the features with those defined for each of the multiple work steps to be categorized and determines whether the work performed by the worker corresponds to the defined work step. Then, for video data that the work step determination module 2034 determines to correspond to a specific work step, it determines the start and end times of that work step. At this time, the work step determination module 2034 may determine that the extracted video data corresponds to a defined work step if the features are above a threshold value defined for each of the multiple work steps to be categorized, or it may calculate a degree of agreement using a predetermined calculation and determine that the data corresponds to a defined work step if the degree of agreement is above a predetermined value.

[0062] The work step determination module 2034 may determine whether a task falls under a task step by using a determination model that has been trained on video data acquired by the video data acquisition module 2033. The work step determination module 2034 may also determine whether a task falls under a task step by using a determination model that has been trained on video data using any machine learning algorithm (supervised machine learning, etc.), deep learning algorithm, neural network (e.g., deep neural network (DNN)) model, etc. The determination model used by the work step determination module 2034 does not need to be a single training model; it may be implemented by switching between multiple independent training models for each task step.

[0063] The work step determination module 2034 stores the start and end times of the determined work step in, for example, the video database 2021.

[0064] The video data splitting module 2035 controls the process of splitting the video data acquired by the video data acquisition module 2033 into work steps specified by start and end times, based on the results of the work step determination module 2034. The video data splitting module 2035 splits the video data into chapters based on the start and end times determined by the work step determination module. Then, the video data splitting module 2035 associates each of the split video data with the name of the work step determined by the work step determination module 2034.

[0065] The video data splitting module 2035 may split video data into chapters at the start and end times, but may also split the video data at a predetermined time (e.g., about 5 seconds) earlier than the start time and at a predetermined time (e.g., about 5 seconds) later than the end time. Furthermore, when splitting video data into chapters for each work step, the video data splitting module 2035 may split the video data so that there is a predetermined time (e.g., about 5 seconds) of overlap at the transition points between work steps. This is to prevent unnatural cutting of preceding and succeeding scenes when splitting video data into chapters for each work step.

[0066] The video data splitting module 2035 stores the split video data, for example, in the display database 2022.

[0067] The display data generation module 2036 controls the process of generating display data for each work step and for each worker, showing the work status of each worker, using video data divided into work steps. The display data generation module 2036 performs predetermined editing and processing on the video data divided into work steps and generates display data to be used as basic data for considering areas for improvement when improving the recorded work process.

[0068] The display data generation module 2036 may, for example, set the name of the work step determined by the work step determination module 2034 as the title for the video data divided into work steps, or it may set the name of the work step as the thumbnail image for the divided video data. In addition, the display data generation module 2036 may, for example, edit the video data divided into work steps to adjust the playback speed to match the video data of a reference work step. The video data of a reference work step may, for example, be sample video data for that work step, and may be video data of the work situation by an experienced worker who serves as a sample, or it may be video data automatically generated by a generation AI, etc.

[0069] The display data generation module 2036 stores the generated display data in, for example, the display database 2022.

[0070] The display data output module 2037 controls the process of outputting the display data generated by the display data generation module 2036. The display data output module 2037 outputs the display data and provides it to the user, for example, as basic data for process improvement regarding the captured work steps. The display data output module 2037 transmits the output display data to the terminal device 10 used by the user via the communication unit 201 and displays it on the display 140.

[0071] The display data output module 2037 may, for example, output a list of display data for each work step for each worker and display it on the display 140 of the terminal device 10. In this case, the output may be for one worker, or for multiple workers (for example, for each designated department or group). The display data output module 2037 may then accept a selection from the list of display data for each work step output by the user and output (play back) the display data related to the selected work step. Furthermore, when outputting (playing back) the display data related to the selected work step, the display data output module 2037 may output (play back) the display data of the selected worker and the display data of other workers simultaneously, or output (play back) them superimposed. Here, the display data of other workers refers to the display data of the work status of the worker to be compared, and may be the display data of the work status of an experienced worker that serves as a model, or it may be display data automatically generated by a generation AI or the like.

[0072] <2 Data Structure> Figure 4 shows an example of the data structure of the video database 2021 in Figure 3. Figure 5 also shows an example of the data structure of the display database 2022 in Figure 3.

[0073] As shown in Figure 4, each record in the video database 2021 includes the fields "Worker ID", "Worker", "Shooting Date and Time", and "Step Split Time", etc.

[0074] The "Worker ID" field is information that identifies each worker performing a specific task, as captured in the video data that is the target of analysis by work analysis system 1.

[0075] The "Worker" field contains the name of the worker performing the specified task, as captured in the video data, which is the target of analysis by the work analysis system 1. In addition to the worker's name, the "Worker" field may also store organizational information such as the company to which the worker belongs.

[0076] The item "Shooting Date and Time" is information indicating the date and time when the video data, which is the target of analysis by work analysis system 1, was shot.

[0077] The item "Step Division Time" is information that shows the results of determining the start and end times of work steps for video data that is the target of analysis by the work analysis system 1. Specifically, it includes the items "Step ID", "Start Time", and "End Time", etc.

[0078] The item "Step ID" is information that identifies each work step, which is the result of the determination made for the video data that is the target of analysis by work analysis system 1.

[0079] The item "Start Time" is information indicating the start time of the work step, which is the result of the determination made for the video data that is the target of analysis by the work analysis system 1.

[0080] The item "Completion Time" is information indicating the completion time of a work step, which is the result of the determination made regarding the video data that is the subject of analysis by the work analysis system 1.

[0081] The video data acquisition module 2033 on server 20 adds a record to the video database 2021 when it acquires video data. Additionally, the work step determination module 2034 determines the start and end times of the work steps in the video data and adds a record to the "Step Split Time" field in the video database 2021.

[0082] As shown in Figure 5, each record in the display database 2022 includes the fields "Step ID", "Step Name", and "Display Data Details", etc.

[0083] The item "Step ID" is information that identifies each work step, which is the result of the determination of the video data that is the target of analysis by work analysis system 1, and corresponds to the item "Step ID" in video database 2021.

[0084] The item "Step Name" is information indicating the name of the work step that was determined from the video data that is the target of analysis by the work analysis system 1. For example, it stores the name of a specific work step, such as "△○ process".

[0085] The item "Display Data Details" contains information about the display data generated from video data, which is the target of analysis by the work analysis system 1. Specifically, it includes the items "Worker ID" and "Display Data," etc.

[0086] The "Worker ID" field is information that identifies each worker performing a specific task, as captured in the video data that is the target of analysis by the work analysis system 1, and corresponds to the "Worker ID" field in the video database 2021.

[0087] The item "Display Data" is the display data generated from the video data that is the target of analysis by the work analysis system 1. The item "Display Data" may not contain the display data itself, but rather the storage location of the server 20 where the display data is stored, or link information that allows access to other web servers, etc.

[0088] The video data splitting module 2035 on server 20 adds a record to the display database 2022 when it splits the video data. Additionally, the display data generation module 2036 adds a record to the "Display Data Details" field in the display database 2022 when it generates the display data.

[0089] <3 operations> The following describes the method of outputting display data by the work analysis system 1 according to the embodiment of this disclosure, with reference to Figure 6.

[0090] Figure 6 is a flowchart showing an example of the flow of display data output processing by the work analysis system 1.

[0091] In step S101, the video data acquisition module 2033 of the server 20 acquires video data of workers performing predetermined tasks at a work site such as a factory. In step S101, video data may be acquired from a camera installed in the terminal device 10 or the server 20, or from an external camera.

[0092] In step S102, the work step determination module 2034 of the server 20 analyzes the video data acquired in step S101 and determines the start and end times of the work steps that represent the processes when the predetermined work performed by the worker captured in the acquired video data is divided into multiple processes. In step S102, for example, the video data is analyzed, and for the predetermined work performed by the worker, feature quantities are extracted from the video data and compared with feature quantities defined for each work step to determine whether or not it corresponds to a defined work step, and for video data that is determined to correspond to a specific work step, the start and end times of that work step are determined.

[0093] In step S103, the video data splitting module 2035 of the server 20 splits the video data acquired in step S101 into work steps specified by the start time and end time, based on the result of the determination in step S102. In step S103, chapter division is performed on the video data for which the start time and end time of the work step have been determined, and each of the split video data is associated with the name of the work step.

[0094] In step S104, the display data generation module 2036 of the server 20 generates display data for each work step and for each worker, showing the work status of each worker, using the video data that was divided into work steps in step S103. In step S104, the video data divided into work steps is edited and processed as required, and generated as display data to be used as basic data for considering areas for improvement when improving the process based on the filmed work.

[0095] In step S105, the display data output module 2037 of the server 20 outputs the display data generated in step S104. In step S105, the output display data is transmitted to the terminal device 10 used by the user via the communication unit 201 and displayed on the display 140.

[0096] As described above, the work analysis system 1 acquires video data of workers performing predetermined tasks at a work site such as a factory, analyzes the video data, and determines the start and end times of work steps that represent the processes when the predetermined tasks performed by the workers captured in the acquired video data are divided into multiple processes. Based on the determination result, the system generates display data by dividing the video data for each work step specified by the start and end times, and outputs the display data. As a result, by using the work analysis system disclosed herein, it becomes possible to acquire display data that serves as basic data for considering what improvements can be made when improving work processes.

[0097] <4 Screen Examples> The following describes examples of each screen for the display data output process by the work analysis system 1, with reference to Figures 7 through 9.

[0098] Figure 7 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 7 shows a screen displaying data for showing the work status of each worker. This corresponds to step S105 in Figure 6.

[0099] As shown in Figure 7, the display data screen 1411 is displayed on the display 140 of the terminal device 10. This display data screen 1411 displays data that has been divided from video data recorded of a predetermined task performed by an operator. For example, the display data screen 1411 shows an operator 1412, and displays the work object 1414 and tools 1415 placed on the workbench 1413 where the operator 1412 is performing the predetermined task. By referring to such a display data screen 1411, the user can consider what improvements can be made to the work process.

[0100] Figure 8 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 8 shows a screen displaying data for showing the work status of each worker. This corresponds to step S105 in Figure 6.

[0101] As shown in Figure 8, the display data screen 1421 is displayed on the display 140 of the terminal device 10. This display data screen 1421 displays a list of display data divided from video data recorded of a predetermined task performed by a worker, and display data 1423 recorded of a predetermined task performed by a worker. In the list of display data screen 1422, for example, thumbnail images 1424 of the display data for each work step are displayed as a list for each work step. In the display data 1423, for example, a worker is displayed, showing the worker performing a predetermined task. When the display data screen 1421 is displayed, if the user selects one of the thumbnail images 1424, the corresponding display data 1423 is output (played). By referring to such a display data screen 1421, the user can consider what improvements can be made when improving the work process.

[0102] Figure 9 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 9 shows a screen displaying data for showing the work status of each worker. This corresponds to step S105 in Figure 6.

[0103] As shown in Figure 9, the display data screen 1431 is displayed on the display 140 of the terminal device 10. This display data screen 1431 displays a list of display data divided from video data recorded of a predetermined task performed by an operator 1432, display data 1433 recorded of a predetermined task performed by one operator, and display data 1434 recorded of a predetermined task performed by another operator. In the list of display data screen 1432, for example, thumbnail images 1435 of the display data for each work step are displayed as a list for each work step. In the display data 1433 and 1434, for example, an operator is displayed, showing the operator performing a predetermined task. When the display data screen 1431 is displayed, if the user selects one of the thumbnail images 1435, the display data 1433 of the corresponding operator is output (played), and the display data 1434 of the other corresponding operator is output (played) simultaneously. By referring to this data display screen 1421, users can consider what improvements can be made to their work processes.

[0104] Note that the example shown in Figure 9 is a configuration in which the display data 1433 of the selected worker and the display data 1434 of other workers are output (played) simultaneously from the thumbnail image 1435, but the system is not limited to this example. For example, the system may be configured to output (play) the display data 1433 of the selected worker and the display data 1434 of other workers superimposed on each other.

[0105] <Summary> As described above, according to this embodiment, video data is acquired from a factory or other work site in which a worker is performing a predetermined task. The video data is analyzed, and the start and end times of the work steps, which represent the processes when the predetermined task performed by the worker as captured in the acquired video data is divided into multiple processes, are determined. Based on the determination result, the video data is divided for each work step specified by the start and end times, and display data is generated and output. As a result, by using the work analysis system according to this disclosure, it becomes possible to acquire display data that serves as basic data for considering what improvements can be made when improving the work process.

[0106] Furthermore, the display data is output (played) from video data recorded of a worker performing a predetermined task. At this time, the display data for each work step is displayed as a list, and the user can select the appropriate display data to display. In addition, the display data recorded of a predetermined task performed by one worker and the display data recorded of a predetermined task performed by another worker are displayed simultaneously or superimposed. This makes it possible to visually understand the situation by comparing the work status of one worker with that of other workers when improving the work process.

[0107] <Second Embodiment> The following describes other embodiments of the work analysis system 1.

[0108] <1. Overall configuration of work analysis system 1> Figure 10 shows the functional configuration of the server 20 according to Embodiment 2 of this disclosure. The overall configuration of the work analysis system 1 and the configuration of the terminal device 10 in the second embodiment are the same as in the first embodiment and will not be described again. The configuration of the server 20 is the same as in the first embodiment, except that it is newly equipped with the functionality of the skeleton data extraction module 2038, as shown in Figure 10. The functionality of the skeleton data extraction module 2038 in the second embodiment will be described below.

[0109] The skeletal data extraction module 2038 analyzes the video data acquired by the video data acquisition module 2033, estimates the positions of the worker's joints as captured in the acquired video data, and controls the process of extracting the worker's skeletal data. The skeletal data extraction module 2038 analyzes the video data for a predetermined task performed by the worker, and for example, identifies and extracts body parts from the worker's body movements using known image recognition techniques. The skeletal data extraction module 2038 connects the points indicating the positions of each identified body joint with straight lines, and uses the information obtained from points and lines as skeletal information. These points are mainly placed on the joints of the body and are configured to change their angle with the straight lines according to the body's movements. This configuration makes it possible to represent body movements using points and straight lines.

[0110] The display data generation module 2036 according to Embodiment 2 may generate display data for displaying the work status of each worker based on the skeletal data extraction module 2038. The display data generation module 2036 performs predetermined editing and processing on the video data divided for each work step, and generates display data by representing the movements of the worker's body with skeletal information obtained as points and lines.

[0111] Furthermore, the display data generation module 2036 may generate display data that shows the characteristic features of the worker's joint movements, which are included in the worker's skeletal data, based on the skeletal data extraction module 2038. The display data generation module 2036 generates display data that shows the characteristic features of the worker's joint movements, for example, including one or more of the following for each of the worker's joints: trajectory, coordinates, velocity, distance traveled, working time, utilization rate, and the degree of influence of each work step on the time required. The display data generation module 2036 analyzes the skeletal data and obtains this data.

[0112] The display data output module 2037 according to Embodiment 2 may output display data based on skeletal data as display data generated by the display data generation module 2036. The display data output module 2037 outputs, for example, display data that represents the movement of the worker's body using skeletal information obtained as points and lines. In this case, the display data output module 2037 may output, for example, display data that shows the movement of some of the worker's joints.

[0113] Furthermore, the display data output module 2037 may output data as display data that shows the characteristic quantities of the worker's joint movements, which are included in the worker's skeletal data. The display data output module 2037 outputs data as display data that shows the characteristic quantities of the worker's joint movements, for example, data that includes one or more of the following for each of the worker's joints: trajectory, coordinates, velocity, distance traveled, working time, working rate, and degree of influence on the time required for work steps. In addition, the display data output module 2037 may output data showing the characteristic quantities of the movement of a predetermined joint of the worker as display data showing the characteristic quantities of the movement of the worker, in a manner different from the data showing the characteristic quantities of the movement of other joints. In this case, the data showing the characteristic quantities of the movement of a predetermined joint of the worker to be output may be data showing the characteristic quantities of the movement of a part that is influencing the movement of the worker's work step (for example, causing a delay), or it may be data showing the characteristic quantities of the movement of a part that has a large difference from the characteristic quantities of the movement of a reference work step.

[0114] Furthermore, the display data output module 2037 may output display data showing the joint movements of one worker based on the skeletal data of one worker, and display data showing the joint movements of other workers based on the skeletal data of other workers. In this case, the display data output module 2037 may output (play back) the skeletal data of one worker and the skeletal data of other workers simultaneously, or output (play back) them superimposed. Here, the skeletal data of other workers is, as mentioned above, skeletal data about the work situation of the worker to be compared, and may be skeletal data of the work situation of a veteran worker who serves as a model, or skeletal data automatically generated by a generation AI, etc. In this case, the display data output module 2037 may output data showing the difference between data showing the feature quantities of joint movements based on the skeletal data of one worker and data showing the feature quantities of joint movements based on the skeletal data of other workers, and may output display data in which the difference deviates by a predetermined value or more. Here, the difference (showing the feature quantities) of joint movements is a concept that includes either a spatial difference, a temporal difference, or both. Specifically, the spatial difference is the difference in spatial position when comparing the movement of one worker's skeletal data with the movement of another worker's skeletal data, and arises, for example, from differences in the speed of movement or the distance traveled at that point. The temporal difference is the difference on the time axis when comparing the movement of one worker's skeletal data with the movement of another worker's skeletal data, and arises, for example, from differences in the speed of movement or the working time at that point.

[0115] <2 Data Structure> The data structure in the second embodiment is the same as in the first embodiment, so it will not be described again.

[0116] <3 operations> The operation in the second embodiment is the same as in the first embodiment, so it will not be described again.

[0117] <4 Screen Examples> The following describes examples of each screen for the display data output processing by the work analysis system 1, with reference to Figures 11 through 14.

[0118] Figure 11 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 11 shows a screen displaying data for showing the work status of each worker. This corresponds to step S105 in Figure 6.

[0119] As shown in Figure 11, the display data screen 1441 is displayed on the display 140 of the terminal device 10. This display data screen 1441 displays skeletal data that has been separated and extracted from video data recorded of a predetermined task performed by a worker. For example, the display data screen 1441 displays the worker's skeletal data 1442. The skeletal data 1442 represents the worker's body as a skeleton and consists of points 1443 indicating the positions of the worker's joints and straight lines 1444 connecting the points 1443. In addition, the display data screen 1441 also displays, for example, the trajectory 1445 of each of the worker's joints. Note that the work object 1414 and tool 1415 shown in Figure 7 may also be displayed on the display data screen 1441 (not shown). By referring to such a display data screen 1441, the user can consider what improvements can be made when improving the work process.

[0120] Figure 12 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 12 shows a screen displaying data for showing the work status of each worker. This corresponds to step S105 in Figure 6.

[0121] Figure 12 shows a state where the skeletal data 1446 of another worker (e.g., a reference worker, a model worker, etc.) is displayed superimposed on the skeletal data 1442 of one worker shown on the display data screen 1441 in Figure 11. For example, by superimposing the display, it shows how the movement of the subject of skeletal data 1442 differs from that of the reference worker's skeletal data 1446. In the example shown in Figure 12, it is shown that the right arm portion 1447 in the skeletal data 1442 of one worker and the right arm portion 1448 in the skeletal data 1446 of the other worker are moving differently. Such differences between the right arm portion 1447 and the right arm portion 1448 arise from either spatial differences, temporal differences, or both, as described above. By referring to such a display data screen 1441, the user can consider what improvements can be made when improving the work process.

[0122] Figure 13 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 13 shows a screen displaying data to show the work status of each worker. This corresponds to step S105 in Figure 6.

[0123] As shown in Figure 13, the display data screen 1451 is displayed on the display 140 of the terminal device 10. This display data screen 1451 displays body parts of the worker that meet predetermined conditions as display data, representing the characteristic quantities of the worker's joint movements based on skeletal data extracted from video data taken of a predetermined task performed by the worker. For example, the display data screen 1451 displays a schematic diagram 1452 showing the worker's body and a body part display 1453 showing the body parts of the worker that meet predetermined conditions. Body parts of the worker that meet predetermined conditions are, for example, body parts that have a large influence (greater than or equal to a predetermined value) on the time required for the work step, and are calculated from the characteristic quantities of the worker's joint movements (may be spatial characteristics, time-axis characteristics, or both) based on skeletal data. Alternatively, body parts of the worker that meet predetermined conditions are, for example, body parts that have a large difference (greater than or equal to a predetermined value) compared to the movements of other workers used for comparison, and are calculated from the characteristic quantities of the worker's joint movements based on skeletal data. As described above, the concept of movement difference includes either a spatial difference, a temporal difference, or both. By referring to such a display data screen 1451, users can consider what improvements can be made when improving their work processes.

[0124] Figure 14 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 14 shows a screen displaying data to show the work status of each worker. This corresponds to step S105 in Figure 6.

[0125] As shown in Figure 14, the display data screen 1461 is displayed on the display 140 of the terminal device 10. This display data screen 1461 displays a graph with a horizontal time axis showing parts of the worker's body that meet predetermined conditions, as data indicating characteristic quantities of the worker's joint movements based on skeletal data extracted from video data taken of a predetermined task performed by the worker. For example, the display data screen 1461 displays a graph 1462 of a reference worker's body parts and a graph 1463 of the target worker's body parts. The parts of the worker's body that meet predetermined conditions are, for example, body parts where the speed, distance traveled, working time, working rate, etc. for each joint of the worker are large (above predetermined values), and are calculated from the characteristic quantities of the worker's joint movements based on skeletal data. As shown in Figure 14, graph 1463 shows a point 1464 that shows a difference in the movement of the worker's left arm, which, as described above, is caused by either a spatial difference, a temporal difference, or both from the reference movement. By referring to this data display screen 1461, users can consider what improvements can be made to their work processes.

[0126] Figure 15 shows an example of a screen displaying data to be shown on the terminal device 10. The example screen in Figure 15 shows a screen displaying data for showing the work status of each worker. This corresponds to step S105 in Figure 6.

[0127] As shown in Figure 15, the display data screen 1471 is displayed on the display 140 of the terminal device 10. This display data screen 1471 displays a time chart of the time required for each work step as display data, which is data showing characteristic quantities of the joint movements of the worker, based on skeletal data extracted from video data taken of a predetermined work performed by the worker. For example, the display data screen 1471 displays a time chart 1472 of the work steps of a reference worker and a time chart 1473 of the work steps of a target worker. The time charts 1472 and 1473 show a time axis horizontally, are divided into points 1474 for each work step, and the points 1474 at each division are connected by straight lines 1475 to show a comparison between the work steps of the reference worker and the work steps of the target worker. By comparing the time charts 1472 and 1473, it is possible to grasp the difference in movement on the time axis between the reference worker and the target worker. Furthermore, if, for example, the worker fails to perform a specific work step, the work step that should have been performed will be displayed on the time chart 1472, but not on the time chart 1473, as shown by the dashed line 1476. This can be recognized by the dashed line 1476. By referring to this display data screen 1471, the user can consider what improvements can be made to the work process.

[0128] <Summary> As described above, according to this embodiment, in a work site such as a factory, video data of workers performing predetermined tasks is analyzed, the positions of the workers' joints captured in the acquired video data are estimated, and the workers' skeletal data is extracted. Then, based on the skeletal data, display data is generated and output to show the work status of each worker. This makes it possible to visually understand the situation by comparing the work status of workers with that of other workers when improving work processes.

[0129] <Basic Computer Hardware Configuration> Figure 16 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary storage 903, and a communication interface IF991. These are electrically connected to each other by a communication bus 921.

[0130] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0131] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0132] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0133] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0134] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0135] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 (Figure 16) is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0136] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0137] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0138] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0139] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0140] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0141] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0142] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0143] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0144] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0145] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered a circuit or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0146] While embodiments relating to this disclosure have been described above, these can be implemented in various other forms and can be carried out with various omissions, substitutions, and modifications. These embodiments and variations, as well as those with omissions, substitutions, and modifications, are included within the technical scope of the claims and their equivalents.

[0147] <Note> The details described in each of the above embodiments are noted below.

[0148] The supplementary information should be prepared at the time of application. [Explanation of symbols]

[0149] 1: Work Analysis System 10: Terminal device 10A: Terminal device 10B: Terminal device 13: Input device 14: Output device 15: Memory 16: Storage section 19: Processor 20: Server 25: Memory 26: Storage 29: Processor 80: Network 81: Wireless base station 82: Wireless LAN router 90: Computer 111: Antenna 112: Antenna 121: First Radio Communication Unit 122: Second Wireless Communication Section 130: Operation Reception Section 131: Keyboard 132: Mouse 140: Display 150: Storage section 151: User Information 160: Control Unit 161: Input operation reception unit 162: Transceiver Unit 163: Data Processing Unit 164: Notification Control Unit 201: Communications Department 202: Storage section 203: Control Unit 901: Processor 902: Main memory 903 :Auxiliary storage device 921: Communications bus 2021: Video Database 2022: Display Database 2031: Receiver control module 2032: Transmit control module 2033: Video data acquisition module 2034: Work step determination module 2035: Video data splitting module 2036: Display data generation module 2037: Display data output module 2038: Skeletal Data Extraction Module

Claims

1. A program to be executed on a computer equipped with a processor and memory, for analyzing the work status of an employee, The program is provided to the processor: The steps include: acquiring video data of the worker performing a predetermined task, The steps include: analyzing the acquired video data and determining the start and end times of the work steps that represent the processes when the predetermined work performed by the worker, as captured in the acquired video data, is divided into multiple processes; The acquired video data is divided into work steps specified by the start and end times determined by the judgment result, and display data is generated for each work step and for each worker to display the work status of each worker. A program that executes the steps of outputting the generated display data.

2. In the step of outputting the generated display data, the display data for each work step is output as a list for each worker. The user selects from the displayed data for each of the aforementioned work steps, which is output as a list. The program according to claim 1, which outputs display data relating to the selected work step.

3. The program according to claim 2, wherein in the step of outputting generated display data, the program simultaneously outputs the display data of one worker and the display data of another worker relating to the selected work step.

4. The program according to claim 2, wherein in the step of outputting generated display data, the program superimposes and outputs the display data of one worker and the display data of another worker relating to the selected work step.

5. The aforementioned program, further, The acquired video data is analyzed, and the position of the worker's joints as captured in the acquired video data is estimated to extract the worker's skeletal data. The program according to any one of claims 1 to 4, wherein in the step of outputting generated display data, it outputs display data showing the movement of the worker's joints based on the extracted worker's skeletal data.

6. The program according to claim 5, wherein in the step of outputting generated display data, data indicating the characteristic quantities of joint movement of the worker, which are included in the extracted worker's skeletal data, is output.

7. The program according to claim 6, wherein in the step of outputting generated display data, the program outputs data that includes one or more of the following as data indicating the characteristic quantities of the movement of the worker's joints: the trajectory of each joint of the worker, coordinates, speed, distance traveled, working time, operating rate, and the degree of influence on the time required for the work step.

8. The program according to claim 6, wherein in the step of outputting generated display data, data indicating the characteristic quantities of movement of a predetermined joint of the worker is output in a different manner from data indicating the characteristic quantities of movement of other joints.

9. The program according to claim 5, wherein in the step of outputting generated display data, it outputs display data showing the joint movements of one worker based on the extracted skeletal data of one worker, and display data showing the joint movements of other workers based on the skeletal data of other workers.

10. The program according to claim 9, wherein in the step of outputting generated display data, it outputs data showing the difference between data showing the characteristic quantities of joint movement of one worker based on the extracted skeletal data of one worker and data showing the characteristic quantities of joint movement of other workers based on the skeletal data of other workers.

11. The program according to claim 9, wherein, in the step of outputting generated display data, the program outputs display data in which the difference between data showing the characteristic quantities of joint movement of one worker based on the extracted skeletal data of one worker and data showing the characteristic quantities of joint movement of other workers based on the skeletal data of other workers deviates by a predetermined value or more.

12. The program according to claim 5, wherein in the step of outputting generated display data, it outputs display data that shows the movement of some of the joints of the worker, which is included in the extracted worker's skeletal data.

13. An information processing device comprising a control unit and a memory, which analyzes the work status of an worker, The control unit, The steps include: acquiring video data of the worker performing a predetermined task, The steps include: analyzing the acquired video data and determining the start and end times of the work steps that represent the processes when the predetermined work performed by the worker, as captured in the acquired video data, is divided into multiple processes; The acquired video data is divided into work steps specified by the start and end times determined by the judgment result, and display data is generated for each work step and for each worker to display the work status of each worker. An information processing device that performs the steps of outputting generated display data.

14. A method for analyzing the work status of an employee, which is performed by a computer having a processor and memory, The above method involves the processor, The steps include: acquiring video data of the worker performing a predetermined task, The steps include: analyzing the acquired video data and determining the start and end times of the work steps that represent the processes when the predetermined work performed by the worker, as captured in the acquired video data, is divided into multiple processes; The acquired video data is divided into work steps specified by the start and end times determined by the judgment result, and display data is generated for each work step and for each worker to display the work status of each worker. A method for performing the steps of outputting the generated display data.

Images