Work analysis device and work analysis method

Abstract

This work analysis device: is capable of accessing a first database that stores an analysis algorithm for detecting the state of work and a second database that stores a work schedule result for each work process of a series of work processes constituting the work; acquires first video data indicating the series of work processes; acquires the analysis algorithm from the first database; acquires the work schedule result of the work process from the second database; detects the state of work in the first video data on the basis of the analysis algorithm; generates a work process completion determination indicating a boundary between consecutive work processes in the series of work processes on the basis of the detection result and the work schedule result; and registers the work process completion determination in the first database.

Description

Work analysis device and work analysis method Incorporation by reference

[0001] This application claims the priority of Japanese Patent Application No. 2024-158449, which was filed on September 12, 2024, and incorporates its content by reference.

[0002] The present invention relates to a work analysis device and a work analysis method for analyzing work.

[0003] In the manufacturing site of a factory, the number of workers, especially skilled workers, is decreasing due to the declining birthrate and aging population. The number of makeshift workers is also decreasing, and there is a problem that on-site work cannot be maintained unless unskilled workers or foreign workers with different native languages are hired. In addition, quality defects are detected and recalls after product shipment are also a problem, and it is important to avoid a decline in the manufacturing quality of products. For this reason, in the production site of a factory, in order to reduce the burden on workers and improve production efficiency, many automation technologies such as industrial robots and process automation systems have been put into practical use.

[0004] However, not all work in the manufacturing site of a factory can be automated, and there are still processes that require human work. The reasons are various, such as the work being too complex, manufacturing in small quantities with many varieties, or being too trivial to automate, but the main factor is that the cost of automation is much higher than the labor cost. For this reason, various measures are taken to reduce the burden on the remaining workers in the manufacturing site of a factory and maintain or improve production efficiency. In addition, in response to the recent production in small quantities with many varieties and changes in the manufacturing site, such measures need to be able to keep up.

[0005] For example, Patent Document 1 discloses a work improvement support system that assists in improving work processes. This work improvement support system comprises: a performance management unit that manages performance data for each task included in the work process; a planning management unit that manages planning data for each task; a work video management unit that manages work video data of the work site where each task is performed by a worker; and a work analysis unit that determines whether a task falls under a predetermined task that should be improved based on the performance data and planning data for the task, and if it is determined that the task falls under a predetermined task, extracts predetermined work video data from the work video data relating to the task that was determined to fall under a predetermined task.

[0006] Furthermore, the work instruction system of Patent Document 2 comprises a computer that generates a work instruction screen and a display device connected to the computer that displays the work instruction screen, the work instruction screen includes a first display area that displays a three-dimensional model based on three-dimensional design data of the object, a second display area that displays at least a part of the attribute information of the three-dimensional design data, and a third display area that displays related information associated with predetermined part information if predetermined part information that has been registered in advance is included in the attribute information.

[0007] Furthermore, the work support device described in Patent Document 3 includes a total work time calculation means and an effective operation time calculation means. The total work time calculation means calculates the total work time from the start to the end of the work for a worker performing a certain process. The effective operation time calculation means calculates the effective operation time as the time during which the worker performs actions that are effective in progressing the process, out of the total work time.

[0008] Furthermore, the work support device described in Patent Document 4 comprises a workload evaluation unit, an improvement measure storage unit, a simulation unit, and an improvement measure determination unit. The workload evaluation unit evaluates the workload, which indicates the degree of burden placed on the worker by the work, from worker information and production equipment information. Worker information is information about workers who work in cooperation with production equipment. Production equipment information is information about production equipment. The improvement measure storage unit stores improvement measures to improve the workload. The simulation unit performs a simulation to estimate the estimated improvement effect, which indicates the degree of improvement in the workload when the improvement measures are applied. Based on the estimated improvement effect, the improvement measure determination unit determines the applicable improvement measures to be applied to at least one of the worker and the production equipment.

[0009] Japanese Patent Publication No. 2019-23803, International Publication No. 2019 / 077861, Japanese Patent Publication No. 2018-165893, International Publication No. 2020 / 217381

[0010] Patent Document 1 states that the end of a work process must be explicitly indicated by the worker using a complete button or next button, and the system cannot automatically determine the end of a work process.

[0011] Patent Document 2 contains the same problems as Patent Document 1, and therefore cannot display 3D models or attribute information only at timings that match the work process that the worker is not good at.

[0012] Patent Document 4 evaluates the workload of workers on a timestamp basis, making it difficult to determine which work process within the assembly work is causing the burden.

[0013] Furthermore, none of the documents, including Patent Document 3, provide a mechanism for determining whether work results are correct or for linking work processes with the possibility of manufacturing defects. Also, none of the documents provide a mechanism for continuously expanding the work support system.

[0014] The present invention aims to facilitate the identification of demarcation points between consecutive work processes.

[0015] The work analysis device of the disclosed technology is a work analysis device having a processor that executes a program and a storage device that stores the program, and is able to access a first database that stores an analysis algorithm for detecting the state of work and a second database that stores work schedule results for each work process of a series of work processes that constitute the work, and the processor is characterized in that it executes a first acquisition process for acquiring first video data showing the series of work processes, a second acquisition process for acquiring the analysis algorithm from the first database, a third acquisition process for acquiring work schedule results for the work processes from the second database, a detection process for detecting the state of work in the first video data acquired by the first acquisition process based on the analysis algorithm acquired by the second acquisition process, a generation process for generating a work process completion determination indicating the boundary between consecutive work processes in the series of work processes based on the detection result from the detection process and the work schedule results acquired by the third acquisition process, and a registration process for registering the work process completion determination generated by the generation process in the first database.

[0016] According to a typical embodiment of the present invention, it is possible to facilitate the identification of demarcations between consecutive work processes. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

[0017] Figure 1 is an explanatory diagram showing an example of the system configuration of the work support system. Figure 2 is a block diagram showing an example of the hardware configuration of the computer shown in Figure 1. Figure 3 is an explanatory diagram showing an example of an analysis algorithm database (hereinafter referred to as DB). Figure 4 is an explanatory diagram showing an example of a work process DB. Figure 5 is an explanatory diagram showing an example of a work performance DB. Figure 6 is an explanatory diagram showing an example of a work support DB. Figure 7 is a sequence diagram showing work analysis development process 1. Figure 8 is a sequence diagram showing work analysis development process 2. Figure 9 is an explanatory diagram showing an example of generating a work process completion determination in step S812. Figure 10 is a sequence diagram showing work support processing. Figure 11 is a flowchart showing a detailed example of the processing procedure of the work analysis process (step S1006) shown in Figure 10. Figure 12 is an explanatory diagram showing an example of the display of the management screen. Figure 13 is an explanatory diagram showing an example of the display of the individual management screen. Figure 14 is an explanatory diagram showing an example of the display of the work support screen.

[0018] <Figure 1 Work Support System> Figure 1 is an explanatory diagram showing an example of the system configuration of a work support system. The work support system 100 includes a library DB management device 101, a work analysis device 102, a work support device 103, and a DB device 104. The library DB management device 101, the work analysis device 102, the work support device 103, and the DB device 104 are connected to each other via a network 108 such as the Internet, LAN (Local Area Network), or WAN (Wide Area Network).

[0019] Although the library DB management device 101, work analysis device 102, work support device 103, and DB device 104 are each independent computers, two or more of the library DB management device 101, work analysis device 102, work support device 103, and DB device 104 may be configured on a single computer.

[0020] Furthermore, the work support system 100 is connected to the management terminal 105, development terminal 106, and field equipment 107 via the network 108, enabling communication between them.

[0021] The library DB management device 101 manages the libraries used in the work support system 100. The library DB management device 101 includes a work support library DB 111, a work process library DB 112, and an analysis algorithm library DB 113.

[0022] The DB111 work support library stores various work support libraries. A work support library is a set of software components, such as functions and classes, that define various support methods used for work support.

[0023] The work process library DB112 is a database that stores work process information libraries for assembly work, which show the procedures for assembling products for various purposes.

[0024] The analysis algorithm library DB113 is a database that stores various analysis algorithm libraries. An analysis algorithm library consists of software components such as functions and classes that make up an analysis algorithm.

[0025] An analysis algorithm is an algorithm that analyzes the object of analysis. The object of analysis is the actions of worker W, the work object 174 and its anomaly detection, and the work process of the assembly work. For example, an analysis algorithm may include an action detection algorithm to detect the actions of worker W, an object detection algorithm to detect the work object 174, an anomaly detection algorithm to detect anomalies in the work object 174, and a process determination algorithm to determine the work process of the assembly work that is actually being performed.

[0026] The library DB management device 101 extracts work support libraries from the work support library DB 111 and stores them in the work support DB 144 of the DB device 104. The library DB management device 101 also extracts analysis algorithm libraries from the analysis algorithm library DB 113 and stores them in the analysis algorithm DB 141.

[0027] The work analysis device 102 performs progress analysis of the assembly work performed by worker W. Specifically, for example, the work analysis device 102 performs tasks such as identifying worker W and the work object 174, detecting worker W's movements, detecting the work object 174, generating display data, recording data to the DB device 104, and controlling the management terminal 105, development terminal 106, and on-site equipment 107.

[0028] The work support device 103 generates work support information and transmits the generated work support information to the field device 107. The work support device 103 registers the generated work support information in the DB device 104.

[0029] The DB device 104 registers data from the library DB management device 101, the work analysis device 102, the work support device 103, the management terminal 105, the development terminal 106, and the field device 107, and reads data to the library DB management device 101, the work analysis device 102, the work support device 103, the management terminal 105, the development terminal 106, and the field device 107.

[0030] The DB device 104 includes an analysis algorithm DB 141, a work process DB 142, a work performance DB 143, and a work support DB 144.

[0031] The analysis algorithm DB 141 is a database that stores analysis algorithm information. The analysis algorithm information includes analysis algorithm libraries from the library DB management device 101.

[0032] The work process DB 142 is a database that stores some work process information read from the work process library DB 112. The work performance DB 143 is a database that stores the work performance information of worker W. The work support DB 144 is a database that stores work support information for assembly work. The work support information includes the work support library from the library DB management device 101.

[0033] The management terminal 105 is a computer used by administrator M to manage the work support system 100. The management terminal 105 is installed, for example, in the management booth 150. The management terminal 105 reads the relevant work support information from the work support library DB 111 via the library DB management device 101 and registers it in the work support DB 144.

[0034] Development terminal 106 is a computer used by developer D to develop or adjust the work support system 100. Development terminal 16 is installed, for example, in the development booth 160. Development terminal 16 applies the work support system 100 to the field equipment 107. Developer D may be a person who provides the work support system 100 or a person who uses the work support system 100.

[0035] The development terminal 106 determines the work object 174 to be assembled in the work booth 170 through operations from developer D. The development terminal 106 also reads the assembly procedure information necessary for product assembly from the work process library DB 112 through operations from developer D and registers it in the work process DB 142. The development terminal 106 also reads the analysis algorithm library necessary for process determination of product assembly from the analysis algorithm library DB 113 through operations from the library DB management device 101 and registers it in the analysis algorithm DB 141.

[0036] The on-site device 107 is a computer that supports the assembly work performed by worker W in the work booth 170. The on-site device 107 is installed in the work booth 170. Worker W assembles the target product in the work booth 170, using tools 173 such as screwdrivers and wrenches as needed on the work object 174.

[0037] Furthermore, the field device 107 is connected to the sensor 171. The sensor 171 detects the work process of worker W within the work booth 170. Specifically, for example, the sensor 171 is a camera that photographs the work process and generates work process video data. The sensor 171 may also be a microphone that collects sound during the work process, or a wearable sensor that detects the movements of worker W. Alternatively, the sensor 171 may be a power tool such as an electric screwdriver, a vibration sensor, or a temperature sensor. In addition, other means may be used as long as they are means to understand the work process of worker W. Furthermore, there may be one or more means corresponding to the sensor 171.

[0038] Furthermore, the field device 107 includes a display or projector for displaying work methods, a speaker for conveying work methods by voice, and a vibration sensor for transmitting work timing and movement patterns. Other means may be used as long as they are means of transmitting work methods, timing, and movement patterns. The field device 107 may have one or more means.

[0039] <Figure 2 Example of Hardware Configuration of Computer (Work Analysis Device 102, Work Support Device 103, DB Device 104, Management Terminal 105, Development Terminal 106, and Field Device 107)> Figure 2 is a block diagram showing an example of the hardware configuration of the computer shown in Figure 1. The computer 200 includes a processor 201, a storage device 202, an input device 203, an output device 204, and a communication interface (communication IF) 205. The processor 201, storage device 202, input device 203, output device 204, and communication IF 205 are connected by a bus 206. The processor 201 controls the computer 200. The storage device 202 is the work area of ​​the processor 201. The storage device 202 is also a non-temporary or temporary recording medium that stores various programs and data. Examples of memory devices 202 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. Input devices 203 input data. Examples of input devices 203 include keyboards, mice, touch panels, numeric keypads, scanners, microphones, and sensors. Output devices 204 output data. Examples of output devices 204 include displays, printers, and speakers. Communication IF 205 connects to network 108 and sends and receives data.

[0040] <Figures 3-6 Databases> Next, we will specifically explain the analysis algorithm DB141 (Figure 3), work process DB142 (Figure 4), work performance DB143 (Figure 5), and work support DB144 (Figure 6) shown in Figure 1.

[0041] [Figure 3 Analysis Algorithm DB 141] Figure 3 is an explanatory diagram showing an example of the analysis algorithm DB 141. The analysis algorithm DB 141 has the following fields: product name 301, work location 302, work environment 303, motion detection algorithm 304, object detection algorithm 305, anomaly detection algorithm 306, and process determination algorithm 307.

[0042] The product name 301 is the name of the work object 174 which is a product. The work location 302 is the site where there is a work booth 170 where the worker W assembles the work object 174 which is a product.

[0043] The working environment 303 indicates the working environment for assembling the work object 174 which is a product. For example, it is an indoor environment with an illuminance of 300 lux only by lighting fixtures at night and a sound pressure of 60 decibels. Even within the same work location 302, the parameters and models of various detection algorithms may vary depending on the lighting conditions and noise conditions during daytime and night time, which can serve as a reference for appropriate algorithm selection.

[0044] The motion detection algorithm 304 is an algorithm for detecting the motion of the worker W during the assembly work on the work object 174. Depending on the work object 174, the detection target of the motion detection algorithm 304 is different. For example, there are algorithms for detecting the motion of fingers with respect to the work motion of the worker W, algorithms for performing skeleton detection of the motion of the whole body, and algorithms for detecting the walking manner of the worker W.

[0045] The object detection algorithm 305 is an algorithm for detecting the presence or absence of the work object 174 and the tool 173 used in the work. For example, there is an algorithm for detecting states such as the work object 174 being placed on, or removed from, the workbench of the work booth 170.

[0046] Also, there are algorithms for detecting situations such as the tool 173 used when assembling the work object 174, for example, a driver or a wrench, being on the work object 174 (that is, during the work of screwing or nutting using the tool 173), or being placed beside the work object 174 (that is, during the manual assembly work without using the tool 173).

[0047] The abnormality detection algorithm 306 is an algorithm for detecting the presence or absence of abnormalities such as scratches on the surface of the workpiece 174 or defects in the shape, and is used to determine the likelihood of product defects 509. For example, it includes algorithms such as image processing such as noise removal and edge extraction of the image data acquired by the sensor 171, image correction for changing contrast and brightness, feature extraction of figures included in the image data, pattern classification and character recognition, and machine learning of these.

[0048] Any of the motion detection algorithm 304, the object detection algorithm 305, and the abnormality detection algorithm 306 may be used as the data in the analysis algorithm library DB113 as it is, but the development terminal 106 may be customized so that appropriate analysis can be performed according to the workpiece 174, the work location 302, and the work environment 303 by the operation of the developer D.

[0049] The process determination algorithm 307 is an algorithm for determining the boundaries of each procedure (assembly work process) obtained by decomposing the procedure for assembling the workpiece 174 for each work process, and it varies depending on the workpiece 174 and the work process. For example, if it is premised that the "inverter device of vehicle type A" is assembled in 4 processes, the algorithm 307 can be expressed as an algorithm showing the boundaries of those 4 processes, and if it is premised that it is assembled in 10 processes, it can be expressed as an algorithm showing the boundaries of those 10 processes. That is, it varies depending on the type of the workpiece 174 and the number of work processes.

[0050] The process determination algorithm 307 is prepared for each workpiece 174. Specifically, for example, the development terminal 106 selects the process determination algorithm 307 so that appropriate analysis can be performed according to the target work location 302 and the workpiece 174 based on the motion detection algorithm 304, the object detection algorithm 305, and the planned work result 403.

[0051] By using the process determination algorithm 307, the work analysis system 110 can automatically determine how far worker W has progressed with the work. In a simple example, if the work procedure 401 is digitized and displayed to worker W via a screen, conventionally, worker W would have had to switch to the next work process screen using the "Complete" or "Next" button at the end of each work process. However, by using the process determination algorithm 307, the work analysis system 110 can automate the screen switching operation. Furthermore, the work analysis system 110 can determine whether or not there are assembly errors in each work process.

[0052] [Figure 4 Work Process DB 142] Figure 4 is an explanatory diagram showing an example of the Work Process DB 142. The Work Process DB 142 is a database that stores information about work processes. The Work Process DB 142 has the following fields: Product Name 301, Work Procedure 401, Target Work Time 402, and Planned Work Result 403. The combination of values ​​of each field in the same row becomes an entry that defines the work process information for the work object 174 of the product name 301.

[0053] Work procedure 401 shows a series of work steps broken down into convenient sections for assembling the work object 174 of product name 301, which is the object to be assembled. Target work time 402 is the target work time set for each step, which is the work time in which each step should be completed. Planned work result 403 is information showing the state of the work object 174 of product name 301 during assembly after each step has been completed.

[0054] The planned work result 403 is the ideal work result for each work step of the work procedure 401. For example, for each work step defined in the work procedure 401, it may be video data such as photographs or drawings showing the work result for the work object 174, or a string of characters indicating the work result. It may also include a time-series operation pattern identified by the cluster number that appears in the k-means method described later in Figure 9. The planned work result 403 is set in the motion detection algorithm 304 and the object detection algorithm 305, respectively.

[0055] [Figure 5 Work Performance DB 143] Figure 5 is an explanatory diagram showing an example of the Work Performance DB 143. The Work Performance DB 143 is a database that stores information about work performance. The Work Performance DB 143 has the following fields: product name 301, manufacturing number 501, worker 502, work procedure 401, work process start time 503, work process end time 504, work process overtime 505, work process video data 506, process result discrepancy 507, work defect possibility 508, and product defect possibility 509. The combination of values ​​for each field in the same row becomes an entry that defines the work performance information for the work object 174 of the product name 301 and manufacturing number 501.

[0056] The serial number 501 is identification information that uniquely identifies an individual workpiece 174, which is specified by the product name 301.

[0057] The worker 502 is identification information that uniquely identifies the worker W performing work on the object 174, which is identified by the product name 301 and the manufacturing number 501.

[0058] The work process start time 503 is the time when the process specified in the work procedure 401 begins. The work process start time 503 may include the date.

[0059] The work process completion time 504 is the time when the process specified in the work procedure 401 is completed. The work process completion time 504 may include the date.

[0060] The work process overtime 505 is the time during which the actual work time for the process defined in the work procedure 401 exceeds the target work time 402. The actual work time is the time obtained by subtracting the work process start time 503 from the work process end time 504, and the work process overtime 505 is the time obtained by subtracting the target work time 402 from the actual work time.

[0061] The work process video data 506 is video data of the work process captured by the sensor 171 from the start time 503 to the end time 504 of the work process. The work process video data 506 may also include audio. Furthermore, the work process video data 506 may be the data itself, or it may be access information (for example, a directory) to the storage location of the work process video data 506.

[0062] Furthermore, the work process video data 506 may not be recorded for each work procedure 401, but rather in fixed time units, such as 2 minutes. Even in that case, it is possible to associate the work process video data 506 with the work procedure 401 by matching the work process start time 503 with the timestamp of the work process video data 506.

[0063] The discrepancy in the process result 507 is information indicating whether or not the final state of the work process identified in the work process video data 506 was determined to be inconsistent with the planned work result 403 (assembly error). For example, if the final state of the work process identified in the work process video data 506 is inconsistent with the planned work result 403 even once, "1" is recorded; if there is no discrepancy even once, "0" is recorded.

[0064] The possibility of work defects 508 is information indicating whether or not there is a possibility of work defects in the process identified in the work procedure 401. Specifically, for example, the possibility of work defects 508 is information indicating whether or not there is a defect in the assembly state of the work object 174 as the final state of the work process identified in the work process video data 506. If there is a defect, "1" is recorded; otherwise, "0" is recorded. If the work process overtime 505 is above the threshold, or if the discrepancy in the process result 507 is "1", then there is a defect and it is recorded as "1".

[0065] The item defect possibility 509 is information indicating whether or not the workpiece 174 may have an item defect. Specifically, for example, the item defect possibility 509 is information indicating whether or not there is an item defect, such as scratches on the surface of the workpiece 174 or defects in its shape, as the final state of the work process identified in the work process video data 506. If there is an item defect, "1" is recorded; otherwise, "0" is recorded.

[0066] [Figure 6 Work Support DB 144] Figure 6 is an explanatory diagram showing an example of the Work Support DB 144. The Work Support DB 144 is a database that stores information about work performance. The Work Support DB 144 has the following fields: Work Support Type 601, Target Work 602, Work Support Algorithm 603, Support Target 604, and Support Performance 605. The combination of values ​​of each field in the same row becomes an entry that defines the work support information.

[0067] Work support type 601 is the name of a type of work support that assists in work processes that the worker 502 is not proficient in. For example, "display-type assembly navigation" includes "assembly navigation using a display," "assembly navigation using an aerial display," and "assembly navigation using projection mapping."

[0068] The target task 602 is the task covered by the work support type 601. For example, if the work support type 601 is "display-type assembly navigation," then the target task 602 might be, for example, "assembly of the inverter device for vehicle type A."

[0069] The work support algorithm 603 is a specific algorithm used for work support specified by the work support type 601. The work support algorithm 603 may, for example, use a library from the work support library DB 111 as is, or it may be updated to suit the site through work analysis and development processing.

[0070] Examples of the types of work support algorithms 603 include "assembly procedure projection," which projects an outline of the assembly procedure for each process; "assembly error detection," which determines whether or not there are assembly errors in each process; "remote site digital connection," which allows for online connection with experts in remote locations to mutually understand the work situation in case of any problems during assembly work and to request assistance; and "assembly procedure caution projection," which displays details and points to note regarding assembly.

[0071] The support target 604 is the target of the work support specified by the work support type 601, such as a work line or worker 502. If the work support algorithm 603 is such as "assembly procedure projection," "assembly error detection," or "remote site digital connection," it is beneficial to all workers, so all workers are set as the support target 604.

[0072] On the other hand, the "projection of points to note in the assembly procedure" work support algorithm 603 is effective for inexperienced workers, but for skilled workers, it may be redundant and even hinder their work. In the example in Figure 6, in order to apply the "projection of points to note in the assembly procedure" work support algorithm 603 only to worker W2, the support target 604 is set to "worker W2".

[0073] Support record 605 is actual information on work support, such as the time, number of times, and duration of work support applied, as specified by work support type 601. By using support record 605 together with support target 604 and support record 605, it becomes easier to set up variations such as subscription-type services and pay-per-use services.

[0074] The registration of work support information from the work support library DB 111 to the work support DB 144 is performed by the management terminal 105. For example, the management terminal 105, at the operation of administrator M, sets "Display-type assembly navigation" as the work support type 601 for the target work 602, "Assembly of inverter device for vehicle type A". The management terminal 105 also, at the operation of administrator M, registers "Assembly procedure projection", "Assembly error judgment", and "Remote site digital connection" as the work support algorithm 603 for all workers who are the support targets 604, and registers "Projection of points to note in the assembly procedure" for a specific worker W2. This makes it possible to provide work support according to the work support algorithm 603 to the corresponding support targets 604.

[0075] Furthermore, the management terminal 105 can also delete entries in the work support DB 144 through the operation of administrator M.

[0076] <Figure 7 Work Analysis Development Process 1> Figure 7 is a sequence diagram showing Work Analysis Development Process 1. Work Analysis Development Process 1 is a process that develops a work analysis specific to the work object 174 by setting work process information and analysis algorithms according to the work object 174. The Work Analysis Development Process is performed, for example, by developer D in the development booth 20.

[0077] (Step S701) The development terminal 106 receives a designation of the work object 174 to be assembled in the work booth 30, based on the operation of developer D. The designated work object 174 is referred to as the designated work object 174.

[0078] (Step S702) The development terminal 106 obtains a work support algorithm 603 related to the designated work object 174. Specifically, for example, the development terminal 106 identifies the designated work object 174 from the target work 602. The development terminal 106 selects a work support algorithm 603 corresponding to the identified target work 602 at the operation of developer D and requests it from the library DB management device 101. The library DB management device 101 reads the work support algorithm 603 requested by the development terminal 106 from the work support library DB 111 and returns it to the development terminal 106. As a result, developer D can confirm the work support algorithm 603 on the development terminal 106.

[0079] (Step S703) The development terminal 106 registers the work support algorithm 603 acquired in step S702 in the work support algorithm DB 144 of the DB device 104. If it is not necessary for developer D to verify the acquired work support algorithm 603, the library DB management device 101 may register the acquired work support algorithm 603 in the work support algorithm DB 144 without returning the work support algorithm 603 acquired in step S702 to the development terminal 106.

[0080] (Step S704) The development terminal 106 acquires work process information related to the designated work object 174. Specifically, for example, the development terminal 106 requests work process information related to the designated work object 174 from the library DB management device 101. The library DB management device 101 reads the work process information requested by the development terminal 106 from the work process library DB 112 and returns it to the development terminal 106. As a result, developer D can confirm the work process information on the development terminal 106.

[0081] (Step S705) The development terminal 106 registers the work process information acquired in step S704 (hereinafter referred to as acquired work process information) in the work process DB 142 of the DB device 104. If it is not necessary for developer D to confirm the acquired work process information, the library DB management device 101 may register the acquired work process information in the work process DB 142 without returning the acquired work process information to the development terminal 106 in step S704.

[0082] (Step S706) The development terminal 106 obtains analysis algorithms (motion detection algorithm 304, object detection algorithm 305, anomaly detection algorithm 306) related to the designated work object 174. Specifically, for example, the development terminal 106 requests the analysis algorithms related to the designated work object 174 from the library DB management device 101. The library DB management device 101 reads the analysis algorithms requested by the development terminal 106 from the work process library DB 112 and returns them to the development terminal 106. This allows the developer D to confirm the analysis algorithms on the development terminal 106.

[0083] (Step S707) The development terminal 106 registers the analysis algorithms (motion detection algorithm 304, object detection algorithm 305, anomaly detection algorithm 306) acquired in step S706 into the analysis algorithm DB 141 of the DB device 104. If confirmation of the acquired analysis algorithms by developer D is not required, the library DB management device 101 may register the acquired analysis algorithms in the analysis algorithm DB 141 without returning them to the development terminal 106 in step S706.

[0084] <Figure 8 Work Analysis Development Process 2> Figure 8 is a sequence diagram showing Work Analysis Development Process 2. Work Analysis Development Process 2 is a process that customizes the analysis algorithm set in Work Analysis Development Process 1 in order to automatically determine the work process of a product in the work process information set in Work Analysis Development Process 1, and is executed by the development terminal 106, work analysis device 102, field device 107, and DB device 104.

[0085] (Step S801) The development terminal 106 receives an assembly work start instruction via operation by developer D and transmits the assembly work start instruction to the work analysis device 102 and the field device 107. The assembly work start instruction includes the product name 301, serial number 501, worker 502, and work procedure 401 of the work object 174 that is the subject of work analysis development.

[0086] (Step S802) The field device 107 identifies entries from the work record DB 143 that correspond to the product name 301, serial number 501, worker 502, and work procedure 401 of the work object 174 included in the assembly work start instruction. For example, if the product name 301 is "Product A", the serial number 501 is "A001", and the worker 502 is "W1", then entries 511-1 to 511-n with work procedures 401 "PA1", "PA2", ..., "PAn" are identified. If entries 511-1 to 511-n are not distinguished, they are referred to as entry 511.

[0087] (Step S803) When the on-site device 107 receives a signal from worker W to start filming, it starts filming the work booth 170 using the sensor 171. If entry 511 is identified, the start time of the work process 503 is recorded in entry 511 of the work performance DB 143 upon the start of filming. Also, in the work booth 170, worker W performs process "PA1" as part of the assembly work of the work object 174, as specified in work procedure 401. As a result, video data 506 of the work process in the work booth 170 is recorded in the work performance DB 143.

[0088] (Step S804) When the currently working process is completed, the field device 107 notifies the DB device 104, the work analysis device 102, and the development terminal 106 of the completion of the process at the operation of worker W. Upon completion of the process, the work process completion time 504 is recorded in entry 511 of the work performance DB 143.

[0089] (Step S805) When the work analysis device 102 receives a request from worker W to stop the shooting, it stops the sensor 171 from shooting the work booth 170.

[0090] (Step S806) Steps S802 to S805 are repeated until the work for all entries 511 is completed. When the work for entries 511-n is completed, the field device 107 sends a completion notification to the development terminal 106 at the operation of worker W.

[0091] (Step S807) When the development terminal 106 receives the completion notification, developer D operates it to send a development instruction to the work analysis device 102.

[0092] (Step S808) When the work analysis device 102 receives a development instruction, it acquires the work process video data 506 obtained in steps S802 to S806 from the work performance DB 143 of the DB device 104.

[0093] (Step S809) The work analysis device 102 obtains an analysis algorithm from the analysis algorithm DB 141. Specifically, for example, the work analysis device 102 sends a request to the DB device 104 to obtain an analysis algorithm that includes the product name 301 of the work object 174, the work location 302 of the worker W, and the work environment 303 of the work location 302. The DB device 104 reads the motion detection algorithm 304 and the object detection algorithm 305 from the analysis algorithm DB 141 as the analysis algorithms for the entry corresponding to the acquisition request and sends them to the work analysis device 102.

[0094] (Step S810) The work analysis device 102 performs assembly motion detection and object detection on the work process video data 506 acquired in step S808 using the analysis algorithms (motion detection algorithm 304, object detection algorithm 305) acquired in step S809. Specifically, for example, the work analysis device 102 detects the assembly motion of worker W over time from the work process video data 506, and detects changes in the state of objects such as the tools 173 used by worker W and the work object 174 over time.

[0095] (Step S811) The work analysis device 102 obtains the work plan result 403 of the work process information that matches the product name 301 and work procedure 401 for the work performed in steps S802 to S806 from the work process DB 142. For example, the work analysis device 102 obtains the target work time 402 and the work plan result 403 of the work process information that matches the product name 301 and work procedure 401 of entry 511 from the work process DB 142.

[0096] (Step S812) The work analysis device 102 associates the assembly operation detection result and object detection result from step S810 with the work schedule result 403 from step S811 to generate a work process completion determination that signifies the end of a work process. The work analysis device 102 registers the generated work process completion determination in the process determination algorithm 307 of the analysis algorithm DB 141.

[0097] [Figure 9 Example of Work Process Completion Determination Generation] Figure 9 is an explanatory diagram showing an example of the generation of a work process completion determination in step S812. In Figure 9, for example, in a work booth 170, worker W is assumed to be performing "assembly work of an inverter device for vehicle type A," assembling a work object 174 (product name 301: "Product A") with dimensions of approximately 30 cm wide, 20 cm deep, and 10 cm high, using multiple parts and fastening them with tools such as screwdrivers 173, thereby completing the product (work processes PA1 to PA4 as an example).

[0098] (Generating work process completion determination by motion detection algorithm 304) The work analysis device 102 detects the skeleton of worker W, such as the hands and arms, for each work process using the motion detection algorithm 304, and detects motion patterns that show the chronological movement of the skeleton. The work analysis device 102 classifies the detected motion patterns of worker W and estimates what kind of work is currently being performed.

[0099] The k-means method is a common analytical technique for clustering and classification. The k-means method can be formulated as a problem of minimizing a value called SSE (Sum of Squared Errors of Prediction) while varying the number of clusters, a method called the elbow method. The value that is considered to have a small change in SSE in the elbow method becomes the expected cluster number in the k-means method, and the worker W's behavioral patterns are classified by the cluster number.

[0100] Graph 901 consists of a horizontal axis representing time and a vertical axis representing cluster number, showing the temporal change in the number of clusters that indicate the operating pattern. The plotted black circles represent the operating pattern corresponding to the cluster number.

[0101] Graph 901 in Figure 9 shows a plot of the operation pattern over time when the number of clusters is 4 (cluster numbers 1 to 4). For each work step of the work procedure 401, the work analysis device 102 determines whether the operation pattern over time detected by the skeletal detection by the operation detection algorithm 304 matches the planned work result 403. If the operation pattern detected by the skeletal detection matches the planned work result 403, the work analysis device 102 determines that the work step has been completed.

[0102] First, the work analysis device 102 performs estimation of work process PA1. For example, work process 911-1 is a time-series operation pattern in which cluster number 4 appears, followed by cluster numbers 1 and 3. The work analysis device 102 compares work process 911-1 with the planned work result 403 of work process PA1 and calculates the agreement rate.

[0103] The agreement rate is the percentage of time-series operation patterns that match the planned work results 403. For example, the denominator is the total number of clusters in the planned work results 403, and the numerator is the number of clusters that match the time-series operation patterns in the planned work results 403. In this example, it is assumed that an agreement rate of 30% is calculated. If the agreement rate is less than a threshold (for example, 70%), the work analysis device 102 continues to detect cluster numbers. The threshold may be the same across work processes or may differ from work process to work process.

[0104] The work analysis device 102 advances the time of the assembly operation detection result detected in step S810. When it detects the appearance of cluster numbers 1, 3, and 4, it compares them with the planned work result 403 of work process PA1 as work process 911-2, which includes work process 911-1, and recalculates the agreement rate. Similarly, the work analysis device 102 advances the time of the assembly operation detection result detected in step S810 until the agreement rate exceeds a threshold, generating work process 911-3 (including work process 911-2), work process 911-4 (including work process 911-3), and so on. In Figure 9, it is assumed that the agreement rate reached 70% in work process 911-4. Since the agreement rate reached above the threshold in work process 911-4, the work analysis device 102 moves on to estimating the next work process PA2 after work process PA1.

[0105] For work processes PA2 and beyond, the work analysis device 102 performs estimation of work processes PA2 to PA4, similar to work process PA1. For example, the work analysis device 102 estimates work process PA2 as work process 912-5, work process PA3 as work process 913-4, and work process PA4 as work process 914-2.

[0106] In other words, the work analysis device 102 determines the end times of work processes 911-4, 912-5, 913-4, and 914-2 as work process completion determinations 931 to 934, which represent the division between work processes PA1 to PA4.

[0107] (Determination of work process completion by object detection algorithm 305) The work analysis device 102 also uses object detection by object detection algorithm 305 to detect the position of tools 173 such as screwdrivers and the shape of the work object 174 (including the state in which it is being assembled) as a time-series state pattern of the object for each work process.

[0108] Graph 902 consists of a horizontal axis representing time and a vertical axis representing cluster numbers, showing the temporal change in cluster numbers that indicate the state patterns of objects. The plotted black circles represent cluster numbers that indicate the state patterns of objects.

[0109] Graph 902 in Figure 9 shows a plot of the object state pattern over time when the number of clusters (cluster numbers 1 to 4) is set to 4. For each work step of the work procedure 401, the work analysis device 102 determines whether the object state pattern over time detected by the object detection algorithm 305 matches the planned work result 403. If the object state pattern over time detected by object detection matches the planned work result 403, the work analysis device 102 determines that the work step has been completed.

[0110] The work analysis device 102 also performs estimation of work process PA1 using the object detection algorithm 305, similar to the motion detection algorithm 304. For example, work process 921-1 is a time-series object state pattern in which cluster number 1 appears followed by cluster number 2. The work analysis device 102 compares work process 921-1 with the work plan result 403 for work process PA1 and calculates the agreement rate.

[0111] The agreement rate is the percentage of time the object's state pattern over time matches the planned work result 403. For example, the denominator is the total number of clusters in the planned work result 403, and the numerator is the number of clusters that match the object's state pattern over time in the planned work result 403. In this example, it is assumed that an agreement rate of 35% was calculated. If the agreement rate is less than a threshold (for example, 80%), the work analyzer 102 continues to detect cluster numbers. The threshold may be the same across work processes or may differ from work process to work process.

[0112] The work analysis device 102 advances the time of the object detection result detected in step S810. When it detects the appearance of cluster numbers 1 and 3, it compares this with the planned work result 403 of work process PA1 as work process 921-2, which includes work process 921-1, and recalculates the agreement rate. Similarly, the work analysis device 102 advances the time of the assembly operation detection result detected in step S810, generating work processes 921-3 (including work process 921-2), ... until the agreement rate exceeds a threshold. In Figure 9, it is assumed that the agreement rate in work process 921-3 is 80%. Since the agreement rate in work process 921-3 is above the threshold, the work analysis device 102 moves on to estimating the next work process PA2 after work process PA1.

[0113] For work processes PA2 and beyond, the work analysis device 102 performs estimation of work processes PA2 to PA4, similar to work process PA1. For example, the work analysis device 102 estimates work process PA2 as work process 922-2, work process PA3 as work process 913-2, and work process PA4 as work process 924-2.

[0114] In other words, the work analysis device 102 determines the end times of work processes 921-3, 912-2, 913-2, and 914-2 as work process completion determinations 941 to 944, which represent the division between work processes PA1 to PA4.

[0115] In addition, in the generation of the work process completion determination (step S812), at least one of the motion detection algorithm 304 (graph 901) and the object detection algorithm 305 (graph 902) is applied.

[0116] For example, by combining the motion detection algorithm 304's time-dependent motion pattern with the object detection algorithm 305's time-dependent object state pattern, the accuracy of generating work process completion determinations is improved compared to using either algorithm alone.

[0117] In this combination, if the agreement rate of both work processes exceeds a threshold, the work analysis device 102 determines the end time of the later work process that exceeds the threshold as the work process completion determination, which signifies the end of that work process. For example, the end time of work process 911-4 is later than the end time of work process 921-3. Therefore, the work analysis device 102 determines the end time of work process 911-4 as the work process completion determination 931, which signifies the end of work process PA1, out of the end times of work process 921-3 and work process 911-4.

[0118] Furthermore, in this combination, if the agreement rate of either of the work processes exceeds a threshold, the work analyzer 102 may determine the end time of the work process that exceeds the threshold as the work process completion determination, which signifies the end of that work process. For example, the agreement rate of work process 921-3 exceeds the threshold before that of work process 911-4. Therefore, the work analyzer 102 determines the end time of work process 921-3 as the work process completion determination 941, which signifies the end of work process PA1.

[0119] With this combination, even if the agreement rate is too low for one of the algorithms to recognize that the work process has been completed, if the other algorithm obtains an agreement rate above a threshold, the work analysis device 102 will determine that the operation of the work process has been completed. In this way, a process determination algorithm is generated that determines the completion of a work process by comparing the time-series motion pattern obtained by skeleton detection and / or the time-series state pattern of an object obtained by the object detection algorithm 305 with the work schedule result 403. The work analysis device 102 records the process determination algorithm thus generated as the process determination algorithm 307 in the analysis algorithm DB 141. In this way, the preparation for automatic determination of work processes is completed.

[0120] <Figure 10 Work Support Processing> Figure 10 is a sequence diagram showing the work support processing. Work support processing is the process that supports the work of worker W. The analysis algorithm DB141 is configured with a process determination algorithm 307, which determines the completion of the work process, based on the processes shown in Figures 8 and 9.

[0121] (Step S1001) The work analysis device 102 transmits a shooting start instruction to the field device 107. The shooting start instruction includes work support information (entry in the work support DB 144) for worker W identified in the support target 604 of the work support DB, the product name 301, serial number 501, worker 502, and work procedure 401 of the work object 174 that worker W is working on. The shooting start instruction is transmitted to the field device 107 of the work booth 170 where worker W is working.

[0122] (Step S1002) The field device 107 starts taking pictures of the work booth 170 with the sensor 171. The work process video data obtained from the photography is recorded in the entry (product name 301, serial number 501, worker 502, and work procedure 401) identified by the instruction to start photography in the work performance DB 143.

[0123] (Step S1003) After step S1001, the work analysis device 102 acquires work process video data from the work performance DB 143 of the DB device 104.

[0124] (Step S1004) The work analysis device 102 obtains an analysis algorithm from the analysis algorithm DB 141. Specifically, for example, the work analysis device 102 sends a request to the DB device 104 to obtain an analysis algorithm that includes the product name 301 identified in step S1001, the work location 302 of the worker 502 identified in step S1001, and the work environment 303 of the work location. The DB device 104 reads the motion detection algorithm 304, object detection algorithm 305, anomaly detection algorithm 306, and process determination algorithm 307 from the analysis algorithm DB 141 as the analysis algorithms for the entry corresponding to the acquisition request, and sends them to the work analysis device 102.

[0125] (Step S1005) The work analysis device 102 acquires work process information. Specifically, for example, the work analysis device 102 sends a request to the DB device 104 to acquire work process information, including the product name 301 identified in step S1001. The DB device 104 reads the target work time 402 and planned work result 403 of the work procedure 401 corresponding to the product name 301 included in the work process information acquisition request from the work process DB 142 as work process information. The DB device 104 transmits the read work process information to the work analysis device 102.

[0126] (Step S1006) The work analysis device 102 performs work analysis processing. Work analysis processing involves analyzing the work process video data acquired in step S1003 using the analysis algorithm acquired in step S1004, and registering the analysis results in the work performance DB 143 or transmitting assembly defect information to the field device 107. Details of the work analysis processing (step S1006) will be described later in Figure 11.

[0127] (Step S1007) The field device 107 receives and displays assembly defect information from the work analysis device 102 through work analysis processing (Step S1006).

[0128] (Step S1008) When the work analysis device 102 has finished the work analysis process (Step S1006), it sends a stop instruction to the field device.

[0129] (Step S1009) When the field device 107 receives a stop command from the work analysis device 102, it stops taking images with the sensor 171, recording work process video data 506, and acquiring work process video data 506 from the DB device 104. This completes the work support process.

[0130] <Figure 11 Work Analysis Process> Figure 11 is a flowchart showing a detailed example of the work analysis process (step S1006) shown in Figure 10.

[0131] (Step S1101) The work analysis device 102 identifies the support target 604 from the work process video data 506. For example, the face image data of worker W is registered as the support target 604 in the work support DB 144, and the work analysis device 102 identifies the support target 604 from the work process video data 506 using known face recognition technology. Note that face recognition technology is just one example, and the device is not limited to face recognition technology as long as it is possible to match the support target 604 in the work support DB 144 (for example, the worker ID that identifies worker W) with the information obtained from the work process video data 506 (the worker ID obtained from the image or 2D code).

[0132] (Step S1102) The work analysis device 102 determines the effectiveness of the work support provided by the support target 604. Specifically, for example, in step S1101, it determines whether the support target 604 was identified in the work process video data 506. If it was not identified (Step S1102: No), the process proceeds to step S1103. If it was identified (Step S1102: Yes), the process proceeds to step S1106.

[0133] If the support recipient 604 consists of multiple workers W, it is sufficient to specify the minimum number of workers required for the work support type 601 and the target work 602. This minimum number of workers is assumed to be registered in the work support library DB 111 and the work support DB 144 (not shown).

[0134] (Step S1103) The work analysis device 102 determines whether the current work process has been completed based on the work process video data 506. Specifically, for example, as shown in Figures 8 and 9, the work analysis device 102 determines how far the current work process has progressed, that is, whether the agreement rate is above the threshold set for the current work process. This can also be rephrased as determining whether it matches the work process completion determination shown in the process determination algorithm 307 (in the example in Figure 9, if it is the first process, the work process completion determination 931 or work process completion determination 941). The work analysis device 102 repeats step S1103 until it is determined that the current work process has been completed, that is, until the agreement rate is above the threshold (Step S1103: No). If it is determined that the current work process has been completed (Step S1103: Yes), the device proceeds to step S1104.

[0135] (Step S1104) After step S1103: Yes or step S1110, the work analysis device 102 identifies the work process overtime 505 and the work process analysis result for the work process that has been determined to be completed, records them in the work performance DB 143, and proceeds to step S1105. Specifically, for example, the work analysis device 102 identifies the work process start time 503 and work end time 504 from the work process video data 506 as the actual work time for the work process that has been determined to be completed, and calculates the actual work time by subtracting the work process start time 503 from the work end time 504. Then, the work analysis device 102 subtracts the target work time 402 from the actual work time, and records 0 if the subtraction result is 0 or less, and the subtraction result if the subtraction result is greater than 0 in the work performance DB 143.

[0136] The recorded work process start time 503 is the date and time corresponding to the playback time of the work process video data 506 when the object detection algorithm 305 detects a state indicating the start of the work process. The recorded work end time 504 is the date and time corresponding to the playback time of the work process video data 506 when it is determined that the current work process has ended (step S1103: Yes).

[0137] The work analysis device 102 identifies, as a result of work process analysis for the work process that has been determined to be completed, a discrepancy in the process result 507 by the motion detection algorithm 304, a possibility of work defect 508 by the object detection algorithm 305, and a possibility of item defect 509 by the anomaly detection algorithm 306, and records them in the work performance DB 143.

[0138] (Step S1105) The work analysis device 102 determines whether the assembly work is completed or not. Specifically, for example, the work analysis device 102 determines whether the final work step of the work procedure 401 acquired in step S1001 is completed or not. If it is determined that the assembly work is not completed (Step S1105: No), the process returns to step S1101. If it is determined that the assembly work is completed (Step S1105: Yes), the work analysis device 102 terminates the work analysis process (Step S1006).

[0139] (Step S1106) The work analysis device 102 determines whether the current work process has exceeded the target work time of 402. If the current work process has not exceeded the target work time of 402 (Step S1106: No), proceed to step S1108. If it is determined that the current work process has exceeded the target work time of 402 (Step S1106: Yes), proceed to step S1107.

[0140] (Step S1107) The work analysis device 102 transmits work support information to the field device 107 of the support target 604 identified in step S1101. As a result, the field device 107 of the support target 604 displays the received work support information. If the target work time 402 is exceeded, it is determined that the work process is difficult for the worker W. For example, if "projection of points to note in the assembly procedure" is set in the work support algorithm 603, the field device 107 of the support target 604 will display the "projection of points to note in the assembly procedure" indicated by the work support algorithm 603 only to the worker W2 who is registered as the support target 604.

[0141] (Step S1108) The work analysis device 102, similar to step S1103, determines whether the current work process has been completed based on the work process video data 506. This can also be rephrased as determining whether it matches the work process completion determination shown in the process determination algorithm 307 (in the example in Figure 9, if it is the first process, it would be work process completion determination 931 or work process completion determination 941). The work analysis device 102 repeats step S1108 until it determines that the current work process has been completed, that is, until the agreement rate exceeds a threshold (Step S1108: No). If it is determined that the current work process has been completed (Step S1108: Yes), the device proceeds to step S1109.

[0142] (Step S1109) The work analysis device 102 determines whether the state of the work process at the time of completion determination matches the planned work result 403. Specifically, for example, as shown in Figures 8 and 9, the work analysis device 102 determines to what extent the work process at the time of completion determination matches the planned work result 403, that is, whether the matching rate is greater than or equal to the threshold set for the work process that has been determined to be completed. If they match (Step S1109: Yes), the process proceeds to step S1104. If they do not match (Step S1109: No), the process proceeds to step S1110.

[0143] (Step S1110) The work analysis device 102 transmits assembly defect information to the field device 107 of the support target 604 identified in step S1101. As a result, the field device 107 of the support target 604 displays the received assembly defect information. Then, the process proceeds to step S1104.

[0144] <Figure 12 Management Screen> Figure 12 is an explanatory diagram showing an example of the management screen 1200 display. The management screen 1200 is displayed on a display which is an example of the output device 204 of the management terminal 105. The management terminal 105 displays the management screen 1200 by acquiring information from the work performance DB 143. The management screen 1200 is a screen for the manager M of the management booth 150 to grasp the work status of the worker W in the target work area 302. The management screen 1200 includes, for example, progress analysis 1201, progress graph 1202, process-specific analysis 1203, and worker video 1204.

[0145] The progress analysis 1201 includes a first display 1211, a second display 1212, and a third display 1213. The first display 1211 is a link that, when clicked, provides access to the work site where assembly work is being performed and the target product. The second display 1212 is a link that, when clicked, provides access to the total planned and actual number of product assemblies at the work site for a certain period, such as a date, and the planned and actual number of assemblies by worker. The third display 1213 is a link that, when clicked, provides access to information such as the number of work process delays, the number of work defects, and the number of potential product defects.

[0146] Progress graph 1202 is a graph that shows the time and progress of each process for each worker. Progress graph 1202 assumes, for example, that a certain product is completed through an assembly process consisting of four work processes, with three workers W1 to W3 engaged in that assembly process, and shows the results of the automatic determination of the completion of each work process.

[0147] In progress graph 1202, circles represent the progress of worker W1, triangles represent the progress of worker W2, and squares represent the progress of worker W3. The steeper the slope of the line connecting the plots, the faster the work process was completed. Conversely, a line with a gentle slope indicates that the work process took a long time.

[0148] Sorting these line graphs by worker W makes it visually easier to identify the work processes that each worker W struggles with (line graphs with a gentle slope). Furthermore, comparing these graphs to the ideal work graph makes it visually easier to assess each worker W's skill level. Skill levels for each worker W can also be assessed by comparing the line graphs between different workers W.

[0149] Process analysis 1203 is information displayed for each worker W by quantifying the progress graph 1202. For example, for each work process of each worker W, the latest actual time for the work process, the average actual work time for the work process, the latest actual work time for all work processes, the average actual work time, and the number of times process delays or work defects were judged, and the number of times there was a possibility of product defects are displayed as analysis results. In addition, the average value and ideal value for each worker W may also be displayed.

[0150] The worker video 1204 displays the video of the work currently being performed by each worker W. Videos of the hands working on the product are particularly desirable. This allows for real-time monitoring of the situation, and by recording it as a log, it is also possible to review it retrospectively.

[0151] In the example in Figure 12, graphs, numerical information, and videos are displayed for three workers W1 to W3. However, in actual workplaces, there are often dozens or even hundreds of workers involved in a task. In such cases, projecting all the information onto a single screen can be difficult to read. Therefore, it may be possible to display information for a specific few individuals (for example, three) based on some selection criteria, or to enable the display of some information, such as videos, on or off. One example of a selection criterion is to prioritize workers W who have caused the most work process delays based on previous analysis results, workers W who are likely to have work defects, and workers W who are likely to have defective products.

[0152] <Figure 13 Individual Management Screen> Figure 13 is an explanatory diagram showing an example of the display of the individual management screen. The individual management screen 1300 is displayed on a display which is an example of the output device 204 of the management terminal 105. The management terminal 105 displays the management screen 1200 by acquiring information from the work performance DB 143. The management screen 1200 is a screen for the manager M of the management booth 150 to grasp the specific work results of a specific worker W. The individual management screen 1300 includes, for example, a selected work video 1301, a selected work progress graph 1302, selected work attributes 1303, and selected work process analysis results 1304.

[0153] The selected work video 1301 is a video of a specific worker W at the time when the third display 1213 "Extraction process by analysis" of the progress analysis 1201 on the management screen 1200 is selected. The selected work video 1301 is read from the work process video data 506 in the work performance DB 143 to the management terminal 105 and displayed on the individual management screen 1300. This allows the administrator M to check specific work videos for work processes that have been delayed, work processes that have been judged as defective, or work processes that may have product defects.

[0154] The selected work progress graph 1302 displays the progress graph of the work process selected in the selected work video 1301 (in the above example, the "extraction process by analysis"). The management terminal 105 may simultaneously display the work progress graph of all work processes during ideal work, and the work progress graph of all work processes including work processes that have experienced delays, work processes that have been judged as defective, or work processes that may have product defects. This allows the manager M to check how far behind the problematic work processes are. The management terminal 105 may also display the line segments or points of the problematic work processes in a way that is easy for the manager M to understand, such as by changing their thickness or color from other process parts.

[0155] The selected work attribute 1303 is attribute information when the work selected in the selected work video 1301 is performed. The attribute information includes, for example, the assembly site (site A), work name (product A assembly), worker name (W1), work time (XXX), manufacturing number (A001), and the work process in question (process 4).

[0156] The selected work process analysis result 1304 is the analysis result (inconsistency in process results 507, possibility of work defects 508, possibility of product defects 509) when the work process selected in the selected work video 1301 was performed. For example, it displays text indicating delays in work speed, the number of times work defects were determined and corrected, and the possibility of assembly defects or material defects in the product.

[0157] <Figure 14 Work Support Screen> Figure 14 is an explanatory diagram showing an example of the display of the work support screen. The work support screen 1400 is displayed on a display which is an example of the output device 204 of the field device 107. The field device 107 displays the work support screen 1400 by acquiring work support information from the work support DB 144 (steps S1107, S1110). The work support screen 1400 includes the process name 1401, the selected work support video 1402, and the work precautions 1403.

[0158] Process name 1401 is the name of the work process that worker W should perform. The selected work support video 1402 is a video that specifically shows how to assemble the work object 174 indicated by process name 1401. Under normal conditions, the field device 107 displays video data of the assembly procedure manual as work navigation.

[0159] If the work support for worker W is effective (step S1102: Yes), the field device 107 may display additional information on the normal work navigation video, or display a more easily understandable video.

[0160] In the example shown in Figure 14, where conventional instructions simply instructed to insert part B into slot 2, the new instructions indicate that part B should be slowly inserted into the left-center slot (Slot 2) so that the mark on part B faces left. This provides more specific instructions that address common mistakes.

[0161] Specifically, the on-site device 107 may, during the work process, mark the target work area on the work object 174 with color or a thick line, or display images, videos, animations, etc., showing how to assemble it. The on-site device 107 may display primarily video and minimize text to facilitate intuitive understanding.

[0162] Work precaution 1403 displays the content shown in the video selection support video 1402 in concrete written form. It is desirable that the text be specific, concise, and avoids misunderstandings.

[0163] Although we have described an example in which the field device 107 displays work support information on a display, which is an example of an output device 204, there are variations that allow work support information to be provided in various other ways. Among the five senses of the worker W, support methods that appeal to sight, hearing, and touch are common.

[0164] As a visual aid, an aerial display may be used instead of a regular display. An aerial display is a means of displaying images in the air by utilizing light reflection, and is also called aerial imaging. The image can be seen with the naked eye without the need for special glasses or other devices. Because it is possible to perceive images in three dimensions, it is an effective means of work support for more concretely understanding work objects 174 that are worked on from various directions, such as up, down, left, and right, as well as from two-dimensional work. Furthermore, by adding sensors to the aerial display, it can also be used as a work support means similar to an aerial touch panel.

[0165] Furthermore, as a visual aid, a projector may be used instead of a regular display. Specifically, projection mapping using a projector can be used to project the next work instruction directly onto the assembly object (workpiece 174), or to project an error message if the work result is incorrect, while using light to indicate the correct work procedure and work location, thereby providing work support that is easy for the worker to understand.

[0166] Furthermore, as a visual aid, Augmented Reality (AR) glasses may be used instead of a regular display. Specifically, AR glasses can be used to display the next work instructions, or to show error messages if the work result is incorrect, while also providing guidance on the correct work procedure and location, thereby providing work support that is easy for workers to understand.

[0167] As a method of support that appeals to the auditory sense, speakers may be used. By conveying error messages when the work result is incorrect and providing voice instructions on the correct work procedure and work location, work support can be made easier for workers to understand. If the sound from the speaker leaks out to the surrounding area and it becomes unclear which worker the support is intended for, it is best to use directional speakers that deliver sound only to a specific area. Alternatively, each worker may be provided with earphones.

[0168] As a tactile support method, vibration sensors can be incorporated into work clothes, gloves, power tools, etc. By applying a specific vibration when the work result is incorrect, errors can be notified, or when a work process is completed correctly, or the correct work direction or orientation can be indicated by vibrations in the intended direction, thus providing work support that is easy for workers to understand.

[0169] It is also acceptable to use support methods that combine multiple senses, such as sight, hearing, and touch. For example, using a display and speakers together is a very natural way to support work. In addition to providing work support through videos and animations, there are various other forms of support, such as support via video call with experts at remote sites, or support through a digital twin or metaverse space constructed in cyberspace via a display.

[0170] <Examples of application of work support system 100> The work support system 100 can be applied to various factories that manufacture parts, equipment, and main products, including automobile parts manufacturing plants, automobile manufacturing plants, machine tool manufacturing plants, railway vehicle parts manufacturing plants, and railway vehicle manufacturing plants, as well as other factories where human labor remains.

[0171] In recent years, many factory production sites have adopted the cell production system, which is suitable for flexibly changing the items produced. The cell production system is a production method that can respond to the shift from mass production of a few product types to small-batch production of many product types. In this system, one person or a small team of workers is responsible for the entire assembly process of a product on a line called a cell, where parts and tools are arranged in a U-shape or similar configuration.

[0172] Field devices 107 are deployed to these cells. Since multiple cells are installed in the factory and manufacturing work is carried out simultaneously, multiple field devices 107 are deployed as needed. One or more field devices 107 are connected via network 108 to a work analysis device 102, a work support device 103, a DB device 104, a management terminal 105, a development terminal 106, and a library DB management device 101. These may all be located within the same factory, but in many cases they are deployed in a distributed manner. For this reason, network 108 consists of a local network within the factory and the internet or dedicated lines connecting to the outside of the factory.

[0173] A typical deployment example involves deploying a management terminal 105 and a development terminal 106 in an office within a factory where one or more field devices 107 are installed. Additionally, a work analysis device 102, a work support device 103, and a database device 104 are deployed in the factory's server room. Furthermore, a library database management device 101 is deployed on the cloud.

[0174] Furthermore, a management terminal 105 and a development terminal 106 are deployed in the office within the factory where one or more field devices 107 are installed. In addition, a work analysis device 102, a work support device 103, and a database device 104 are deployed on the cloud, and a library database management device 101 is deployed on the cloud.

[0175] Furthermore, when classifying these pieces of equipment from the perspective of asset owners, a typical example is that a company providing work identification, analysis, support, and development system services may own the library DB management device 101 and lease its functions only when needed to companies that utilize the work identification, analysis, support, and development system services. Companies utilizing the work identification, analysis, support, and development system services may own and constantly utilize the work analysis device 102, work support device 103, DB device 104, management terminal 105, development terminal 106, and field device 107.

[0176] Furthermore, a company providing work tracking, analysis, support, and development system services may own a library DB management device 101, a work analysis device 102, a work support device 103, a DB device 104, a management terminal 105, and a development terminal 106, and may lease these functions to companies that utilize the work tracking, analysis, support, and development system services only when needed. Companies utilizing the work tracking, analysis, support, and development system services may own and constantly use the field equipment 107. The above are typical examples of asset ownership, but the system is not limited to these forms.

[0177] Thus, according to this embodiment, it is possible to easily identify the boundaries between consecutive work processes. Furthermore, by recording the work process excess time 505, discrepancies in process results 507, the possibility of work defects 508, and the possibility of product defects 509 as analysis results, it is possible to automatically identify the work processes that are burdensome for each worker 502, automatically point out the possibility of manufacturing defects, and automatically suggest improvement measures only for the burdensome work processes.

[0178] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all of the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, some of the configurations of one embodiment may be added to those of another embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with other configurations.

[0179] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0180] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or on recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0181] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected.

[0182] 100 Work support system 101 Library DB management device 102 Work analysis device 103 Work support device 104 DB device 105 Management terminal 106 Development terminal 107 Field equipment 108 Network 110 Work analysis system 171 Sensor 173 Tool 174 Work object 201 Processor 202 Storage device 301 Product name 302 Work location 303 Work environment 304 Motion detection algorithm 305 Object detection algorithm 306 Anomaly detection algorithm 307 Process judgment algorithm 401 Work procedure 402 Target work time 403 Planned work result 501 Serial number 502 Worker 503 Work process start time 504 Work process end time 504 Work completion time 505 Work process overtime 506 Work process video data 507 Discrepancy in process results 508 Possibility of work defects 509 Possibility of product defects 601 Type of work support 602 Target work 603 Work support algorithm 604 Target of support 605 Support record 931 Work process completion judgment 941 Work process completion judgment

Claims

1. A work analysis device having a processor for executing a program and a storage device for storing the program, wherein the processor is able to access a first database for storing an analysis algorithm for detecting the state of work and a second database for storing planned work results for each of a series of work processes constituting the work, and the processor is characterized by executing: a first acquisition process for acquiring first video data showing the series of work processes; a second acquisition process for acquiring the analysis algorithm from the first database; a third acquisition process for acquiring planned work results for the work processes from the second database; a detection process for detecting the state of work in the first video data acquired by the first acquisition process based on the analysis algorithm acquired by the second acquisition process; a generation process for generating a work process completion determination indicating the boundary between consecutive work processes in the series of work processes based on the detection result from the detection process and the planned work results acquired by the third acquisition process; and a registration process for registering the work process completion determination generated by the generation process in the first database.

2. A work analysis device according to claim 1, wherein the first database stores the analysis algorithm for each work object, the second database stores the planned work results for each work process of the series of work processes for each work object, in the first acquisition process the processor acquires the first video data relating to a specific work object, in the second acquisition process the processor acquires a specific analysis algorithm relating to the specific work object from the first database, in the third acquisition process the processor acquires a specific planned work result relating to the specific work object from the second database, and in the registration process the processor registers the work process completion determination in the first database in association with the specific analysis algorithm.

3. A work analysis device according to claim 2, wherein the first database stores the analysis algorithm for each combination consisting of the work object and the work location where the work is performed on the work object, and in the second acquisition process, the processor acquires a specific analysis algorithm from the first database for a specific combination consisting of a specific work object and a specific work location where the work is performed on the work object.

4. A work analysis device according to claim 3, wherein the first database stores the analysis algorithm for each combination consisting of the work object, the work location, and the work environment in the work location, and in the second acquisition process, the processor acquires a specific analysis algorithm from the first database relating to a specific combination consisting of a specific work object and a specific work environment in the specific work object.

5. A work analysis device according to claim 1, wherein the analysis algorithm includes a motion detection algorithm for detecting the movements of a worker performing the work.

6. A work analysis apparatus according to claim 1, wherein the analysis algorithm includes an object detection algorithm for detecting an object.

7. A work analysis device according to claim 2, wherein the processor performs: a fourth acquisition process for acquiring second video data relating to a specific work object; a fifth acquisition process for acquiring a specific analysis algorithm relating to a specific work object from a first database; a fifth acquisition process for acquiring a specific work schedule result and a specific target work time relating to a specific work object from a second database; and an analysis process for identifying the state of work at the time of completion of the current work process from the second video data based on a specific work process completion determination in the specific analysis algorithm, and analyzing the current work process based on the identified state of work and the specific work schedule result in the current work process.

8. A work analysis apparatus according to claim 7, wherein the processor performs the analysis process for each of the work steps of a series of work steps relating to the specific work object.

9. A work analysis method performed by a work analysis apparatus having a processor for executing a program and a storage device for storing the program, wherein the work analysis apparatus has access to a first database for storing an analysis algorithm for detecting the state of work and a second database for storing planned work results for each of a series of work processes constituting the work, and the processor performs: a first acquisition process for acquiring first video data showing the series of work processes; a second acquisition process for acquiring the analysis algorithm from the first database; a third acquisition process for acquiring planned work results for the work processes from the second database; a detection process for detecting the state of work in the first video data acquired by the first acquisition process based on the analysis algorithm acquired by the second acquisition process; a generation process for generating a work process completion determination indicating the boundary between consecutive work processes in the series of work processes based on the detection result from the detection process and the planned work results acquired by the third acquisition process; and a registration process for registering the work process completion determination generated by the generation process in the first database.

Images