Work analysis system

Abstract

To easily, quickly, and accurately grasp whether or not improvements to the entire work process, including work environment, are necessary.SOLUTION: A work analysis system comprises: an environmental data acquisition unit (step S1) that acquires work environment data; a worker data acquisition unit (step S2) that acquires three-dimensional data related to a worker, including the worker's movements; a data integration unit (step S3) that integrates the environmental data and the worker data by aligning the positions of the coordinate systems of the environmental data and the worker data; an analysis unit (step S4) that analyzes at least one of waste, inconsistency, and overload of the worker's movements from the worker data; and a display unit (step S10) that displays video of the worker at a work site based on the integrated environmental data and worker data. The display unit is configured to display the analysis results of any of waste, inconsistency, and overload obtained by the analysis unit in the video in association with the actions of the worker that produced the analysis results.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a system for analyzing operations such as the manufacturing or assembly of various products by workers, such as the processing, mounting or removal of parts, and further tightening. [Background technology]

[0002] For the purpose of improving or streamlining the manufacturing and assembly of various products, it has been common practice to capture a video of a worker's movements while actually performing the work, obtain data on the worker's movements from the video, and analyze the work. One example is described in Patent Document 1. The method described in Patent Document 1 is intended to enhance the effectiveness of work learning, and involves detecting the movements of the worker's arms, hands, etc. based on a video of the worker, identifying a completed task among tasks included in a predetermined work scenario based on the movements, and superimposing and displaying an image stored in association with the task specified in the work scenario as the next task after the completed task on an image captured from the video after the movement detection.

[0003] Patent Document 2 also describes an invention aimed at providing appropriate movement support by grasping the details of a worker's movements, and is a system configured to acquire an image of the worker, acquire information about the worker's movements, estimate the worker's skeleton from the image of the worker, and correct the estimated skeleton information using information about the worker's details of the worker's movements. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 017737 [Patent Document 2] Patent Publication No. 2021-056922 Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Patent Document 1 and the system described in Patent Document 2 film a worker performing an actual task and evaluate the actual task based on the video, allowing for accurate and easy understanding of the actual situation. Furthermore, the method described in Patent Document 1 displays the actual motion captured as a video overlaid with actions prepared in advance as a task scenario, allowing for visual understanding of whether the actual task deviates from the specified task (or model task), and the extent to which this deviation exists. Furthermore, the system described in Patent Document 2 uses information about the worker's motion as so-called training data, and uses this to correct the detailed motions of the videoed worker, allowing for effective correction or learning of the task.

[0006] However, the conventional inventions described in Patent Documents 1 and 2 are designed to compare so-called actual movements with reference movements. Therefore, even if a worker can learn to perform tasks in accordance with the reference movements, they are unable to achieve further improvement. For example, if a worker's physique, such as height or arm length, differs from that assumed in the reference movements, the reference movements are likely to result in poor workability for that worker. However, the inventions described in Patent Documents 1 and 2 are unable to identify such work improvement items. Furthermore, because the standard for analyzing or evaluating work is limited to the so-called reference movements, it is impossible to obtain information on the need for improvements to the so-called work environment, such as the equipment at the work site, the workpieces (workpieces), and the layout of the tools used. In other words, conventional inventions have not adequately acquired and analyzed information on the presence or absence of waste, unevenness (work stability), or overstress in individual tasks or series of tasks performed by workers, and the direction of improvement.

[0007] The present invention has been made in light of the above-mentioned circumstances, and aims to provide a system that makes it possible to easily, quickly, and accurately grasp the need for improvements and modifications to the entire work process, including not only the work procedures and their contents, but also the work environment, in order to reduce the burden on workers and improve work efficiency. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the present invention provides a work analysis system that analyzes work being performed by a worker based on video obtained by filming the worker working at a work site, and includes an environmental data acquisition unit that acquires environmental data, which is three-dimensional data of the work site; a worker data acquisition unit that acquires three-dimensional data about the worker, including the worker's movements, using video filmed of the worker at the work site; a data integration unit that integrates the environmental data and the worker data by aligning the positions of coordinate systems of the environmental data and the worker data; an analysis unit that analyzes at least one of waste, unevenness and strain in the worker's movements from the worker data; and a display unit that displays video of the worker at the work site based on the integrated environmental data and worker data, and the display unit is configured to display the analysis results of any of the waste, unevenness and strain obtained by the analysis unit in the video, corresponding to the worker's actions that produced the analysis results.

[0009] In the present invention, the video display device may further include a display image generation unit that adjusts at least one of the angle of view, the display field of view, and the playback speed of the video.

[0010] In the present invention, the environmental data may include three-dimensional data of equipment installed at the work site, three-dimensional data of the work object on which the worker performs work, and three-dimensional data of the jigs or tools used by the worker. [Effects of the Invention]

[0011] According to the present invention, environmental data and worker data, each of which is three-dimensional data, are acquired, and these data are used to display a video of the worker's movements at the work site, thereby visually reproducing the worker's movements at an actual work site. In other words, not only the so-called actual work on the work object but also the additional work accompanying the actual work can be accurately grasped. Furthermore, since the work data reflecting the worker's movements at the work site is used to analyze whether the work is wasteful, uneven, or excessive, the analysis results also reflect the relative relationship between the so-called environment at the work site and the worker. Therefore, the analysis results reflect not only the worker's hand and body movements but also the equipment, jigs, and tools that make up the work environment, as well as their layout, and thus provide information on not only work procedures but also the need for improvements to the work environment.

[0012] Furthermore, according to the present invention, by providing a display image generating unit that can adjust the angle of view and the display field of view, the work site and the worker can be visually grasped from multiple directions, and the details can be enlarged to view, etc., so that the actual work performed can be understood from multiple angles and in more detail. As a result, it is possible to avoid or reduce overlooking waste, unevenness, or unreasonableness in the work, and more effective work improvements can be achieved. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram showing how a work video is obtained by filming the work state of a worker at a work site. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of a controller. [Figure 3] 10 is a flowchart illustrating an example of control of analysis of work and display of the analysis results in an embodiment of the present invention. [Figure 4] FIG. 10 is a schematic diagram showing an example of a display when an analysis result indicates overstrain. DETAILED DESCRIPTION OF THE INVENTION

[0014] Next, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below is merely an example of how the present invention may be implemented, and is not intended to limit the present invention.

[0015] The system according to the present invention is configured to analyze the overall movement of a worker, including walking, to identify waste (hereinafter referred to as "muda"), unevenness (hereinafter referred to as "mura"), and excessive effort (hereinafter referred to as "muri"), and to extract improvement items with the primary goal of reducing the burden on the worker. FIG. 1 schematically illustrates an example of a workplace targeted by an embodiment of the present invention. Here, a worker 2 performs a predetermined task, such as assembling parts, tightening screws, or polishing, on a transported work object (workpiece) 1. Shelves 4 on which parts, jigs, tools, etc. are placed, leaving a predetermined work space, are installed next to a transport equipment 3, such as a conveyor, that moves the workpiece 1. In this embodiment of the present invention, the workpiece 1, transport equipment 3, and shelves 4 are referred to as the work environment, and a camera 5 is provided to capture both the work environment and the worker 2. The camera 5 may be a 3D scanner that captures three-dimensional data of the work environment or a camera that captures the worker 2.

[0016] A controller 6 is provided to process image data obtained by the camera 5. The controller 6 is mainly composed of a computer that performs calculations according to a program prepared and stored in advance and outputs the results of the calculations. A user device 7, which corresponds to the display unit in the embodiment of the present invention, such as a keyboard and mouse for inputting data and selecting the process to be executed, or a monitor for displaying moving images and the results of the calculations, is connected to the controller 6 via wire or wirelessly.

[0017] The main function of the controller 6 is to process video of the work environment to extract points for improvement such as waste, inconsistency, and overload at the work site, and to display the extracted results on the user device 7. Therefore, in addition to image data from the camera 5, three-dimensional data of the work environment and data related to the worker 2 are input to the controller 6. An example of the functional configuration of the controller 6, which performs predetermined processing based on this data, is shown in the block diagram of Figure 2.

[0018] The controller 6 includes a storage unit 6A that stores various data. One example of this data is work environment data, which includes three-dimensional data of the work site, three-dimensional data of the workpiece 1, and three-dimensional data of jigs and tools. These data may be data that have been prepared and input in advance. Therefore, the storage unit 6A corresponds to the environment data acquisition unit in the embodiment of the present invention.

[0019] Other examples of data stored in the memory unit 6A include data related to the worker (subject), including skeletal data, task name data, task time data, motion analysis data, workload data, and video data. Based on images of the worker 2 captured by the camera 5, a skeletal model of the worker 2 is created using conventional data processing, and data including the movements of the skeletal model when the worker performs a specific task constitutes the skeletal data. Task name data is stored to identify the task. The start and end of each task can be determined by viewing the image, and the start and end timings are input from the user device 7 to determine the task time. This task time is stored as task time data. Furthermore, because a series of the worker's movements are obtained as image data, it is possible to distinguish between movements directly related to the task and other movements by viewing the image. These movements are marked within the series and stored as motion analysis data. Furthermore, postures and movements for determining the workload on the worker are extracted based on the skeletal data and stored as workload data. Video of the work environment including the worker 2 being measured is stored as video data. Therefore, the storage unit 6A corresponds to the worker data acquisition unit in the embodiment of the present invention.

[0020] The controller 6 is provided with an improvement point processing unit 6B that extracts items to be improved based on the work time data, motion analysis data, and workload data. The improvement point processing unit 6B includes a workload determination unit, a work variation determination unit, and a wasteful motion determination unit. The movements, including posture, of the worker 2 are obtained as workload data as described above, and the workload determination unit evaluates the posture or movement from an ergonomic perspective based on that data, and analyzes and determines the degree of strain (so-called overstrain) on the worker 2. In addition, since the length of time required for a task can be determined based on the previously obtained work time data, the work variation determination unit analyzes and determines the variation in the task (so-called unevenness) based on the longest, shortest, and average work times for multiple repetitions of the same task by the same worker. Furthermore, direct work such as assembling and tightening, and so-called indirect movements such as preparing jigs and tools can be seen from the video, and these are identified by the user device 7, and the wasteful movement determination unit analyzes and determines so-called wasteful movements of the worker 2 based on the data (images) thus identified. Therefore, the improvement point processing unit 6B corresponds to the analysis unit in the embodiment of the present invention.

[0021] The controller 6 is provided with an information setting unit 6C. The information setting unit 6C is a functional unit that processes data to be output and displayed on the monitor, and includes a target timing setting unit, an observation viewpoint setting unit, a color information setting unit, and a display data setting unit. The target timing setting unit, for example, sets the display speed (playback speed) of a video, such as fast-forward, slow-motion, or fast-rewind. Since image data is three-dimensional data of the work environment and the worker, it can be processed into an image with an appropriate orientation and position in three-dimensional space. The observation viewpoint setting unit appropriately changes the viewpoint of the image viewed through the monitor based on operations from the user device, and also zooms in and out to a desired position in the image, enlarging or reducing the image on the monitor (or adjusting the display field of view). The color information setting unit performs a process of applying a different color to parts of the image that should be highlighted or selected, and displaying them on the monitor. This can be performed by inputting information from the user device 7. The display data setting unit has a function of processing to set the image to be displayed on the monitor, with the image selection being performed, for example, by the user device 7. The selectable images may be images captured by the camera 5, or may be images that have been input in advance to the controller 6 as image data, or images captured by another camera.

[0022] The controller 6 is provided with a visualization processing unit 6D that performs processing to display and visualize images on the user device 7. Data for visualization is created by the information setting unit 6C, and the visualization processing unit 6D is mainly composed of an information display screen generation unit that generates a display image based on that data. Therefore, the display image is a video captured by the camera 5 overlaid with a skeletal model of the worker 2, analyzed and evaluated improvement points, their contents, and appropriate images selected by the user device 7. The visualization processing unit 6D corresponds to the data integration unit and display image generation unit of the present invention.

[0023] An example of the control executed by the controller 6, i.e., an analysis of work being performed at a work site, will be described with reference to the flowchart shown in Fig. 3. Note that the flowchart shown in Fig. 3 shows the main controls for analyzing work in order, but the control of each step does not necessarily have to be executed in the order shown in Fig. 3, and the order of the control of each step can be reversed within a technically permissible range, and some controls can also be executed simultaneously in parallel.

[0024] When control is started, first, a three-dimensional (3D) scan of the work site is performed (step S1), and the human skeleton is measured during work (step S2). The 3D scan in step S1 is the acquisition of work environment data, and includes the acquisition of image data acquired by camera 5 and pre-prepared 3D data about equipment installed at the work site. The measurement of the human skeleton in step S2 may be a control that creates a skeletal model of worker 2 using the joints of worker 2 and the lines connecting the joints, and measures the height, arm length, etc. of that skeletal model. Note that joints may be identified using images of the human body in the video, or by attaching some kind of identification tag to parts of the worker corresponding to the joints during filming and then measuring the identification tag in the video.

[0025] The coordinate system of the 3D scan data of the work site and the skeletal data of the worker 2 are aligned (step S3). This control is a control for inserting the worker converted into skeletal data into the work site, and is data processing for displaying the worker converted into skeletal data as if he were present at the work site.

[0026] On the other hand, based on the skeleton data obtained in step S2, the movements of the worker 2 that are the source of the skeleton data are analyzed for strain, inconsistency, and waste (step S4). This analysis in step S4 is a function of the improvement point processing unit 6B shown in Fig. 2, and an example of this analysis has been described for the improvement point processing unit 6B.

[0027] The analysis example shown in FIG. 3 is an example in which analysis is performed on each task in a series of multiple consecutive processes. Therefore, after the analysis starts, the first data (including skeletal data) is read (step S5). For example, since the skeletal data is calculated for each frame (or frame) based on the original video or data obtained by a sensor, the data including the skeletal data is read for each frame (or frame). The read data is compared with the analysis results obtained in step S4 (step S6). That is, the analysis results of strain, inconsistency, and waste are imported in accordance with the movements of the worker 2 contained in the read data. Next, it is determined whether strain, inconsistency, or waste has been extracted from the read work data (skeletal data with movement) (step S7). If strain, inconsistency, or waste has been extracted and the result of the determination in step S5 is "yes," the extracted data of strain, inconsistency, or waste is superimposed on the skeletal data (step S8). In other words, the analysis results are linked to the skeletal data that produced them and processed to display them simultaneously.

[0028] When the so-called superposition process of step S8 is performed, the results of the analysis of strain, unevenness, and waste are superimposed on the skeleton data after the environmental data and coordinate system have been aligned in step S3. The data for the display image thus obtained is then adjusted for the field of view and angle of view to be displayed (step S9). Note that if the determination result in step S7 is "NO," the process of step S8 is not performed and the process proceeds to step S9.

[0029] The adjustment in step S9 is a function of the information setting unit 6C described above, and its details are as described above for the information setting unit 6C. Then, the adjusted image (video), i.e., the skeletal model of the worker and a 3D image of the work site, are output to be displayed on the user device 7 (step S10). This data processing, in which the analysis results of waste and the like are superimposed on the skeletal model of the work site, is performed on data categorized by task, etc. Therefore, following step S10, it is determined whether the data for which processing up to step S10 has been completed is the last data of a series of multiple data (step S11). If the result of the determination in step S11 is "NO," the next data is read (step S12), and the process returns to step S6, where the control of the subsequent steps is performed in order. When the processing up to step S10 for the final (last) data of the series of multiple data is completed, the result of the determination in step S11 becomes "YES," and the routine shown in FIG. 3 is terminated.

[0030] Fig. 1 shows an example of a display image performed as described above. It shows a schematic example of an analysis result in which unevenness is found, and displays the work content (e.g., bolt removal) and "unevenness" along with the longest and shortest times near the worker 2, who is represented by a skeleton model. Fig. 1 also shows an example in which work content data 8 and work video 9 captured by camera 5 are imported and displayed as optionally selectable images.

[0031] Figure 4 shows a schematic example of an analysis result that indicates "impossible." In this example, the elbow bend angle θ is greater than the expected angle, resulting in an "impossible" judgment, and workers 2A and 2B are displayed side by side. In this case, the worker's name, the task content (e.g., removing a bolt), and the "impossible" notation and the nature of the "impossible" task are displayed overlaid on the skeletal models of the two workers 2A and 2B. Note that by using different colors for workers 2A and 2B, it is easier to distinguish between the analysis results for each worker 2A and 2B.

[0032] The display when the analysis result indicates "waste" is almost the same as the display when the analysis result indicates "impossible," and the notation of "waste" and the details of the waste (for example, turning around more than 90 degrees while walking) are displayed in a position close to the skeletal model of worker 2.

[0033] It is preferable that the analysis results of the worker's 2 movements, such as waste, inconsistency, and overload, be displayed concretely as images in addition to being displayed in text. However, since the position of the camera 5 is fixed, there are parts that cannot be seen by keeping the viewpoint at the position of the camera 5. In such cases, the angle of view is adjusted by the user device 7 to display the parts that are generating the analysis results. Because the environmental data and skeletal data are 3D data, it is easy to adjust and change the angle of view, and it is also easy to zoom in or out as needed to see details or the overall picture.

[0034] As described above, in the embodiment of the present invention, the movements of the worker 2 at the work site are displayed, including their relative positional relationship with the work environment, and the analysis results of the movements, such as unevenness and overstress, are also displayed superimposed on the worker, making it easy to grasp the improvement items (points for improvement). In particular, because the movements of the worker 2 can be grasped in relation to the work environment, it becomes possible to easily and reliably grasp the points for improvement in the work environment that are causing waste, overstress, or unevenness for the worker 2.

[0035] The present invention is not limited to the above-described embodiments, and the movements to be analyzed may be any movements made during work, and are not limited to walking, stretching and contracting the arms, or turning around, but may also include bending and stretching the legs, bending or twisting the waist, rolling the upper body sideways, crossing the hands, etc. [Explanation of symbols]

[0036] 1 Work 2,2A,2B worker 3. Transport equipment 4 Shelves 5. Camera 6 Controller 6A storage section 6B Improvement point processing section 6C Information setting section 6D visualization processing unit 7 User Device 8. Work content data 9 Work video

Claims

1. A work analysis system that analyzes work performed by a worker based on a video obtained by filming the worker performing work at a work site, an environmental data acquisition unit that acquires environmental data, which is three-dimensional data of the work site; a worker data acquisition unit that acquires three-dimensional data relating to the worker, including the worker's movements, using video captured of the worker at the work site; a data integration unit that integrates the environmental data and the worker data by aligning the positions of the coordinate systems of the environmental data and the worker data; an analysis unit that analyzes at least one of waste, unevenness, and strain in the worker's movements from the worker data; a display unit that displays a video of the worker at the work site based on the integrated environmental data and the worker data, The display unit is configured to display, in the video, any of the analysis results of the waste, the unevenness, and the unreasonableness obtained by the analysis unit in association with the action of the worker that produced the analysis result. A work analysis system characterized by:

2. The work analysis system according to claim 1, The display image generating unit adjusts at least one of the angle of view, the display field of view, and the playback speed of the video. A work analysis system characterized by:

3. The work analysis system according to claim 1 or 2, The environmental data includes three-dimensional data of equipment installed at the work site, three-dimensional data of a work object on which the worker performs work, and three-dimensional data of a jig or tool used by the worker. A work analysis system characterized by:

Images