Work monitoring method
The method improves work monitoring by using cameras to analyze worker movements and workpiece status, providing accurate real-time feedback to enhance production quality and efficiency.
Patent Information
- Application Number
- JP2023071115
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-24
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2043-04-24
AI Technical Summary
Existing methods for monitoring worker actions during manufacturing are inadequate, as they rely solely on post-operation inspection of workpiece appearance, failing to account for dangerous movements and varying operation states, making it difficult to accurately assess work quality.
A work monitoring method that utilizes cameras to capture worker movements and workpiece status, determining appropriateness through image analysis, including pre- and post-operation video analysis, and providing real-time feedback to workers.
Enhances the accuracy of judging work appropriateness by considering both worker motions and workpiece states, enabling timely correction of inappropriate actions and improving production efficiency.
Smart Images

Figure 2024156548000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a work monitoring method. [Background technology]
[0002] When a product is manufactured manually, a worker performs a predetermined operation on a workpiece in each manufacturing process. In order to improve the quality of the product, it is desirable to be able to monitor the operations performed by the worker and detect inappropriate operations.
[0003] In the information processing system described in Patent Document 1, a correlation between the actions of a worker performing work on a workpiece and the inspection results of the workpiece's appearance after the work is performed is estimated, and the estimated correlation is fed back to the worker. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2022-155848 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, in the above method, the appropriateness of the worker's actions is judged based on the results of an inspection of the workpiece's appearance, and therefore inappropriate actions (e.g., inefficient or dangerous actions) by the worker are overlooked unless there is a problem with the appearance of the workpiece. In addition, when a worker performs multiple actions, the appropriate state of the workpiece after each action differs, making it difficult to determine whether the workpiece is good or bad without specifying the content of the action. Therefore, there is room for improvement in the method of monitoring the work performed by workers.
[0006] In view of the above problems, an object of the present invention is to more accurately determine the appropriateness of work performed by an operator. [Means for solving the problem]
[0007] The gist of the present disclosure is as follows.
[0008] (1) A work monitoring method for monitoring work performed by a worker, comprising the steps of: determining whether the worker is performing a predetermined action on a workpiece; and determining whether the workpiece is good or bad when it is determined that the worker is performing the predetermined action.
[0009] (2) The work monitoring method described in (1) above, wherein the quality of the work is determined based on an image of the work generated by a camera worn by the worker so as to photograph the work.
[0010] (3) A work monitoring method as described in (1) or (2) above, in which the quality of the work is determined based on a video of the work.
[0011] (4) The work monitoring method described in (3) above, wherein the length of the video is changed depending on the content of the work being performed by the worker or the type of the work.
[0012] (5) A work monitoring method as described in (3) or (4) above, wherein the video is captured before and after a determination is made as to whether or not the worker is performing the specified action on the work. Effect of the Invention
[0013] According to the present invention, the suitability of work performed by a worker can be determined with greater accuracy. [Brief description of the drawings]
[0014] [Figure 1] 1 is a diagram illustrating an outline of a configuration of a work monitoring system according to an embodiment of the present invention. [Diagram 2] 4 is a flow chart showing a control routine executed by a processor of a process PC to monitor work performed by an operator. [Diagram 3] FIG. 13 is a diagram showing an example of coordinate data of a human skeleton. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, like components are designated by like reference numerals.
[0016] Fig. 1 is a diagram showing a schematic configuration of a work monitoring system 1 according to an embodiment of the present invention. The work monitoring system 1 is installed, for example, in a factory for producing products (such as an automobile assembly factory) and monitors work performed by workers to produce the products. As shown in Fig. 1, the work monitoring system 1 includes a process personal computer (PC) 2, a process camera 3, a work camera 4, and a monitor 5.
[0017] The process PC2 is an example of a work monitoring device that monitors work performed by a worker. In this embodiment, the process PC2 detects inappropriate work performed by a worker in a manufacturing process and notifies the worker of the detection result. As shown in Fig. 1, the process PC2 includes an input / output interface 21, a memory 22, and a processor 23. The input / output interface 21 and the memory 22 are connected to the processor 23 via signal lines.
[0018] The input / output interface 21 has an interface circuit for connecting the process PC 2 to peripheral devices. Outputs from peripheral devices such as the process camera 3 and the work camera 4 are input to the process PC 2 via the input / output interface 21, and command signals generated by the processor 23 are output to peripheral devices such as the monitor 5 via the input / output interface 21.
[0019] The memory 22 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs, data, etc. that are used when the processor 23 executes various processes.
[0020] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include an arithmetic circuit such as a logic arithmetic unit or a numerical arithmetic unit.
[0021] The process camera 3 is installed at a predetermined position in the factory (e.g., on the ceiling, on top of a pillar, etc.) so as to capture images of workers in the manufacturing process. The process camera 3 captures an overhead image of the worker and generates an image of the worker. The process camera 3 is connected to the process PC 2 by wire or wirelessly, and the output of the process camera 3, i.e., the image generated by the process camera 3, is transmitted to the process PC 2. In this embodiment, the process camera 3 captures video of the worker and generates a video of the worker as the image of the worker.
[0022] The work camera 4 is installed in a factory so as to capture images of a workpiece (e.g., a vehicle part, etc.) being worked on by a worker. In this embodiment, the work camera 4 is attached to the head (e.g., the forehead) of the worker so as to capture images of the workpiece from the same viewpoint as the worker. Note that the work camera 4 may also be attached to another part of the worker, such as the neck. The work camera 4 is connected to the process PC2 by wire or wirelessly, and the output of the work camera 4, i.e., the image generated by the work camera 4, is transmitted to the process PC2. In this embodiment, the work camera 4 captures a video of the workpiece and generates the video of the workpiece as an image of the workpiece.
[0023] The monitor 5 has a display for displaying information and is provided near the worker so that the information can be visually recognized by the worker. The monitor 5 is connected to the process PC2 by wire or wirelessly, and displays information to be presented to the worker in response to the output from the process PC2.
[0024] In addition, a production instruction server 6 and a work procedure storage server 7 are connected to the process PC 2 and transmit predetermined information to the process PC 2. The production instruction server 6 is provided, for example, in a factory, and stores information related to production instructions for products produced in the factory (for example, product type, model number, lot number, production quantity, etc.). The work procedure storage server 7 is provided, for example, in a factory, and stores a work procedure manual that describes the procedures of work performed by workers. The production instruction server 6 and the work procedure storage server 7 may be integrated into one server. Also, the production instruction server 6 and the work procedure storage server 7 may be omitted, and the information stored by the production instruction server 6 and the work procedure storage server 7 may be stored in the memory 22 of the process PC 2.
[0025] The work monitoring method according to this embodiment is executed by a computer, and in this embodiment, is executed by the processor 23 of the process PC2. When a defect occurs in the work performed by the worker, it is desirable to notify the worker of the defect and prompt the worker to correct the work. Even if no defect occurs in the work performed by the worker, if the worker's actions are inappropriate, the production efficiency of the product may decrease. Furthermore, when a worker performs multiple actions, the appropriate state of the work after the work differs for each action, so it is difficult to judge whether the work is good or bad without identifying the content of the work.
[0026] Therefore, in this embodiment, the processor 23 of the process PC2 judges whether the work performed by the worker is appropriate based on the actions of the worker performing the work on the work and the state of the work performed by the worker. The actions of the worker are captured by the process camera 3, and the state of the work is captured by the work camera 4. Therefore, the processor 23 judges whether the work performed by the worker is appropriate based on the output of the process camera 3 and the output of the work camera 4.
[0027] Specifically, the processor 23 judges whether the worker is performing a predetermined operation on the workpiece based on the output of the process camera 3, and when it is judged that the worker is performing the predetermined operation, judges the quality of the workpiece based on the output of the workpiece camera 4. That is, the work monitoring method according to this embodiment includes a step of judging whether the worker is performing a predetermined operation on the workpiece, and a step of judging the quality of the workpiece when it is judged that the worker is performing the predetermined operation. According to the above method, the suitability of the work is judged taking into account the worker's actions in addition to the state of the workpiece, so that the suitability of the work can be judged more accurately. In addition, even when the worker performs multiple operations according to the work procedure manual, the work to be monitored can be identified based on the worker's actions, and thus the quality of the work can be judged more accurately.
[0028] Furthermore, when the processor 23 determines that the worker is not performing a predetermined operation on the workpiece, it notifies the worker of a warning. This makes it possible to prompt the worker to improve his / her operation when the worker is performing an inappropriate operation.
[0029] In addition, the processor 23 notifies the worker of the result of the quality judgment of the work. This makes it possible to prevent the worker from continuing the work without noticing the work's defect. Also, if a defect occurs in the work, it becomes possible to carry out the work's correction work before the work is sent to the next process.
[0030] 2 is a flowchart showing a control routine executed by the processor 23 of the process PC2 to monitor work performed by workers. This control routine is executed by the processor 23 when production of a product is started in the manufacturing process. When multiple workers in the manufacturing process are monitored, this control routine is executed for each of the multiple workers.
[0031] First, in step S101, the processor 23 acquires a work procedure manual from the work procedure storage server 7. For example, a work procedure manual stored in the work procedure storage server 7 is downloaded to the memory 22 of the process PC 2. When a plurality of products with different work procedures are handled by the worker to be monitored, the processor 23 acquires work procedures for all the products.
[0032] Next, in step S102, the processor 23 determines whether the production instruction has been switched. For example, when the type of product flowing through the manufacturing process is changed, the production instruction server 6 transmits a new production instruction to the process PC 2, and the processor 23 determines that the production instruction has been switched when the processor 23 receives the new production instruction from the production instruction server 6.
[0033] If it is determined in step S102 that the production instructions have been switched, this control routine proceeds to step S103. In step S103, the processor 23 updates the production information. For example, the processor 23 extracts a work procedure manual corresponding to the new production instructions, and updates the work procedure manual to be referred to in order to identify the worker's actions. Note that only the work procedure manual for the product to be produced may be acquired in step S101, and in step S103, the processor 23 may acquire the work procedure manual corresponding to the new production instructions from the work procedure storage server 7. On the other hand, if it is determined in step S102 that the production instructions have not been switched, this control routine skips step S103 and proceeds to step S104.
[0034] In step S104, processor 23 identifies a predetermined action to be performed by the worker based on the description in the work procedure manual. For example, if the work to be monitored determined from the work procedure manual is tightening a torque wrench, the action of rotating the torque wrench is identified as the predetermined action. If the work to be monitored determined from the work procedure manual is fitting a grommet seal, the action of fitting the grommet seal (e.g., pushing a hand toward a vehicle) is identified as the predetermined action. If the work to be monitored determined from the work procedure manual is part assembly work, the action of assembling the part on the workpiece is identified as the predetermined action. If the work to be monitored determined from the work procedure manual is spot welding, the action of pulling a hand back after spot welding is identified as the predetermined action.
[0035] Furthermore, when a worker performs a plurality of actions, the processor 23 identifies the predetermined actions to be performed by the worker based on the order of the actions described in the work procedure manual. That is, the processor 23 identifies the predetermined actions to be performed by the worker by taking into account the previous actions of the worker. Note that, when the work procedure manual includes information regarding the work time of each task, the processor 23 may identify the predetermined actions to be performed by the worker based on the work time.
[0036] Next, in step S105, the processor 23 determines whether or not the worker is performing a predetermined action based on the image of the worker captured by the process camera 3. In this embodiment, the process camera 3 constantly captures an image of the worker and periodically transmits the video of the worker to the process PC 2.
[0037] For example, the processor 23 uses a detector that has been trained in advance to detect the skeleton of the worker from the image data of the worker to determine whether the worker is performing a predetermined motion. Such a detector detects feature points such as human joints and detects coordinate data of the human skeleton using a machine learning model such as deep learning. FIG. 3 is a diagram showing an example of coordinate data of the human skeleton. In FIG. 3, black circles indicate coordinate points of feature points, and the human skeleton is shown by the connection state of multiple coordinate points. The processor 23 identifies the motion of the worker based on the coordinate data of the worker's skeleton, and determines whether the worker is performing the predetermined motion by comparing the motion of the worker with the predetermined motion.
[0038] If it is determined in step S105 that the worker is not performing the predetermined action, this control routine proceeds to step S106. In step S106, the processor 23 notifies the worker of a warning via the monitor 5. For example, the processor 23 displays characters or an image indicating that the worker's action is inappropriate on the monitor 5. Note that the processor 23 may notify the worker of an auditory warning, such as a buzzer sound or voice, instead of or in addition to a visual warning.
[0039] On the other hand, if it is determined in step S105 that the worker is performing a predetermined action, this control routine proceeds to step S107. In step S107, the processor 23 determines whether the workpiece worked by the worker is good or bad based on the image of the workpiece generated by the work camera 4. In this embodiment, the work camera 4 constantly captures images of the workpiece and periodically transmits videos of the workpiece to the process PC2.
[0040] For example, the processor 23 judges the quality of the workpiece worked by the worker using a classifier that has been trained in advance to output a result of judging the quality of the workpiece from image data of the workpiece. Examples of such classifiers include machine learning models such as neural networks, support vector machines, and random forests.
[0041] In this embodiment, the quality of the workpiece is judged based on an image of the workpiece generated by a work camera 4 worn by an operator so as to photograph the workpiece. This makes it possible to obtain a clear image of the workpiece without impeding the work performed by the operator, thereby improving the accuracy of the quality of the workpiece.
[0042] In this embodiment, the quality of the workpiece is judged based on a video of the workpiece. Therefore, the quality of the workpiece can be judged by taking into account the time-series changes in the workpiece when the worker performs the work, and the accuracy of the quality of the workpiece can be improved. In this case, the time-series image data of the workpiece is input to the classifier.
[0043] The length of the video of the workpiece may be changed depending on the type of workpiece or the content of the workpiece being performed by the worker. This allows the quality of the workpiece to be judged using a video of a length appropriate for the content of the workpiece or the type of workpiece, thereby improving the accuracy of the quality of the workpiece. The content of the workpiece is identified from the work procedure manual, and the type of workpiece is identified from information related to the production instructions.
[0044] Furthermore, in this embodiment, the video of the workpiece is taken before and after the timing at which it is determined whether the worker is performing a predetermined operation on the workpiece. In other words, the video of the workpiece is taken across the timing at which it is determined whether the worker is performing a predetermined operation on the workpiece. Therefore, regardless of the timing at which the appropriateness of the operation is determined, the quality of the workpiece can be determined using an image taken at a timing appropriate for determining whether the workpiece is good or bad, and the accuracy of the quality of the workpiece can be improved.
[0045] After step S107, in step S108, the processor 23 notifies the worker of the result of the pass / fail judgment of the workpiece via the monitor 5. For example, the processor 23 displays characters or an image indicating the result of the pass / fail judgment of the workpiece on the monitor 5. The processor 23 may notify the worker of the result of the pass / fail judgment of the workpiece with a buzzer sound or voice. The processor 23 may also notify the worker of at least one of a visual warning and an audible warning only when the result of the pass / fail judgment of the workpiece is bad.
[0046] After step S106 or step S108, this control routine proceeds to step S109. In step S109, the processor 23 determines whether or not the production of the product has been completed. For example, when the production of the product is completed, a production completion notification is sent from the production instruction server 6 to the process PC 2. If it is determined in step S109 that the production of the product has been completed, this control routine ends. On the other hand, if it is determined that the production of the product has not been completed, this control routine returns to step S102, and steps S102 to S108 are repeatedly executed until the determination in step S109 is affirmative.
[0047] If the worker to be monitored always performs the same work, steps S101 to S104 may be omitted.
[0048] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims. For example, the work camera 4 may be provided near the worker so as to capture an image of the work being worked on by the worker.
[0049] Also, the process camera 3 may be omitted, and a motion sensor that detects human movements may be attached to the worker. In this case, the processor 23 of the process PC 2 determines whether the worker is performing a predetermined movement on the workpiece based on the output of the motion sensor.
[0050] Furthermore, the work camera 4 and the monitor 5 may be integrally configured and worn by a worker as a wearable device (for example, smart glasses such as MR (Mixed Reality) glasses).
[0051] In addition, information may be transmitted wirelessly between the process PC2 and the peripheral devices, and the process PC2 may be provided outside the factory. The work monitoring method according to the present embodiment may be used to monitor work performed by workers in a facility such as a logistics facility. [Explanation of symbols]
[0052] 2 Process PC 23 Processors 3 Process Camera 4 Work Camera
Claims
1. A work monitoring method for monitoring work performed by a worker, comprising: A step of determining whether the worker is performing a predetermined action on a workpiece; a step of determining whether the workpiece is good or bad when it is determined that the worker is performing the predetermined operation; A method for monitoring work, comprising:
2. The work monitoring method according to claim 1 , wherein the quality of the work is determined based on an image of the work generated by a camera worn by the worker so as to photograph the work.
3. 3. The work monitoring method according to claim 1, wherein the quality of the work is judged based on a video of the work.
4. The work monitoring method according to claim 3 , wherein a length of the video is changed depending on the content of the work being performed by the worker or the type of the work.
5. The work monitoring method according to claim 3 , wherein the video is captured around a time when it is determined whether the worker is performing the predetermined action on the workpiece.
Citation Information
Patent Citations
Work operation inspection system
JP2016042332A
Inspection device
JP2018055370A
Work support system and work support method
JP2023028120A
Information processing system
JP2022155848A
