Work assistance system

Abstract

To enable confirmation of restarting when work being monitored is restarted after being interrupted.SOLUTION: A terminal device 10 detects a stage of work being carried out by a worker on the basis of the skeleton coordinate of the worker extracted from an image of an imaging device 20. Meanwhile, the terminal device 10 determines whether or not the detected stage of work is being carried out normally in prescribed order so as to monitor work. The terminal device 10 further determines that work has been normally restored when each stage of work is determined as being not carried out normally in prescribed order, and when it is detected after the determination that a stage of work relevant to execution order at the time immediately preceding the determination is being carried out by the worker.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a work support system. [Background technology]

[0002] A work support system described in Patent Document 1 is known as a system for supporting work such as part assembly at a manufacturing site. This work support system is equipped with a camera that captures images of the worker's hands. This work support system determines whether the parts have been assembled correctly based on the skeletal coordinates of the fingers acquired from the image captured by the camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-163293 A Summary of the Invention [Problem to be solved by the invention]

[0004] There are cases where a worker temporarily stops working. The work support system described above does not provide a means for confirming whether or not the worker will resume work after such a stoppage. [Means for solving the problem]

[0005] The work support system for solving the above problem is a system that supports a worker's work through a plurality of stages performed in a predetermined order, and includes an imaging device that images the worker, and a processing unit. The processing unit of the work support system is configured to detect the stage of the work being performed by the worker based on the skeletal coordinates of the worker extracted from the image of the imaging device, and to monitor the work by determining whether the detected stage of the work is being performed normally in the predetermined order, and when it is determined that each stage of the work is not being performed normally in the predetermined order, to determine that the work has returned to normal when it is detected that the stage of the work corresponding to the execution order at the time of the determination is being performed after the determination.

[0006] The processing unit of the task support system can be configured to detect the task stage based on the skeletal coordinates by using a learning model trained by machine learning. Effect of the Invention

[0007] The above-described work support system has an advantage that, when the work being monitored is resumed after being interrupted, the resumption of the work can be confirmed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a schematic plan view of a workplace. [Diagram 2] 1 is a diagram illustrating a schematic configuration of an embodiment of a work support system. [Diagram 3] FIG. 13 is a diagram showing an example of a setting mode of skeleton coordinates. [Figure 4] 4 is a flowchart of a work monitoring routine executed in the work support system. [Diagram 5] FIG. 13 is a diagram illustrating a configuration of a learning model for determining whether a task is correct or incorrect. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the work support system will be described in detail with reference to FIGS. <About the work to be supported> First, the work that the work support system of this embodiment supports will be described with reference to Fig. 1. In this embodiment, an example will be described in which the work support system is configured to support the assembly work of parts to a workpiece W. The work support system of this embodiment can also be configured to support work other than assembly work, such as welding work, painting work, and inspection work.

[0010] FIG. 1 shows a schematic plan view of a work area where parts are assembled. The assembly work here is performed by one worker P. In the work area, a workbench 30 is installed on which a workpiece W is placed during work. On the side of the workbench 30, part boxes 32A-32D are installed, in which parts A-D to be assembled to the workpiece W are respectively stored. On the side of the workbench 30 opposite to the side on which the part boxes 32A-32D are installed, bolt boxes 31a-31d are installed, in which bolts a-d used for assembling the parts A-D are respectively stored. Indicators 33 are installed in front of the bolt boxes 31a-31d and the part boxes 32A-32D. Furthermore, a photographing device 20 is installed in the work area. The photographing device 20 is installed so as to photograph the workbench 30, the bolt boxes 31a-31d, and the part boxes 32A-32D, as well as the worker P during work.

[0011] The assembly work is performed by taking out bolts a-d from bolt boxes 31a-31d, taking out parts A-D from parts boxes 32A-32D, and assembling parts A-D to workpiece W using bolts a-d. Each of parts A-D is assembled to workpiece W in a predetermined order. Also, each of parts A-D is assembled to workpiece W using different bolts a-d. Specifically, bolt a is used to assemble part A, bolt b is used to assemble part B, bolt c is used to assemble part C, and bolt d is used to assemble part D.

[0012] <About the configuration of the work support system> Next, the configuration of the work support system of this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the work support system includes a terminal device 10 and an image capture device 20.

[0013] The functions of the terminal device 10 may be realized by any hardware or any combination of hardware and software. An example of the configuration of the terminal device 10 shown in FIG. 1 includes a main storage device 101, an auxiliary storage device 102, a CPU 103, an interface device 104, a display device 105, and an input device 106, and a bus B1 connecting them. The auxiliary storage device 102 stores a program for realizing the functions of the terminal device 10, and stores necessary files and data. When a program start instruction is received, the main storage device 101 reads out and stores the program from the auxiliary storage device 102. The CPU 103 executes various programs stored in the main storage device 101. The terminal device 10 realizes various functions according to the program executed by the CPU 103. The interface device 104 is used as an interface for connecting to an external communication line B2. The image capture device 20 and the notification device 21 are connected to the interface device 104 via the communication line B2. The display device 105 displays, for example, a GUI (Graphical User Interface) according to a program executed by the CPU 103. The input device 106 is used to allow a user or an administrator of the terminal device 10 to input various operation instructions related to the terminal device 10. In the work assistance system of this embodiment, the terminal device 10 corresponds to a processing unit.

[0014] The above-mentioned photographing device 20 is connected to the terminal device 10 via the communication line B2. The notification device 21 is also connected to the terminal device 10 via the communication line B2. The notification device 21 is a device that notifies the worker P of the correctness or incorrectness of the work through hearing and / or vision. Examples of the notification device 21 are a speaker, a buzzer, a monitor, and an electronic bulletin board.

[0015] <About determining whether skeletal coordinates and tasks are correct> While worker P is working, the photographing device 20 transmits the photographed image to the terminal device 10. The terminal device 10 extracts skeletal coordinates C1 to C9 of worker P from the image received from the photographing device 20. The skeletal coordinates C1 to C9 represent the positions of each part of worker P in the image. The extraction of the skeletal coordinates C1 to C9 from the image is performed using, for example, artificial intelligence.

[0016] 3 shows an example of setting skeletal coordinates C1 to C9. In this example, the head (C1), left and right shoulder joints (C2, C3), left and right elbow joints (C4, C5), left and right hands (C6, C7), and left and right hip joints (C8, C9) of the worker P are set as extraction points of skeletal coordinates. In this embodiment, the skeletal coordinates C1 to C9 are two-dimensional coordinates, but in cases where image data from the imaging device 20 includes depth information of the subject, the skeletal coordinates C1 to C9 may be three-dimensional coordinates.

[0017] The work support system of this embodiment performs work monitoring by determining whether a work is normal or abnormal based on the skeleton coordinates C1 to C9. An abnormality in work here indicates a state in which the worker P has suspended the work, for example, when leaving the work to clean. The terminal device 10 determines whether a work is normal or abnormal in the following manner.

[0018] In this embodiment, a series of actions of the worker P from the start to the end of the part assembly work is divided into a plurality of stages in time series. When the worker P is working normally without interruption, the actions of the worker P are generally uniform. Therefore, when the work is being performed normally, each of the skeletal coordinates C1 to C9 of the worker P at each stage falls within a certain range. In the following description, the distribution range of each of the skeletal coordinates C1 to C9 of the worker P at each stage when the work is being performed normally is referred to as a normal frame. When the worker P interrupts the work and takes an action outside the work, at least one of the skeletal coordinates C1 to C9 of the worker P deviates from the normal frame. Therefore, if the normal frame of each of the skeletal coordinates C1 to C9 for each stage is obtained in advance, it is possible to determine whether the work is being performed normally or not depending on whether each of the skeletal coordinates C1 to C9 is within the normal frame of the current stage.

[0019] In the following description, the order of a work stage when the work stages are arranged in chronological order is referred to as the frame number FN. The terminal device 10 is aware of the current work stage based on the value of the frame number FN. The terminal device 10 counts up the frame number FN each time the work stage progresses. When the count-up causes the value of the frame number FN to exceed the value indicating the final work stage, the terminal device 10 resets the value of the frame number FN to its initial value of "1."

[0020] <About work monitoring> Next, work monitoring by the work support system of this embodiment will be described with reference to Fig. 4. Fig. 4 shows a flowchart of a work monitoring routine executed by the terminal device 10 for work monitoring. While work monitoring is being performed, the terminal device 10 repeatedly executes this routine at every predetermined control period.

[0021] When starting this routine, the terminal device 10 first acquires an image captured by the photographing device 20 (S10). Then, the terminal device 10 extracts skeleton coordinates C1 to C9 of the worker P from the acquired image (S12). Next, the terminal device 10 determines whether or not an interruption flag is set (S14). The interruption flag is a flag that indicates whether or not the work is interrupted.

[0022] If the interruption flag is not set (S14: NO), the terminal device 10 judges whether or not each of the skeleton coordinates C1 to C9 of the worker P is within the normal frame corresponding to the current frame number FN (S16). Then, if each of the skeleton coordinates C1 to C9 is within the normal frame (S16: YES), the terminal device 10 judges that the work is normal. In this case, the terminal device 10 counts up the frame number FN (S18) and then ends the processing of this routine in the current control cycle. On the other hand, if one or more of the skeleton coordinates C1 to C9 are outside the normal frame (S16: NO), the terminal device 10 judges that the work is abnormal. In this case, the terminal device 10 sets the interruption flag (S20), starts measuring the interruption time (S22), and then ends the processing of this routine in the current control cycle. The interruption time represents the time that has elapsed since the terminal device 10 judged that the work is abnormal.

[0023] If the interruption flag is set (S14: YES), the terminal device 10 judges whether each of the skeleton coordinates C1 to C9 is within the pre-interruption regular frame (S24). The pre-interruption regular frame is a regular frame corresponding to the stage of the task in the execution order immediately before it is judged that the skeleton coordinates C1 to C9 are out of the regular frame. Specifically, the pre-interruption regular frame is a regular frame corresponding to the frame number (=FN-1) that is one less than the current value.

[0024] If each of the skeleton coordinates C1 to C9 is within the normal frame before interruption (S24: YES), the terminal device 10 clears the interruption flag (S26) and ends the measurement of the interruption time (S28). Then, the terminal device 10 ends the processing of this routine in the current control cycle.

[0025] On the other hand, if one or more of the skeleton coordinates C1 to C9 deviate from the normal frame before interruption, the terminal device 10 determines that the interruption of the work continues (S32). In this case, the terminal device 10 determines whether each of the skeleton coordinates C1 to C9 is within the initial frame. The initial frame represents the normal frame of the first stage of the work, that is, the normal frame at the start of the work.

[0026] Here, when the terminal device 10 judges that each of the skeleton coordinates C1 to C9 is within the initial frame (S32: YES), it judges that a new task has started and that the task has been redone from the first stage. In this case, the terminal device 10 resets the value of the frame number FN to "1" (S34), and then ends the task of this routine in the current control cycle.

[0027] On the other hand, when the terminal device 10 determines that one or more of the skeleton coordinates C1 to C9 are out of the initial frame (S32: NO), it determines whether the interruption time is equal to or longer than a preset time TX (S36). If the interruption time is less than "TX", the terminal device 10 ends the processing of this routine in the current control cycle. On the other hand, if the interruption time is equal to or longer than "TX" (S36: YES), the terminal device 10 stops work monitoring (S38). That is, the terminal device 10 stops the execution of the processing of this routine for each control cycle until work monitoring is resumed manually or the like.

[0028] <About determining whether work is correct or not> Next, the determinations at S16, S24, and S32 in Fig. 4 will be described with reference to Fig. 5. The terminal device 10 determines the type of work to be performed using the learning model shown in Fig. 5. This learning model is learned by machine learning. The auxiliary storage device 102 of the terminal device 10 stores the learned learning model in advance.

[0029] The learning model in Fig. 5 is configured as a neural network having an input layer consisting of nine nodes, an intermediate layer consisting of M nodes, and an output layer consisting of L nodes. The input values ​​X1, X2, ..., X9 of each node in the input layer are the skeleton coordinates C1 to C9 of the worker P. The output values ​​Y1, Y2, ..., YL of each node in the output layer are the likelihood of each stage of the work.

[0030] Here, the input values ​​to each node in the middle layer are U1, U2, ..., UM, and the output values ​​of each node in the middle layer are Z1, Z2, ..., ZM. In addition, "i" is a natural number from "1" to "9", "j" is a natural number from "1" to "M", and "k" is a natural number from "1" to "L". The input value Uj of each node in the middle layer is calculated as the sum of the values ​​obtained by multiplying the input values ​​X1, X2, ..., X9 of the input layer by the weight Vij. The output value Zj of each node in the middle layer is calculated as the return value of the activation function F with the input value Uj of the node as an argument. An example of the activation function F is the sigmoid function. The output values ​​Y1, Y2, ..., YL of each node in the output layer are calculated as the sum of the values ​​obtained by multiplying the output values ​​Zj of each node in the middle layer by the weight Wjk.

[0031] Such a learning model is trained using a data set consisting of skeleton coordinates C1 to C9 and the likelihood of each task stage as training data. The values ​​of skeleton coordinates C1 to C9 in the data set are extracted from the video of the worker P captured by the imaging device 20. The likelihood values ​​of each task stage in the data set are set so that the likelihood value of the task stage that the worker P was performing at the time of capture is set to "1" and the likelihood values ​​of the other tasks stages are set to "0."

[0032] The terminal device 10 performs the following process for each data set to learn the learning model. The terminal device 10 inputs the time series data of the skeleton coordinates C1 to C9 in the data set to the neural network as the input values ​​X1 to XN of the input layer, and calculates the output values ​​Y1 to YL of the neural network for the input. The terminal device 10 then corrects the values ​​of the weights Vij and Wjk using the error backpropagation method so that the error between the calculated output values ​​Y1 to YL and each value of the one-dimensional array indicating the type of work in the data set is reduced. The terminal device 10 repeats the correction process of the weights Vij and Wjk until the error is equal to or smaller than a predetermined value, thereby learning the neural network. Note that such learning of the learning model may be performed by a device other than the terminal device 10, and the learned learning model may be stored in the auxiliary storage device 102.

[0033] The terminal device 10 performs the determinations of S16, S24, and S32 in FIG. 4 in the following manner. First, the terminal device 10 inputs the skeleton coordinates C1 to C9 of the worker P extracted from the image to the input layer of the learning model as the values ​​of each input value U1 to U9, respectively. The terminal device 10 calculates the output values ​​Y1 to YL of the learning model for each input. Then, when the maximum value among the output values ​​Y1 to YL is equal to or greater than a predetermined threshold, the terminal device 10 determines that the task stage corresponding to the maximum value is the task stage currently being performed by the worker P. In this way, the terminal device 10 detects the task stage currently being performed by the worker P based on the skeleton coordinates C1 to C9.

[0034] Then, for example, in S16 of Fig. 4, if the stage of the work being performed by the detected worker P matches the stage of the work corresponding to the current frame number FN, the terminal device 10 determines that each of the skeletal coordinates C1 to C9 of the worker P is within the regular frame. Also, in S24 of Fig. 4, if the stage of the work being performed by the detected worker P matches the stage of the work with the frame number value "FN-1", the terminal device 10 determines that each of the skeletal coordinates C1 to C9 of the worker P is within the pre-interruption regular frame. Furthermore, in S32 of Fig. 4, if the stage of the work being performed by the detected worker P is the stage of the work that is first in the order of performance in the work, the terminal device 10 determines that each of the skeletal coordinates C1 to C9 is within the initial frame.

[0035] <Actions and Effects of the Embodiment> The operation and effects of this embodiment will be described. The terminal device 10 detects the stage of the work being performed by the worker P based on the transition of the skeletal coordinates C1 to C9 of the worker P extracted from the image captured by the image capturing device 20. The skeletal coordinates C1 to C9 represent the positions of the torso, limbs, head, etc. of the worker P. Therefore, based on the skeletal coordinates C1 to C9, it is possible to confirm at what stage the worker P is currently in among the series of movements of the worker P from the start to the end of the work. The terminal device 10 then performs work monitoring by determining whether the detected work stages are being performed normally in the appropriate order.

[0036] There are cases where the worker P temporarily stops the work during the work. In such a case, the skeleton coordinates C1 to C9 of the worker P are out of the normal frame. Therefore, the terminal device 10 at this time determines that the work is not being performed normally.

[0037] After this determination, the terminal device 10 determines that the work has been resumed if it detects that the skeleton coordinates C1 to C9 of the worker P are within the pre-interruption regular frame. The pre-interruption regular frame is a regular frame corresponding to the stage of the work that corresponds to the execution order immediately before the interruption. When resuming the interrupted work, the worker P resumes the interrupted action. Therefore, the resumption of the work can be determined by the skeleton coordinates C1 to C9 being located within the pre-interruption regular frame. Note that the terminal device 10 stops monitoring the work if it is not determined that the work has returned to normal within a preset time TX from the time when it was determined that the work was not being performed normally.

[0038] There are cases where the worker P abandons a task midway and starts a new task. When the terminal device 10 detects that the skeleton coordinates C1 to C9 of the worker P are within the initial frame after the interruption of the task, it determines that a new task has started. Then, the terminal device 10 resets the number of frames FN to the initial value "1" and continues monitoring the task.

[0039] According to the work assistance system of the present embodiment described above, the following effects can be obtained. (1) The terminal device 10 detects the stage of the work being performed by the worker P based on the skeleton coordinates C1 to C9 of the worker P extracted from the image of the photographing device 20. The terminal device 10 then monitors the work by determining whether the detected work stages are being performed normally in a predetermined order. Furthermore, if the terminal device 10 determines that each work stage is not being performed normally in a predetermined order, and detects after the determination that the worker P is performing the work stage that corresponds to the execution order immediately before the determination, the terminal device 10 determines that the work has returned to normal. Therefore, the work support system of this embodiment can confirm the resumption of the interrupted work.

[0040] (2) The terminal device 10 stops monitoring the work if it is not determined that the work has returned to normal within a preset time TX from the time when it is determined that the work is not being performed normally. Therefore, if the work is interrupted for a long period of time, the work monitoring can be automatically stopped.

[0041] (3) When the terminal device 10 detects that the worker P is performing the first task in the sequence after determining that the task is not being performed normally, the terminal device 10 determines that a new task has started. Therefore, even if the worker P abandons a task midway and starts a new task, the task monitoring can be continued.

[0042] (4) The terminal device 10 detects the work stage based on the skeleton coordinates C1 to C9 using a learning model learned by machine learning. There is variation in the physique of the worker P and the movements during the work. By using the learning model, it is possible to easily detect the work stage that reflects such variation.

[0043] (Other embodiments) The above embodiment can be modified as follows: The above embodiment and the following modifications can be combined with each other to the extent that no technical contradiction occurs.

[0044] The task support system of the above embodiment may be configured not to execute the process of determining whether or not a new task has been restarted. Specifically, the processes of S32 and S34 in FIG. In the above embodiment, the terminal device 10 stops work monitoring if it does not determine that the work has returned to normal within a preset time TX after the work is interrupted. The timing of stopping work monitoring after the work is interrupted may be changed. For example, the terminal device 10 may be configured to immediately stop work monitoring at the time when it is determined that the work is not being performed normally. Furthermore, the terminal device 10 may be configured not to automatically stop work monitoring after the work is interrupted.

[0045] A learning model having a configuration different from that shown in Fig. 5 may be used as a learning model used to detect the stage of a task based on the skeleton coordinates C1 to C9. Also, the stage of a task may be detected without using a learning model trained by machine learning.

[0046] The number and positions of the skeleton coordinates of worker P extracted from the image may be changed. [Explanation of symbols]

[0047] 10 Terminal Equipment 101 Main storage 102 Auxiliary storage 103 CPU 104 Interface device 105 Display device 106 Input Devices 20 Imaging Equipment 21 Notification device 22 Control circuit 30 Workbench 31a~31d Bolt box 32A~32D Parts Box B1 Bus B2 Communication Line P worker Double work

Claims

1. A system for assisting a worker in performing a task through a plurality of steps performed in a predetermined order, comprising: The system includes an imaging device that images the worker, and a processing unit, The processing unit includes: detecting a stage of the work being performed by the worker based on the skeleton coordinates of the worker extracted from the image of the photographing device, and determining whether the detected stage of the work is being performed normally in the predetermined order, thereby monitoring the work; when it is determined that each stage of the work is not being performed normally in the predetermined order, if it is detected after the determination that the worker is performing the stage of the work that corresponds to the execution order at the time immediately before the determination, it is determined that the work has returned to normal; This is a work support system.

2. The work support system according to claim 1 , wherein the processing unit stops the work monitoring when it is determined that the work is not being performed normally.

3. The work support system according to claim 1 , wherein the processing unit stops the work monitoring when it is not determined that the work has returned to normal within a predetermined time from the time when it is determined that the work is not being performed normally.

4. 2. The work support system according to claim 1, wherein the processing unit determines that the work has been newly started when it detects that the worker is performing the first stage of the work after determining that the work is not being performed normally.

5. The task support system according to claim 1 , wherein the processing unit detects the task stage based on the skeletal coordinates by using a learning model trained by machine learning.

Images