Work support system

Abstract

To improve accuracy of determining whether the work is correct.SOLUTION: A work support system includes: an imaging apparatus 20 which images a workplace where a worker works; and a terminal device 10 which determines whether the work is correct on the basis of time-series data of skeleton coordinates of the worker extracted from the images of the imaging apparatus 20. The terminal device 10 determines whether hands of the worker are positioned in a prescribed work area in determination, on the basis of the images. The terminal device 10 performs the determination without using time-series data of skeleton coordinates obtained when determining that the hands of the worker are not positioned in the work area.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a work support system. [Background technology]

[0002] A known system for detecting the actions of a person in a video is described in Patent Document 1. This system detects actions by extracting the skeletal coordinates of the person for each frame of the video and calculating, from the skeletal coordinates, motion features that represent the person's posture and its changes over time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-003434 A Summary of the Invention [Problem to be solved by the invention]

[0004] At manufacturing sites, tasks such as part assembly, welding, painting, and inspection are performed. It is conceivable to judge whether such tasks are performed correctly based on the skeletal coordinates of the worker extracted from video. In between tasks, the worker may take actions that are not directly related to the task, such as wiping sweat. In such a correctness judgment based on skeletal coordinates, if the worker takes an action that is not directly related to the task, it may be erroneously judged that the worker is performing the wrong task. [Means for solving the problem]

[0005] A work support system that solves the above problem includes a photographing device that photographs a workplace where a worker performs work, and a processing unit that performs a judgment on the correctness of the work based on time series data of the worker's skeletal coordinates extracted from an image of the photographing device, wherein the processing unit is configured to determine whether or not the worker's hands are located within a predetermined work area based on the image, and to perform the judgment on the correctness without using the time series data of the skeletal coordinates when it is determined that the worker's hands are not located within the work area.

[0006] The task assistance system can be configured to perform the correct / incorrect judgment using a learning model trained by machine learning, for example. Effect of the Invention

[0007] The above-described task support system has the effect of improving the accuracy of determining whether a task is correct or incorrect. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a schematic plan view of a workplace. [Diagram 2] 1 is a diagram illustrating a schematic configuration of an embodiment of a work support system. [Diagram 3] FIG. 13 is a diagram showing an example of a setting mode of skeleton coordinates. [Figure 4] 4 is a flowchart showing a flow of processes executed in the work support system. [Diagram 5] 1 is a diagram illustrating a configuration of a learning model used by the work support system for carrying-in determination. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the work support system will be described in detail with reference to FIGS. <About the work to be supported> First, the work that the work support system of this embodiment supports will be described with reference to Fig. 1. In this embodiment, an example will be described in which the work support system is configured to support the assembly work of parts to a workpiece W. The work support system of this embodiment can also be configured to support work other than assembly work, such as welding work, painting work, and inspection work.

[0010] FIG. 1 shows a schematic plan view of a work area where parts are assembled. The assembly work here is performed by one worker P. In the work area, a workbench 30 is installed on which a workpiece W is placed during work. On the side of the workbench 30, part boxes 32A-32D are installed, in which parts A-D to be assembled to the workpiece W are respectively stored. On the side of the workbench 30 opposite to the side on which the part boxes 32A-32D are installed, bolt boxes 31a-31d are installed, in which bolts a-d used for assembling the parts A-D are respectively stored. Indicators 33 are installed in front of the bolt boxes 31a-31d and the part boxes 32A-32D. Furthermore, a photographing device 20 is installed in the work area. The photographing device 20 is installed so as to photograph the workbench 30, the bolt boxes 31a-31d, and the part boxes 32A-32D, as well as the worker P during work.

[0011] The assembly work is performed by taking out bolts a-d from bolt boxes 31a-31d, taking out parts A-D from parts boxes 32A-32D, and assembling parts A-D to workpiece W using bolts a-d. Each of parts A-D to be attached to workpiece W is determined in advance. Each of parts A-D is assembled to workpiece W using a different bolt a-d. Specifically, bolt a is used to assemble part A, bolt b is used to assemble part B, bolt c is used to assemble part C, and bolt d is used to assemble part D.

[0012] In the work support system of this embodiment, an image of a work site captured by the image capturing device 20 is divided into a work area R1 and a non-work area R2. As shown in FIG. 1, the work area R1 includes a workbench 30, bolt boxes 31a to 31d, and parts boxes 32A to 32D. A worker P performs an assembly work using his / her own hands. The work area R1 is set so as to include all the places where the hands of the worker P are positioned when the worker P is performing an action directly related to the work. In contrast, the non-work area R2 is set as an area other than the work area R1, that is, an area where the hands of the worker P are not positioned when the worker P is performing an action directly related to the work.

[0013] <About the Work Support System Configuration> Next, the configuration of the work support system of this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the work support system includes a terminal device 10 and an image capture device 20.

[0014] The functions of the terminal device 10 may be realized by any hardware or any combination of hardware and software. An example of the configuration of the terminal device 10 shown in FIG. 1 includes a main storage device 101, an auxiliary storage device 102, a CPU 103, an interface device 104, a display device 105, and an input device 106, and a bus B1 connecting them. The auxiliary storage device 102 stores a program for realizing the functions of the terminal device 10, and stores necessary files and data. When a program start instruction is received, the main storage device 101 reads out and stores the program from the auxiliary storage device 102. The CPU 103 executes various programs stored in the main storage device 101. The terminal device 10 realizes various functions according to the program executed by the CPU 103. The interface device 104 is used as an interface for connecting to an external communication line B2. The image capture device 20 and the notification device 21 are connected to the interface device 104 via the communication line B2. The display device 105 displays, for example, a GUI (Graphical User Interface) according to a program executed by the CPU 103. The input device 106 is used to allow a user or an administrator of the terminal device 10 to input various operational instructions relating to the terminal device 10 .

[0015] The above-mentioned photographing device 20 is connected to the terminal device 10 via the communication line B2. Also connected to the terminal device 10 via the communication line B2 are a notification device 21 and a control circuit 22 for the indicators 33. The notification device 21 is a device that notifies the worker P of the correctness of the work through hearing and / or vision. Examples of the notification device 21 are a speaker, a buzzer, a monitor, and an electronic bulletin board. The control circuit 22 is an electronic circuit that turns on / off each indicator 33 in response to an instruction from the terminal device 10.

[0016] <About time series data of skeletal coordinates> While worker P is working, the photographing device 20 transmits the photographed image to the terminal device 10. The terminal device 10 extracts skeletal coordinates C1 to C9 of worker P from the image received from the photographing device 20. The skeletal coordinates C1 to C9 represent the positions of each part of worker P in the image. The extraction of the skeletal coordinates C1 to C9 from the image is performed using, for example, artificial intelligence.

[0017] 3 shows an example of setting skeletal coordinates C1 to C9. In this example, the head (C1), left and right shoulder joints (C2, C3), left and right elbow joints (C4, C5), left and right hands (C6, C7), and left and right hip joints (C8, C9) of the worker P are set as extraction points of skeletal coordinates. In this embodiment, the skeletal coordinates C1 to C9 are two-dimensional coordinates, but in cases where image data from the imaging device 20 includes depth information of the subject, the skeletal coordinates C1 to C9 may be three-dimensional coordinates.

[0018] The main memory device 101 of the terminal device 10 is provided with a storage area for storing the skeleton coordinates C1 to C9. The storage area is configured to be able to store a maximum of Q sets of skeleton coordinates C1 to C9. The storage area is configured as a FIFO (First-In First-Out) type data structure. In other words, when new skeleton coordinates C1 to C9 are added to the storage area when Q sets of skeleton coordinates C1 to C9 have already been stored in the storage area, the oldest set of skeleton coordinates C1 to C9 is deleted from the storage area.

[0019] <About work support processing> Next, the process for work support executed by the terminal device 10 in the work support system of this embodiment will be described with reference to Fig. 4. Fig. 4 shows a flowchart of the process executed by the terminal device 10 for work support. The terminal device 10 repeatedly executes the process of Fig. 4 at every predetermined control period while the worker P is performing the work.

[0020] When the terminal device 10 starts the process of FIG. 4, it first issues an instruction for the next work to be performed by the worker P (S10). In the following description, the next work to be performed by the worker P will be referred to as the next work. Information on the work procedure is stored in the auxiliary storage device 102 of the terminal device 10. The terminal device 10 identifies the next work based on that information. The instruction for the next work is issued by lighting up the indicators 33 installed on the bolt boxes 31a to 31d and the parts boxes 32A to 32D. For example, when the next work is "removing part A," the terminal device 10 lights up the indicator 33 of the parts box 32A in which part A is stored.

[0021] Next, the terminal device 10 acquires an image captured by the image capturing device 20 (S12). Then, the terminal device 10 extracts skeletal coordinates C1 to C9 of the worker P from the acquired image (S14). Next, the terminal device 10 determines whether or not the hand of the worker P is located within the working area R1 (S16). Specifically, the terminal device 10 determines whether or not at least one of the skeletal coordinates C6, C7 of the left and right hands is within the working area R1.

[0022] When the terminal device 10 determines that the hands of the worker P are not located within the working area R1 (S16: NO), it clears all of the skeleton coordinates C1 to C9 stored in the memory area (S22). Then, the terminal device 10 ends the process of FIG. 4 in the current control cycle.

[0023] On the other hand, when the terminal device 10 determines that the hand of the worker P is located within the working area R1 (S16: YES), it stores the currently extracted skeletal coordinates C1 to C9 in the storage area (S20). Then, the terminal device 10 determines whether or not Q sets of skeletal coordinates C1 to C9 are stored in the storage area (S24). When the terminal device 10 determines that the number of skeletal coordinates C1 to C9 stored in the storage area is less than Q sets (S24: NO), it ends the process of FIG. 4 in the current control cycle.

[0024] On the other hand, when the terminal device 10 determines that the Q sets of skeleton coordinates C1 to C9 are stored in the storage area (S24: YES), it generates time series data in which the Q sets of skeleton coordinates C1 to C9 are arranged in chronological order (S26). Then, the terminal device 10 determines the type of work performed by the worker P based on the generated time series data (S28). Details of the determination of the type of work will be described later. In the following description, the work whose type has been determined will be referred to as a performed work.

[0025] Next, the terminal device 10 judges whether the performed work, the type of which has been judged, matches the next work instructed in S10 (S30). If the terminal device 10 judges that the performed work and the next work match (S30: YES), it judges that the next work instructed in S10 has been completed. In this case, the terminal device 10 updates the next work (S32) and ends the process of FIG. 4 in the current control cycle. On the other hand, if the terminal device 10 judges that the performed work and the next work do not match (S30: NO), it judges that the worker P performed an incorrect work. In this case, the terminal device 10 commands the notification device 21 to notify the worker P of the incorrect work (S34) and ends the process of FIG. 4 in the current control cycle. The notification of the incorrect work to the worker P may be a simple warning, or may be a notification of the specific content of the incorrect work.

[0026] <Determination of the type of actual work> Next, the determination of the type of work to be performed in S28 of Fig. 4 will be described with reference to Fig. 5. The terminal device 10 performs the determination of the type of work to be performed using the learning model shown in Fig. 5. This learning model is learned by machine learning. The auxiliary storage device 102 of the terminal device 10 stores the learned learning model in advance.

[0027] The learning model in Fig. 5 is configured as a neural network having an input layer consisting of N nodes, an intermediate layer consisting of M nodes, and an output layer consisting of L nodes. The input values ​​X1, X2, ..., XN of each node in the input layer are time-series data of skeleton coordinates C1 to C9 of worker P. The output values ​​Y1, Y2, ..., YL of each node in the output layer are likelihoods for each type of work. The types of work include each work performed by worker P when assembling parts.

[0028] Here, the input values ​​to each node in the middle layer are U1, U2, ..., UM, and the output values ​​of each node in the middle layer are Z1, Z2, ..., ZM. Also, "i" is a natural number from "1" to "N", "j" is a natural number from "1" to "M", and "k" is a natural number from "1" to "L". The input value Uj of each node in the middle layer is calculated as the sum of the values ​​obtained by multiplying the input values ​​X1, X2, ..., XN of the input layer by the weight Vij. The output value Zj of each node in the middle layer is calculated as the return value of the activation function F with the input value Uj of the node as an argument. An example of the activation function F is the sigmoid function. The output values ​​Y1, Y2, ..., YL of each node in the output layer are calculated as the sum of the values ​​obtained by multiplying the output values ​​Zj of each node in the middle layer by the weight Wjk.

[0029] Such a learning model is trained using a data set as training data, which is configured with time series data of skeletal coordinates C1 to C9 when worker P performs each task, and a one-dimensional array in which the likelihood of the task performed by worker P is set to 1 and the likelihood of other tasks is set to 0. The data set is created by acquiring time series data of skeletal coordinates C1 to C9 of worker P from the video of the image capture device 20 when worker P performs each task. The training data also includes a data set configured with time series data of skeletal coordinates C1 to C9 when worker P takes an action not directly related to the task with either the left or right hand positioned within the working area R1, and a one-dimensional array in which the likelihood of all tasks is set to 0.

[0030] The terminal device 10 performs the following process for each data set to learn the learning model. The terminal device 10 inputs the time series data of the skeleton coordinates C1 to C9 in the data set to the neural network as the input values ​​X1 to XN of the input layer, and calculates the output values ​​Y1 to YL of the neural network for the input. The terminal device 10 then corrects the values ​​of the weights Vij and Wjk using the error backpropagation method so that the error between the calculated output values ​​Y1 to YL and each value of the one-dimensional array indicating the type of work in the data set is reduced. The terminal device 10 repeats the correction process of the weights Vij and Wjk until the error is equal to or smaller than a predetermined value, thereby learning the neural network. Note that such learning of the learning model may be performed by a device other than the terminal device 10, and the learned learning model may be stored in the auxiliary storage device 102.

[0031] In S28 of Fig. 4, the terminal device 10 inputs the time-series data of the skeleton coordinates C1 to C9 generated in S26 into the learning model as the values ​​of the input values ​​X1 to XN. Next, the terminal device 10 calculates the output values ​​Y1 to YL of the learning model for that input. Then, the terminal device 10 determines the type of the work being performed by finding the maximum value among the calculated output values ​​Y1 to YL. In practice, the terminal device 10 reserves the determination if any of the output values ​​Y1 to YL does not reach a predetermined value. In this case, the terminal device 10 ends the processing of Fig. 4 for the current control cycle without performing the processing of S30 to S34.

[0032] <Actions and Effects of the Embodiments> The operation and effects of this embodiment will be described. The work support system of this embodiment includes a camera 20 that captures images of a workplace where a worker P performs work, and a terminal device 10 that performs a judgment on the correctness of a work based on time-series data of skeletal coordinates C1-C9 of the worker P extracted from an image of the camera 20. From the time-series data of the skeletal coordinates C1-C9, it is possible to identify the motions being performed by the worker P. Therefore, based on the time-series data of the skeletal coordinates C1-C9, it is possible to identify the type of work being performed by the worker P and judge whether the work is correct or incorrect.

[0033] In addition, during breaks in work, the worker P may take actions that are not directly related to the work, such as scratching his head, wiping sweat, or adjusting his glasses. In order to ensure the accuracy of the correct / incorrect judgment, it is necessary to prevent such non-work actions from being judged as incorrect work. If the time-series data of the skeleton coordinates C1 to C9 when the worker P takes non-work actions is also used for the correct / incorrect judgment, the learning model used to judge the type of work is required to be able to reliably distinguish between work and non-work actions. However, since there are countless types of actions that the worker P can take as non-work actions, it is difficult to list them all. Therefore, it is difficult to train a learning model so that it can distinguish between work and non-work actions.

[0034] As described above, in this embodiment, the image captured by the image capture device 20 is divided into a working area R1 and a non-working area R2. In the working area R1, an area in the image where the hands of the worker P may be located when taking out bolts or parts and assembling them to the workpiece W is set. Therefore, when the worker P is performing work to assemble parts, at least one of the left and right hands is considered to be located within the working area R1. In contrast, when both hands are located within the non-working area R2, the worker P is considered not to be performing work to assemble parts.

[0035] When determining whether a task is correct or incorrect, the terminal device 10 in the task support system of this embodiment determines whether the hand of the worker P is located within the predefined task area R1 based on the image. The terminal device 10 performs the determination without using the time-series data of the skeletal coordinates C1-C9 when it is determined that the hand of the worker P is not located within the task area R1. Therefore, in the task support system of this embodiment, the task is determined to be correct or incorrect after most of the time-series data of the skeletal coordinates C1-C9 corresponding to actions other than the task are excluded in advance.

[0036] According to the work assistance system of the present embodiment described above, the following effects can be obtained. (1) The terminal device 10 judges whether or not the hands of the worker P are located within the predefined working area R1 based on the image of the photographing device 20. Then, the terminal device 10 judges whether the work is correct or incorrect without using the time-series data of the skeleton coordinates when it is determined that the hands of the worker P are not located within the working area R1. It becomes difficult to judge whether the work is correct or incorrect based on the time-series data of the skeleton coordinates C1 to C9 when the worker P is performing an action other than the work. Therefore, the work support system of this embodiment has the effect of improving the accuracy of the judgment of the correctness of the work.

[0037] (2) In the correctness judgment, the terminal device 10 judges whether a plurality of tasks for part assembly are performed in a predetermined order or not. Therefore, it can accurately judge whether the tasks are performed in the correct order or not.

[0038] (3) The work support system of this embodiment includes an indicator 33 and a control circuit 22 for instructing the worker P on the next task to be performed. Furthermore, the terminal device 10 of the work support system of this embodiment is configured to be able to determine whether each task related to part assembly is correct or incorrect. The terminal device 10 is also configured to perform a correct or incorrect determination of the task instructed by the indicator 33. Therefore, the terminal device 10 can inform the worker P of the work procedure and accurately determine whether the task is being performed according to the correct procedure.

[0039] (4) The terminal device 10 determines whether or not the hands of the worker P are located within the working area R1 based on the skeletal coordinates C1 to C9 extracted from the image captured by the photographing device 20. Therefore, the extraction of the skeletal coordinates C1 to C9 and the detection of the hand positions to determine whether or not they are located within the working area R1 can be performed simultaneously.

[0040] (5) The terminal device 10 judges whether a task is correct or incorrect using a learning model learned by machine learning. Therefore, a task support system that judges whether a task is correct or incorrect can be easily constructed. (6) The time series data of the skeleton coordinates C1 to C9 when the hands are not located within the working area R1 is excluded in advance, and then the correctness of the work is judged using the learning model. This makes it easier to learn the learning model. Furthermore, since the correctness judgment is not performed when the hands of the worker P are not located within the working area R1, the frequency of correctness judgment using the learning model is reduced. As a result, the calculation load on the terminal device 10 is reduced.

[0041] <About correspondence> In the work support system of the above embodiment, the terminal device 10 corresponds to a processing unit that judges whether a work is correct or incorrect based on time-series data of the skeleton coordinates C1 to C9 of the worker P extracted from the image of the photographing device 20. In addition, in the case of the above work support system, the terminal device 10, the control circuit 22, and the indicator 33 constitute an instruction unit that instructs the worker P on the next work to be performed.

[0042] (Other embodiments) This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction.

[0043] In the above embodiment, it was determined whether or not the hands of the worker P are located within the working area R1 based on the skeletal coordinates C6, C7 of the left and right hands of the worker P extracted from the image of the camera device 20. The above determination may be made by directly detecting the positions of the hands from the image of the camera device 20 without using the skeletal coordinates C6, C7.

[0044] A learning model having a different configuration from that shown in Fig. 5 may be used as a learning model used to judge whether an operation is correct or incorrect based on the time-series data of the skeleton coordinates C1 to C9. Also, the correctness of an operation may be judged without using a learning model trained by machine learning.

[0045] The hand positions may be detected before the extraction of the skeletal coordinates C1 to C9. In this case, if it is determined that the hands are not located within the working area R1, the extraction of the skeletal coordinates C1 to C9 may not be performed.

[0046] The instruction to perform the next task may be given by a method other than lighting the indicator 33. For example, the instruction unit may be configured to instruct the next task using voice, text, video, or the like. In cases where it is assumed that the worker P has memorized the work procedures, the instruction unit may omit the instruction for the next work.

[0047] When the order of tasks does not matter, or when there is only a single task, the correctness of the tasks may be determined without determining whether the tasks are performed in a predetermined order.

[0048] The working area R1 and the non-working area R2 may be divided in a manner different from that shown in FIG. If it is possible to determine the type of work performed by worker P based on time-series data, the number and positions of skeleton coordinates extracted from the image may be changed. [Explanation of symbols]

[0049] 10 Terminal Equipment 101 Main storage 102 Auxiliary storage 103 CPU 104 Interface device 105 Display device 106 Input Devices 20 Imaging Equipment 21 Notification device 22 Control circuit 30 Workbench 31a~31d Bolt box 32A~32D Parts Box 33 Indicator B1 Bus B2 Communication Line P worker Double work

Claims

1. A work support system comprising: a photographing device that photographs a workplace where a worker performs work; and a processing unit that performs a judgment of whether the work is correct or incorrect based on time-series data of skeletal coordinates of the worker extracted from an image of the photographing device, The processing unit determines whether or not the worker's hand is located within a predetermined working area based on the image, and performs the correct / incorrect determination without using the time-series data of the skeleton coordinates when it is determined that the worker's hand is not located within the working area. Work support system.

2. There are a plurality of operations as the operations, The processing unit determines whether the plurality of tasks are performed in a predetermined order in the correctness determination. The work support system according to claim 1 .

3. There are a plurality of operations as the operations, The work support system includes an instruction unit that instructs the worker on the next work to be performed, The processing unit is configured to be able to determine whether each of the plurality of tasks is correct or not, and to perform the correctness determination for the task instructed by the instruction unit. The work support system according to claim 1 .

4. The work support system according to claim 1 , wherein the processing unit determines whether or not the worker's hands are positioned within the work area based on the skeletal coordinates.

5. The task support system according to claim 1 , wherein the determination of correctness is performed using a learning model trained by machine learning.

Images