Work support system

Abstract

To prevent a predetermined work from being determined to be completed even though the predetermined work is not yet completed.SOLUTION: A control device acquires time-series data of skeletal coordinates of a worker appearing in a first image (S13). The control device determines whether or not a distance DIS from a work position to the worker is equal to or greater than a distance determination value DISth (S17). The control device determines whether or not the worker has performed a predetermined completion action based on the time-series data of the skeletal coordinates (S19). When determining that the worker has performed the completion action (S19: YES) in a state in which it is determined that the distance DIS is equal to or greater than the distance determination value DISth (S17: YES), the control device determines that a predetermined work has been completed.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a work support system for supporting a worker in performing work at a predetermined work position. [Background technology]

[0002] Patent Document 1 discloses a system that assists a worker in assembling a workpiece that has been transported to a work position by a belt conveyor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-108186 A Summary of the Invention [Problem to be solved by the invention]

[0004] When the worker has completed assembling the workpiece, the belt conveyor removes the workpiece from the worker's work position and transports a new workpiece to the work position. One method for determining that the worker has completed assembling the workpiece is to have the worker operate a specific device such as a button. When the worker completes assembling the workpiece, he or she operates the specific device. This confirms that the assembly is complete, so the workpiece is removed from the work position and the next workpiece is transported to the work position.

[0005] However, there are cases where an operator mistakenly operates a particular device during the assembly of a workpiece, resulting in the assembly of the workpiece being determined to be complete when in fact it is not. [Means for solving the problem]

[0006] A work support system for solving the above problem is a work support system including an imaging device that images a worker performing a specified task at a work position, and a control device that acquires the image captured by the imaging device, in which the control device executes the following operations: acquires time series data of skeletal coordinates of the worker appearing in the image; determines whether the distance from the work position to the worker is equal to or greater than a distance judgment value; determines whether the worker has performed a specified completion action that indicates that the specified task has been completed, based on the time series data of skeletal coordinates; and determines that the specified task has been completed when it is determined that the worker has performed the completion action under a situation in which it has been determined that the distance from the work position to the worker is equal to or greater than the distance judgment value.

[0007] In addition, the control device may use a trained model that has been subjected to machine learning when determining whether or not the worker has performed the completion action. Effect of the Invention

[0008] The above-described work assistance system has an effect of preventing a predetermined work from being determined to be completed when the predetermined work has not yet been completed. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an outline of a configuration of a work support system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of how skeleton coordinates are set. [Diagram 3] FIG. 3 is a schematic diagram showing a state in which a worker has left the work position in the work support system of FIG. [Figure 4] FIG. 4 is a diagram illustrating an example of a first operation of the completion work. [Diagram 5] FIG. 5 is a diagram illustrating an example of a second operation of the completion work. [Figure 6]FIG. 6 is a flowchart showing a flow of processing executed by the control device provided in the work support system of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the work support system will be described with reference to FIGS. <Overall configuration of work support system> 1, the work support system 10 includes a conveying device 20 that conveys an article 100 in a predetermined conveying direction Z. The conveying device 20 conveys the article 100 in the predetermined conveying direction Z with the article 100 placed on a conveying surface 211 of a belt conveyor 21.

[0011] When the conveying device 20 has conveyed the product 100 to the working position 50, it stops the operation of the belt conveyor 21. In this state, the worker 200 performs a predetermined task at the working position 50. Examples of the predetermined task include assembling the product 100, painting the product 100, and inspecting the product. Then, when the task on the product 100 is completed, the conveying device 20 resumes the operation of the belt conveyor 21. As a result, the product 100 is conveyed out from the working position 50 in the conveying direction Z.

[0012] The work support system 10 includes a first imaging device 31, a second imaging device 32, and a control device 40. The first imaging device 31 captures an image of the worker 200. For example, the first imaging device 31 captures an image of the worker 200 from the front of the worker 200.

[0013] The second imaging device 32 captures an image of the transported item 100 that has been transported to the work position 50. For example, the second imaging device 32 is disposed above the belt conveyor 21. The second imaging device 32 captures an image of the work position 50 from above the belt conveyor 21.

[0014] Data of the first image IMG1, which is an image captured by the first imaging device 31, is transmitted to the control device 40. Data of the second image IMG2, which is an image captured by the second imaging device 32, is transmitted to the control device 40. In other words, the control device 40 can obtain data of the images IMG1, IMG2 captured by the multiple imaging devices 31, 32.

[0015] <Control device> The control device 40 includes a processing circuit 41. One example of the processing circuit 41 is an electronic control device. In this case, the processing circuit 41 includes a CPU 42 and a memory 43. The memory 43 stores a control program executed by the CPU 42. The CPU 42 performs overall control of the task support system 10 by executing the control program.

[0016] The control device 40 includes a learning device 46 in which a trained model 47 is stored. The trained model 47 is a model that has been subjected to machine learning. An example of the trained model 47 is a neural network. Details of the trained model 47 will be described later.

[0017] <Acquisition of worker's skeletal coordinates> 2, the control device 40 acquires skeletal coordinates of the worker 200 appearing in the first image IMG1. For example, the control device 40 analyzes the first image IMG1 to acquire a plurality of skeletal coordinates of the worker 200 as follows:

[0018] · Skeleton coordinates C1 of the head of worker 200 Skeletal coordinates C2 of the left shoulder joint of worker 200 Skeletal coordinates C3 of the right shoulder joint of worker 200 Skeletal coordinates C4 of the left elbow joint of worker 200 Skeletal coordinates C5 of the right elbow joint of worker 200 Skeleton coordinates C6 of the left hand of worker 200 Skeleton coordinates C7 of the right hand of worker 200 Skeletal coordinate C8 of the left hip joint of worker 200 Skeletal coordinate C9 of the right hip joint of worker 200 In this embodiment, the skeleton coordinates C1 to C9 are two-dimensional coordinates, but in cases where the first image IMG1 includes depth information of the subject, the skeleton coordinates C1 to C9 may be three-dimensional coordinates.

[0019] The control device 40 executes an acquisition process of a plurality of skeleton coordinates C1 to C9 at each predetermined control cycle. That is, the control device 40 can acquire time-series data of the plurality of skeleton coordinates C1 to C9 by repeatedly executing the acquisition process.

[0020] <Processing for determining the worker's position> 3, in the determination process, the control device 40 acquires, for example, a distance DIS from the work position 50 to the worker 200. If the first image IMG1 includes depth information of the subject, the control device 40 can acquire the distance DIS by analyzing the first image IMG1.

[0021] Next, the control device 40 determines whether the distance DIS is equal to or greater than the distance judgment value DISth. For example, a distance at which the worker 200 cannot touch the transported item 100 located at the work position 50 is set as the distance judgment value DISth. Therefore, if the distance DIS is equal to or greater than the distance judgment value DISth, it is considered that the worker 200 is not performing the above-mentioned predetermined process. On the other hand, if the distance DIS is less than the distance judgment value DISth, it is considered that the worker 200 may be performing the above-mentioned predetermined work.

[0022] <Process for determining whether or not a completion action has been performed> The control device 40 determines whether or not the worker 200 has performed a predetermined completion motion based on the time-series data of the multiple skeleton coordinates C1 to C9. The completion motion is a motion that indicates that a predetermined task has been completed.

[0023] The completion action includes a first action and a second action different from the first action. The first action and the second action are preferably actions that the worker 200 would not perform when working on the transported item 100.

[0024] An example of the first motion is shown in Fig. 4. In the example shown in Fig. 4, the first motion is a motion in which the worker 200 places both hands on top of his / her head. An example of the second motion is shown in Fig. 5. In the example shown in Fig. 5, the second motion is a motion in which the worker 200 raises both hands up.

[0025] Based on the time-series data of the multiple skeleton coordinates C1 to C9, the control device 40 checks whether the worker 200 has performed a first movement. The control device 40 also checks whether the worker 200 has performed a second movement. Then, when the control device 40 checks that the worker 200 has performed both the first movement and the second movement, it determines that the worker 200 has performed a completion movement.

[0026] In this embodiment, the control device 40 checks whether the worker 200 has performed a first movement based on the time series data of the multiple skeletal coordinates C1 to C9. When the control device 40 checks that the worker 200 has performed the first movement, the control device 40 checks whether the worker 200 has performed a second movement based on the time series data of the multiple skeletal coordinates C1 to C9. Then, when the control device 40 checks that the worker 200 has performed the second movement, it determines that the worker 200 has performed a completion movement.

[0027] The control device 40 can determine whether or not the worker 200 has performed the completion movement by using the trained model 47. In this case, a learning model that outputs an index value X indicating the likelihood that the worker 200 has performed the completion movement when time-series data of a plurality of skeleton coordinates C1 to C9 is input is adopted as the trained model 47.

[0028] The control device 40 inputs the time-series data of multiple skeleton coordinates C1 to C9 as input variables to the trained model 47. The control device 40 acquires an index value X, which is an output variable of the trained model 47 at this time. Then, the control device 40 judges whether the worker 200 has performed a completion movement based on the acquired index value X. For example, when the index value X is equal to or greater than a threshold, the control device 40 judges that the worker 200 has performed a completion movement. On the other hand, when the index value X is less than the threshold, the control device 40 judges that the worker 200 has not performed a completion movement.

[0029] <Process for determining whether the work is completed or not> The control device 40 determines whether the worker 200 has completed the predetermined task. Specifically, when the control device 40 determines that the worker 200 has performed a completion action under a condition in which the control device 40 has determined that the distance DIS is equal to or greater than the distance determination value DISth, the control device 40 determines that the worker 200 has completed the predetermined task. On the other hand, when at least one of the determination that the distance DIS is equal to or greater than the distance determination value DISth and the determination that the worker 200 has performed a completion action is not satisfied, the control device 40 determines that the worker 200 has not completed the predetermined task.

[0030] <Process flow when assisting workers with their work> The flow of processing executed by the control device 40 to assist the work performed by the worker 200 will be described with reference to FIG.

[0031] In step S11, the control device 40 starts monitoring the worker 200. That is, the control device 40 causes the first imaging device 31 and the second imaging device 32 to start imaging. In the next step S13, the control device 40 analyzes the first image IMG1 captured by the first imaging device 31 to obtain a plurality of skeleton coordinates C1 to C9 of the worker 200.

[0032] In the next step S15, the control device 40 acquires the distance DIS from the work position 50 to the worker 200. Then, in step S17, the control device 40 determines whether the distance DIS is equal to or greater than the distance determination value DISth. If the distance DIS is equal to or greater than the distance determination value DISth (S17: YES), the control device 40 shifts the process to step S19. On the other hand, if the distance DIS is less than the distance determination value DISth (S17: NO), the control device 40 shifts the process to step S13.

[0033] In step S19, the control device 40 determines whether or not the worker 200 has performed the completion movement based on the time-series data of the multiple skeleton coordinates C1 to C9. If the control device 40 determines that the worker 200 has performed the completion movement (S19: YES), the control device 40 shifts the process to step S21. On the other hand, if the control device 40 determines that the worker 200 has not performed the completion movement (S19: NO), the control device 40 shifts the process to step S13.

[0034] In step S21, the control device 40 determines whether or not there is an incomplete step in the predetermined work. The predetermined work may include multiple steps. If there is a step among the multiple steps that the worker 200 has not yet performed, there is a possibility that the predetermined work has not actually been completed. In this embodiment, the second imaging device 32 captures an image of the work position 50. Therefore, the control device 40 can determine whether or not there is an incomplete step by analyzing the second image IMG2. If the control device 40 determines that there is an incomplete step (S21: YES), the control device 40 shifts the process to step S27. On the other hand, if the control device 40 determines that there is no incomplete step (S21: NO), the control device 40 shifts the process to step S23.

[0035] In step S23, the control device 40 starts the conveying process. Specifically, the control device 40 operates the belt conveyor 21 to convey the transported product 100 from the work position 50 in a predetermined conveying direction Z, and then conveys a new transported product 100 to the work position 50.

[0036] In the next step S25, the control device 40 determines whether the conveyance is complete. For example, when the control device 40 analyzes the second image IMG2 and confirms that the new transported product 100 has been transported to the work position 50, the control device 40 determines that the conveyance is complete. When the control device 40 determines that the conveyance is not yet complete (S25: NO), the control device 40 continues the conveyance process. On the other hand, when the control device 40 determines that the conveyance is complete (S25: YES), the control device 40 ends the conveyance process. In other words, the control device 40 stops the operation of the belt conveyor 21. After that, the control device 40 temporarily ends the process flow shown in FIG. 6.

[0037] In step S27, the control device 40 notifies the worker 200 that there is an incomplete process. The notification method may be a method of displaying an image showing the incomplete process on a display screen, or a method of notifying the worker 200 of the incomplete process by voice. After that, the control device 40 temporarily ends the process flow shown in FIG.

[0038] <Actions and Effects of the Present Embodiment> (1) The control device 40 acquires time-series data of a plurality of skeletal coordinates C1-C9 of the worker 200 appearing in the first image IMG1 (S13). The control device 40 judges whether or not the distance DIS from the work position 50 to the worker 200 is equal to or greater than the distance judgment value DISth (S17). The control device 40 judges whether or not the worker 200 has performed a completion action based on the time-series data of the plurality of skeletal coordinates C1-C9 (S19). Then, in a situation in which the control device 40 judges that the distance DIS is equal to or greater than the distance judgment value DISth (S17: YES), if the control device 40 judges that the worker 200 has performed a completion action (S19: YES), it judges that the predetermined work has been completed.

[0039] According to the above configuration, when the predetermined task is completed, the worker 200 performs a completion action while away from the task position 50. Then, when the worker 200 performs a completion action while away from the task position 50 such that the distance DIS is equal to or greater than the distance judgment value DISth, the control device 40 determines that the predetermined task is completed. In other words, there is no need to provide a specific device such as a button for the worker 200 to operate when the predetermined task is finished.

[0040] Therefore, the work assistance system 10 can prevent the predetermined work from being determined to be completed when the predetermined work has not yet been completed. (2) The completion movement includes a first movement and a second movement different from the first movement. Then, the control device 40 determines that the worker 200 has performed the completion movement when it confirms, based on the time-series data of the multiple skeleton coordinates C1 to C9, that the worker 200 has performed both the first movement and the second movement.

[0041] Assume that the completion action includes only one of the first action and the second action. In this case, the worker 200 may accidentally perform the completion action. As a result, the control device 40 may determine that the predetermined task has been completed, even though the worker 200 has not actually completed the predetermined task.

[0042] In this embodiment, the completion action includes a plurality of actions. In this case, even if the worker 200 accidentally performs one of the plurality of actions, the possibility that the worker 200 will perform the remaining actions is very low.

[0043] In this regard, the control device 40 is configured to determine that the worker 200 has performed the completion action when it is confirmed that the worker 200 has performed all of the multiple actions. Therefore, the control device 40 can increase the accuracy of determining whether or not a predetermined task has been completed.

[0044] (3) When the control device 40 confirms that the worker 200 has performed the first motion and then the second motion, it determines that the worker 200 has performed the completion motion. In other words, unless the worker 200 performs the first motion and then the second motion in that order, the worker 200 cannot inform the control device 40 that the predetermined task has been completed. Therefore, the control device 40 can increase the accuracy of determining whether the predetermined task has been completed.

[0045] <Example of change> The above embodiment can be modified as follows: The above embodiment and the following modifications can be combined with each other to the extent that no technical contradiction occurs.

[0046] The trained model by machine learning does not have to be a neural network as long as it can output an index value indicating the likelihood that the worker 200 has performed the completion action when the time-series data of the skeleton coordinates is input. For example, a support vector machine may be adopted as the trained model.

[0047] It is not essential to use the trained model 47 when determining whether or not the worker 200 has performed the completion action. When determining whether or not the worker 200 has performed the completion action, the order of execution of the first action and the second action does not matter. In other words, even if the worker 200 performs the second action and then the first action, the control device 40 may determine that the worker 200 has performed the completion action.

[0048] The first operation may be different from the operation shown in FIG. 4 as long as it is different from the second operation. The second operation may be different from the operation shown in FIG. 5 as long as it is different from the first operation.

[0049] A completion action may contain more than two actions. A completed action may be an action that contains only one action. The processing circuit 41 is not limited to a circuit having a CPU and a ROM and executing software processing. In other words, the processing circuit 41 may have any one of the following configurations (a), (b), and (c).

[0050] (a) The processing circuitry 41 includes one or more processors that execute various processes according to a computer program. The processor includes a CPU and memory such as RAM and ROM. The memory stores program codes or instructions configured to cause the CPU to execute processes. The memory, i.e., computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.

[0051] (b) The processing circuit 41 includes one or more dedicated hardware circuits for executing various processes. Examples of the dedicated hardware circuits include application specific integrated circuits, i.e., ASIC and FPGA. ASIC is an abbreviation for "Application Specific Integrated Circuit" and "FPGA" is an abbreviation for "Field Programmable Gate Array."

[0052] (c) The processing circuitry 41 includes a processor that executes some of the various processes in accordance with a computer program, and a dedicated hardware circuit that executes the remaining processes among the various processes.

[0053] The term "at least one" used herein means "one or more" of the desired options. As an example, the term "at least one" used herein means "only one option" or "both of two options" if the number of options is two. As another example, the term "at least one" used herein means "only one option" or "any combination of two or more options" if the number of options is three or more. [Explanation of symbols]

[0054] 10...work support system, 20...transportation device, 31...first imaging device, 32...second transport device, 40...control device, 46...learning device, 47...trained model, 50...work position, 100...transported item, 200...worker.

Claims

1. an imaging device that images a worker performing a predetermined task at a work position; A control device for acquiring an image captured by the imaging device, The control device includes: acquiring time series data of skeletal coordinates of the worker shown in the image; determining whether a distance from the work position to the worker is equal to or greater than a distance determination value; determining whether or not the worker has performed a predetermined completion motion, which is a motion indicating that the predetermined task has been completed, based on the time-series data of the skeleton coordinates; and determining that the predetermined task is completed when it is determined that the worker has performed the completion action under a condition in which it is determined that the distance from the task position to the worker is equal to or greater than the distance determination value. Work support system.

2. The completion action includes a first action and a second action different from the first action, The control device determines that the worker has performed the completion movement when it has confirmed, based on the time-series data of the skeletal coordinates, that the worker has performed both the first movement and the second movement. The work support system according to claim 1 .

3. When the control device confirms that the worker has performed the first movement and then the second movement based on the time-series data of the skeletal coordinates, it determines that the worker has performed the completion movement. The work support system according to claim 2 .

4. the predetermined work is a work to be performed on a transported item that has been transported to the work position, a conveying device that conveys the conveyed product in a predetermined conveying direction, When the control device determines that the predetermined operation for the transported item is completed, the control device controls the transport device to remove the transported item from the operation position and transport a new transported item to the operation position. The work support system according to any one of claims 1 to 3.

5. The control device has a trained model that has been subjected to machine learning, the trained model is configured to output an index value indicating a high probability that the worker has performed the completion motion when the time-series data of the skeletal coordinates is input, The control device determines whether or not the worker has performed the completion motion based on an index value output from the trained model when the time-series data of the skeletal coordinates is input to the trained model. The work support system according to claim 1 .

Images