Work determination device, work determination method, and storage medium

Abstract

The work determination device has: an image acquisition unit that acquires an image of a work in which a worker holds a component to mount the component on a semi-finished product; a component detection unit that detects the component from the image acquired by the image acquisition unit; a holding determination unit that determines whether the worker holds the component based on a detection result of the component detection unit, and determines whether the component held by the worker is a component that should be mounted on the semi-finished product in the work; and a notification control unit that controls to perform a prescribed notification when the holding determination unit determines that the component held by the worker is not the component that should be mounted on the semi-finished product in the work.

Description

Technical Field

[0001] This invention relates to a technique for determining whether a task has been performed correctly. Background Technology

[0002] Errors or omissions in product manufacturing operations that affect product quality are considered a problem. Patent Document 1 discloses a technique for determining whether an operator's physical activities (e.g., hand and foot movements) are consistent with prescribed activities based on captured images (videos), and for determining which operation the operator's actions belong to based on the determination result.

[0003] Prior technology documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-003781 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, workers' physical conditions vary, and they may not always make decisions during a particular task. Therefore, even using the technology described in Patent Document 1, it is not possible to determine with high precision whether the task is being performed correctly, and errors or omissions may sometimes occur.

[0008] The present invention is proposed in view of the above-mentioned actual situation, and its purpose is to provide a technology that can accurately determine whether the operation is performed correctly in order to prevent errors or omissions in the operation.

[0009] Methods for solving problems

[0010] To achieve the above objectives, the present invention employs the following structure.

[0011] The first aspect of this invention provides a work determination device, characterized by comprising: an image acquisition unit that acquires an image of a worker holding a component to install it onto a semi-finished product; a component detection unit that detects the image acquired by the image acquisition unit; a holding determination unit that determines, based on the detection result of the component detection unit, whether the worker is holding a component and whether the component held by the worker is the component that should be installed onto the semi-finished product in the work; and a notification control unit that performs control to issue a prescribed notification if the holding determination unit determines that the component held by the worker is not the component that should be installed onto the semi-finished product in the work. The prescribed notification may be, for example, a warning (reminder) regarding incorrect work performance, implemented through the display of graphics (icons, messages), sound output, or the illumination of lights. Prescribed notifications are issued to workers, managers overseeing the work (work supervisors), etc.

[0012] The components (types, shapes, sizes, etc.) that should be installed on the semi-finished product during the operation are always fixed and do not depend on the operator. Therefore, based on the above structure, by determining whether the component held by the operator is the one that should be installed on the semi-finished product during the operation, it is possible to determine with high accuracy whether the operation is being performed correctly. Furthermore, by issuing a pre-defined notification when it is determined that the component held by the operator is not the one that should be installed on the semi-finished product during the operation, errors and omissions in the operation can be prevented.

[0013] In order to install the component being held onto the semi-finished product, the operator often moves the component significantly. Therefore, if the movement distance of the component detected by the component detection unit is greater than or equal to a first threshold, the holding determination unit can determine that the operator is holding the component.

[0014] Even if a component is not held by the operator, it will move due to factors such as the shaking of the worktable. Generally, a held component moves more than an unheld component. Therefore, when multiple components are detected by the component detection unit, the holding determination unit can select the component with the largest movement distance from among the multiple components, and if the movement distance of the selected component is greater than or equal to the first threshold, it can determine that the operator is holding that component. In this way, even when multiple components are detected, the component held by the operator can be detected (determined) with high accuracy.

[0015] It may also include an information acquisition unit, which acquires information indicating the correspondence between the above-mentioned operation and the component that should be installed in the above-mentioned semi-finished product in the operation. The holding determination unit determines whether the component held by the operator is the component that should be installed in the above-mentioned semi-finished product in the above-mentioned operation based on the detection result of the component detection unit and the information acquired by the information acquisition unit.

[0016] It may also include a work completion determination unit. If the holding determination unit determines that the component held by the operator is a component that should be installed on the semi-finished product during the work, this work completion determination unit determines whether the work completion conditions are met based on the detection results of the component detection unit. In this way, it can not only determine whether the work was performed correctly, but also whether the work was completed correctly.

[0017] Generally, when components are installed onto a semi-finished product, the distance between them decreases. Therefore, the aforementioned conditions may also include that the distance between the component held by the operator and the semi-finished product is below a second threshold. Furthermore, generally, the relative position and orientation of the component relative to the semi-finished product are largely predetermined when components are installed onto the semi-finished product. Therefore, the aforementioned conditions may also include that the relative position of the component relative to the semi-finished product is a predetermined position, and the aforementioned conditions may also include that the relative orientation of the component relative to the semi-finished product is a predetermined orientation.

[0018] The image acquisition unit can also acquire images showing the process of placing the semi-finished product in a designated location after the operator has completed one or more tasks. The component detection unit further detects the semi-finished product from the images acquired by the image acquisition unit. The task completion determination unit determines whether the semi-finished product has been placed in the designated location based on the detection results of the component detection unit. The notification control unit further controls the process so that if the semi-finished product has been placed in the designated location before the task completion determination unit determines that all of the one or more tasks have been completed, the specified notification is issued. This more reliably prevents the omission of tasks.

[0019] It may also include a hand detection unit that detects the operator's hand from the image acquired by the image acquisition unit. If the gripping determination unit determines that the part being gripped by the operator is a component to be installed on the semi-finished product during the operation, and the part being gripped is not detected by the component detection unit, the operation completion determination unit uses the position of the hand detected by the hand detection unit as the position of the part being gripped by the operator. In this way, even if the part being gripped by the operator is hidden by the operator's hand and not detected, it is possible to determine whether the operation is completed.

[0020] A second aspect of the present invention provides a job determination method, characterized by comprising: an image acquisition step, which acquires an image of a worker holding a component to install the component onto a semi-finished product; a component detection step, which detects the component from the image acquired in the image acquisition step; a holding determination step, which determines whether the worker is holding a component based on the detection result in the component detection step, and determines whether the component held by the worker is a component that should be installed onto the semi-finished product in the job; and a notification control step, which performs control to issue a predetermined notification if, in the holding determination step, it is determined that the component held by the worker is not a component that should be installed onto the semi-finished product in the job.

[0021] Furthermore, the present invention can be implemented as a job determination system having at least one of the above-described structures or functions. Additionally, the present invention can also be implemented as a job determination method or a control method for a job determination system including at least a portion of the above-described processes, a program for causing a computer to execute these methods, or a computer-readable recording medium that non-temporarily records such a program. The above-described structures and processes can be combined with each other to constitute the present invention, provided that no technical contradiction arises.

[0022] Invention Effects

[0023] According to the present invention, the accuracy of determining whether the operation is performed correctly can prevent errors and omissions in the operation. Attached Figure Description

[0024] Figure 1 This is a block diagram illustrating a structural example of the job determination device of the present invention.

[0025] Figure 2 (A) is a schematic diagram showing a general configuration example of the job determination system of the first embodiment. Figure 2 (B) is a block diagram showing a structural example of the PC (job determination device) of the first embodiment.

[0026] Figure 3 This is an example diagram showing an operational image of the first embodiment.

[0027] Figure 4 This is a diagram showing an example of the operation information of the first embodiment.

[0028] Figure 5 This is a flowchart illustrating an example of the processing flow of the first embodiment.

[0029] Figure 6 This is a block diagram illustrating a structural example of the PC (job determination device) in the second embodiment.

[0030] Figure 7 This is a flowchart illustrating an example of the processing flow in the second embodiment.

[0031] Figure 8 This is a flowchart illustrating an example of the processing flow in the third embodiment.

[0032] Figure 9 (A)~9(E) are diagrams showing an example of the state of parts and semi-finished products. Detailed Implementation

[0033] <Application Examples>

[0034] The application examples of the present invention will be described. Workers have diverse physical characteristics, and in a given task, a worker may not always make the correct action. Therefore, conventional techniques for determining whether a worker's body movements (e.g., hand and foot movements) are consistent with prescribed movements cannot accurately determine whether the task is being performed correctly, sometimes resulting in errors or omissions.

[0035] Figure 1 This is a block diagram illustrating a structural example of the work determination device 100 of the present invention. The work determination device 100 includes: an image acquisition unit 101, a component detection unit 102, a gripping determination unit 103, and a notification control unit 104. The image acquisition unit 101 acquires an image of a worker gripping a component to install it onto a semi-finished product. The component detection unit 102 detects the image acquired by the image acquisition unit 101. The gripping determination unit 103 determines whether the worker is gripping a component based on the detection result of the component detection unit 102, and determines whether the component gripped by the worker is a component that should be installed onto the semi-finished product during the work. If the gripping determination unit 103 determines that the component gripped by the worker is not a component that should be installed onto the semi-finished product during the work, the notification control unit 104 performs control to issue a prescribed notification. Image acquisition unit 101 is an example of an image acquisition unit of the present invention, component detection unit 102 is an example of a component detection unit of the present invention, grip determination unit 103 is an example of a grip determination unit of the present invention, and notification control unit 104 is an example of a notification control unit of the present invention. Here, the specified notification is, for example, a warning (reminder) indicating that work has not been performed correctly, and is implemented through the display of graphics (icon messages), sound output, lighting of lights, etc. Specified notifications are given to operators, managers (work supervisors), etc.

[0036] The components (types, shapes, sizes, etc.) that should be installed on the semi-finished product during the operation are always fixed and do not depend on the operator. Therefore, based on the above structure, by determining whether the component held by the operator is the one that should be installed on the semi-finished product during the operation, it is possible to determine with high accuracy whether the operation is being performed correctly. Furthermore, by issuing a pre-defined notification when it is determined that the component held by the operator is not the one that should be installed on the semi-finished product during the operation, errors and omissions in the operation can be prevented.

[0037] <First Embodiment>

[0038] The first embodiment of the present invention will be described.

[0039] Figure 2(A) is a schematic diagram showing a general configuration example of the job determination system according to the first embodiment. The job determination system of the first embodiment includes a camera 10, a PC 200 (personal computer; job determination device), and a notification unit 20. The camera 10 and the PC 200 are interconnected via wired or wireless means. Similarly, the notification unit 20 and the PC 200 are also interconnected via wired or wireless means.

[0040] Camera 10 takes pictures using an image sensor and outputs the captured images to PC 200. In the first embodiment, camera 10 outputs a work image to PC 200, showing the operator holding a component and installing it onto a semi-finished product. PC 200 determines whether the work is being performed correctly based on the work image, and if it determines that the work is not being performed correctly, it outputs a control signal to notification unit 20 to issue a specified notification. Notification unit 20 includes at least one of a monitor, a speaker, and a light. Based on the control signal from PC 200, notification unit 20 issues warnings (reminders) such as displaying graphics (icons, messages) on the monitor, outputting sound from the speaker, or turning on the light, indicating that the work is not being performed correctly.

[0041] Furthermore, in the first embodiment, the camera 10, PC 200, and notification unit 20 are independent devices, but they may not be. For example, at least two of the camera 10, PC 200, and notification unit 20 may be included in one device. Specifically, PC 200 may include at least one of the camera 10 and notification unit 20. Additionally, there is no particular limitation on the location where PC 200 and notification unit 20 are installed. For example, PC 200 and notification unit 20 may be installed in the same room as camera 10, but they may not be. When notifying the operator, notification unit 20 can be installed in the same room as camera 10 or the operator, or the operator's mobile terminal (smartphone, tablet, etc.) can be used as notification unit 20. When notifying the supervisor, notification unit 20 can be installed in the same room as the supervisor, or the supervisor's mobile terminal can be used as notification unit 20. PC 200 may or may not be a computer in the cloud.

[0042] Figure 2 (B) is a block diagram showing a structural example of PC200. PC200 includes: an input unit 210, a storage unit 220, a control unit 230, and an output unit 240.

[0043] The input unit 210 acquires work images from the camera 10 and outputs them to the control unit 230. In the first embodiment, the camera 10 captures moving images, and the input unit 210 sequentially acquires one frame of the moving image from the camera 10 and outputs it to the control unit 230. The image acquisition cycle of the input unit 210 is not particularly limited, but in the first embodiment, the input unit 210 acquires work images (one frame of animation) synchronously with the camera 10's capture. In other words, the input unit 210 performs the process of acquiring work images from the camera 10 and outputting them to the control unit 230 at the frame rate captured by the camera 10 (e.g., 30fps). Alternatively, the camera 10 may sequentially capture still images; in this case, the input unit 210 sequentially acquires still images from the camera 10 and outputs them to the control unit 230. The input unit 210 is an example of the image acquisition unit of the present invention.

[0044] Figure 3 This is an example diagram showing an operational image of the first embodiment. Figure 3 The work image 300 shows the current workbench 301, component container (case) 302, semi-finished product 303, and operator 304. Multiple components 302a to 302i are separately stored in the component container 302. This allows for easy and high-precision detection of components and calculation of component movement distances. Furthermore, in the first embodiment, the work image may also be an image obtained by capturing a process in which the operator performs one or more tasks and, after completing all of those tasks, places the semi-finished product in a designated location. Moreover, the workbench 301 is marked with a tape or similar marking to indicate the designated location 305 for placing the semi-finished product 303 after completing all processes. This allows for easy and high-precision detection of the next process placement location 305 and determination of whether the semi-finished product 303 has been placed in the next process placement location 305.

[0045] Furthermore, there is no particular limitation on the number of parts placed on the workbench 301, and the parts may not be stored in the part container 302. Parts can also be stacked, but it is preferable to separate them according to the work sequence to facilitate easy and high-precision inspection of parts, calculation of part movement distances, etc. Additionally, there are no particular limitations on the shooting range of the work image or the objects reflected in the work image. It is sufficient to capture the appearance of the work or process; for example, the part container 302 (the parts placed on the workbench 301) may not be reflected in the work image.

[0046] Storage unit 220 stores programs executed by control unit 230, various data used by control unit 230, etc. For example, storage unit 220 is an auxiliary storage device such as a hard disk drive or solid-state drive. In the first embodiment, information indicating the correspondence between a task and a component to be installed in the semi-finished product during that task (task information) is pre-stored in storage unit 220. By using the task information, the current task, the next task, and the components to be used in the task can be easily grasped. Furthermore, PC 200 may have an information acquisition unit for acquiring task information, and the task information can be stored on an external device of PC 200. The information acquisition unit is an example of the information acquisition unit of the present invention.

[0047] Figure 4 This is a diagram showing an example of the operation information of the first embodiment. Figure 4 The work information 400 shows the components to be installed on the semi-finished product in the order of the work processes. Specifically, it shows that processes 1 to 3 are performed in sequence during product manufacturing. In process 1, operations 1-1 to 1-3 are performed in sequence; in process 2, operations 2-1 to 2-3 are performed in sequence; and in process 3, operations 3-1 to 3-3 are performed in sequence. Furthermore, it shows that component 302a is used in operation 1-1, component 302b is used in operation 1-2, and component 302c is used in operation 1-3. Similarly, it shows that component 302d is used in operation 2-1, component 302e is used in operation 2-2, component 302f is used in operation 2-3, component 302g is used in operation 3-1, component 302h is used in operation 3-2, and component 302i is used in operation 3-3.

[0048] The control unit 230 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and performs control of each component and various information processing. In the first embodiment, the control unit 230 acquires a job image from the input unit 210 and job information from the storage unit 220, and determines whether the current job is being performed correctly based on the job image and job information. Furthermore, if the control unit 230 determines that the current job is not being performed correctly, it outputs a control signal for issuing a specified notification to the output unit 240.

[0049] The output unit 240 outputs the control signal output from the control unit 230 to the notification unit 20.

[0050] The control unit 230 will be described in more detail. The control unit 230 includes: a component detection unit 231, a grip determination unit 232, a work completion determination unit 233, and a notification control unit 234.

[0051] The component detection unit 231 acquires a work image from the input unit 210, detects components from the acquired work image, and outputs the detection results (component type, position, orientation, etc.) to the holding determination unit 232 and the work completion determination unit 233. Furthermore, the component detection unit 231 detects semi-finished products from the acquired work image and outputs the detection results (semi-finished product position, orientation, etc.) to the work completion determination unit 233. The component detection unit 231 is an example of the component detection unit of the present invention.

[0052] Furthermore, the component detection and semi-finished product detection performed by the component detection unit 231 can use any algorithm. For example, components and semi-finished products can be detected using existing object detection methods. Specifically, a detector (recognizer) that combines image features such as HoG and Haar-like can be used to detect facial expressions. Alternatively, a learned model generated through existing machine learning can be used to detect components and semi-finished products. Specifically, a learned model generated through deep learning (e.g., R-CNN, Fast R-CNN, YOLO, SSD, etc.) can also be used to detect components and semi-finished products.

[0053] The gripping determination unit 232 determines whether the operator is gripping a component based on the detection results (component detection results) of the component detection unit 231, and determines whether the component gripped by the operator is a component that should be installed on the semi-finished product in the current operation. Specifically, the gripping determination unit 232 obtains operation information from the storage unit 220, and determines whether the component gripped by the operator is a component that should be installed on the semi-finished product in the current operation based on the component detection results and the operation information. The operation information is used to determine the components that should be installed on the semi-finished product in the current operation. Furthermore, if the gripping determination unit 232 determines that the component gripped by the operator is a component that should be installed on the semi-finished product in the current operation, it notifies the operation completion determination unit 233 of this situation. Furthermore, if the gripping determination unit 232 determines that the component gripped by the operator is not a component that should be installed on the semi-finished product in the current operation, it notifies the notification control unit 234 of this situation. The gripping determination unit 232 is an example of the gripping determination unit of the present invention. In the first embodiment, since the gripping determination unit 232 obtains the operation information from the storage unit 220, it can be said that the gripping determination unit 232 also serves as the information acquisition unit of the present invention.

[0054] If the holding determination unit 232 determines that the component held by the operator is a component that should be installed on the semi-finished product in the current operation, the operation completion determination unit 233 determines whether the prescribed conditions for completing the current operation are met. This determination is based on the detection results of the component detection unit 231 (the detection results of the component and the semi-finished product). Furthermore, the operation completion determination unit 233 determines whether the semi-finished product has been placed in the next process placement location based on the detection results of the component detection unit 231 (the detection results of the semi-finished product). For example, if the component detection unit 231 can detect the next process placement location (the mark of the next process placement location) from the operation image, and the operation completion determination unit 233 detects a semi-finished product in the detected next process placement location, it can determine that a semi-finished product has been placed in the next process placement location. Furthermore, if the operation completion determination unit 233 has prior knowledge of the next process placement location, it is not necessary for the component detection unit 231 to detect the next process placement location. In the first embodiment, the operation completion determination unit 233 obtains operation information from the storage unit 220 and, based on the obtained operation information, understands each process and each operation. Based on this, by performing the above two determinations, it is possible to determine whether the semi-finished product has been placed in the next process placement location before all operations of the current process are completed. Furthermore, if the work completion determination unit 233 determines that the semi-finished product has been placed in the next process placement location before all operations of the current process are completed, it notifies the notification control unit 234 of this situation. The work completion determination unit 233 is an example of the work completion determination unit of the present invention. In the first embodiment, since the work completion determination unit 233 obtains work information from the storage unit 220, it can be said that the work completion determination unit 233 also serves as the information acquisition unit of the present invention.

[0055] The notification control unit 234 outputs a control signal for issuing a prescribed notification to the output unit 240 based on notifications from the gripping determination unit 232 or the work completion determination unit 233. In other words, if the gripping determination unit 232 determines that the part being gripped by the operator is not a part that should be installed on the semi-finished product in the current operation, the notification control unit 234 controls the issuance of the prescribed notification. Furthermore, if the work completion determination unit 233 determines that the semi-finished product should be placed in the next process placement location before all operations in the current process are completed, the notification control unit 234 controls the issuance of the prescribed notification. The notification control unit 234 is an example of a notification control unit of the present invention.

[0056] Figure 5 This is a flowchart illustrating an example of the processing flow in the first embodiment. PC200 executes repeatedly. Figure 5 The processing flow. Figure 5 The processing cycle is not specifically limited, and it is repeated at the frame rate (e.g., 30fps) captured by camera 10. Figure 5The processing flow.

[0057] exist Figure 5 In the processing flow, a current job flag is used to determine the current task (and process). The current job flag corresponds to any task indicated by the task information. For example, current job flag = 1 corresponds to task 1-1 of process 1, current job flag = 2 corresponds to task 1-2 of process 1, and current job flag = 3 corresponds to task 1-3 of process 1. Current job flag = 4 corresponds to task 2-1 of process 2, current job flag = 5 corresponds to task 2-2 of process 2, and current job flag = 6 corresponds to task 2-3 of process 2. Current job flag = 7 corresponds to task 3-1 of process 3, current job flag = 8 corresponds to task 3-2 of process 3, and current job flag = 9 corresponds to task 3-3 of process 3. Initially, the current job flag is set to 1.

[0058] First, the input unit 210 acquires the working image from the camera 10 (step S500).

[0059] Next, the component inspection unit 231 acquires a work image from the input unit 210 and inspects components and semi-finished products from the acquired work image (step S501). Figure 3 In the case of working image 300, components 302a~302i and semi-finished product 303 are inspected.

[0060] Next, the work completion determination unit 233 determines whether the semi-finished product has been placed in the next process placement location based on the detection result (semi-finished product detection result) of step S501 (step S502). If it is determined that the semi-finished product has been placed in the next process placement location (step S502: Yes), the process proceeds to step S503; if it is determined that the semi-finished product has not been placed in the next process placement location (step S502: No), the process proceeds to step S505.

[0061] In step S503, the job completion determination unit 233 determines whether all operations of the current process are completed. If it is determined that all operations of the current process are completed (step S503: Yes) Figure 5 The processing flow ends. If it is determined that the current process is incomplete (step S503: no), the processing proceeds to step S504.

[0062] exist Figure 5 In the processing flow, each time a task is determined to be completed, the current task flag is updated (incremented by 1), thereby updating the task currently being processed to the next task. Therefore, the task completion determination unit 233 can determine whether all tasks in the current process are completed based on the current task flag and task information.

[0063] For example, if the current operation flag is 1-3, then the current process is process 1, and it can be determined that at least one of operations 1-1 to 1-3 in process 1 is incomplete. If the semi-finished product is placed in the next process placement location for the first time after the current operation flag becomes 4, then the current process is process 1, and it can be determined that all operations 1-1 to 1-3 in process 1 are completed. If, even if the current operation flag is 4, the semi-finished product was placed in the next process placement location while the current operation flag is 4, then the current process is process 2, and it can be determined that operation 2-1 in process 2 is incomplete.

[0064] Similarly, if the current operation flag is 5 or 6, the current process is process 2, and it can be determined that at least one of operations 2-2 and 2-3 of process 2 is incomplete. If the semi-finished product is placed in the next process placement location for the first time after the current operation flag becomes 7, the current process is process 2, and it can be determined that all operations 2-1 to 2-3 of process 2 are completed. If, even if the current operation flag is 7, the semi-finished product is placed in the next process placement location while the current operation flag is 7, the current process is process 3, and it can be determined that operation 3-1 of process 3 is incomplete. If the current operation flag is 8 or 9, the current process is process 3, and it can be determined that at least one of operations 3-2 and 3-3 of process 3 is incomplete. If the current operation flag is 10, the current process is process 3, and it can be determined that all operations 3-1 to 3-3 of process 3 are completed.

[0065] In step S504, the notification control unit 234 outputs a control signal for making a specified notification to the output unit 240. As a result, the output unit 240 sends a control signal to the notification unit 20, and the specified notification is made by the notification unit 20.

[0066] In step S505, the work completion determination unit 233 determines whether the gripping state setting (the setting of the gripping state of the part to be installed on the semi-finished product in the current work) is valid. If the gripping state setting is valid (step S505: Yes), step S511 initiates the process; if the gripping state setting is valid (step S505: No), step S506 initiates the process.

[0067] In step S506, the gripping determination unit 232 calculates the movement distance D1 of the component detected in step S501. If multiple components are detected in step S501, the movement distance D1 of each of those components is calculated. The method for calculating the movement distance D1 is not particularly limited. For example, the distance between the component's detection position at the start of the current operation and the component's current detection position can be calculated as the movement distance D1. Alternatively, the distance between the component's detection position a predetermined time ago and the component's current detection position can be calculated; in other words, the movement distance of the component during the predetermined period can be used as the movement distance D1.

[0068] In step S507, if multiple components are detected in step S501, the gripping determination unit 232 selects the component with the largest movement distance D1 calculated in step S506 from among the multiple components. If only one component is detected in step S501, that component is selected. Even if a component is not gripped by the operator, it will move due to the shaking of the worktable, etc., but generally, gripped components move more than ungripped components. Therefore, through the processing in step S507, it is possible to select the component from among the multiple components that is more likely to be gripped by the operator.

[0069] Next, the gripping determination unit 232 determines whether the movement distance D1 of the component selected in step S507 is greater than or equal to the threshold Th1 (step S508). The operator often moves the component significantly in order to install it onto the semi-finished product. Therefore, the process in step S508 can determine whether the operator is gripping the component. If it is determined that the movement distance D1 of the selected component is greater than or equal to the threshold Th1 (step S508: Yes), in other words, if it is determined that the operator is gripping the component, the process proceeds to step S509. If it is determined that the movement distance D1 of the selected component is less than the threshold Th1 (step S508: No), in other words, if it is determined that the operator is not gripping the component... Figure 5 The processing flow has ended.

[0070] In step S509, the holding determination unit 232 determines whether the component held by the operator (the component selected in step S507) is the correct component (the component that should be installed on the semi-finished product in the current operation). If the component held by the operator is determined to be the correct component (step S509: Yes), the process proceeds to step S510. If the component held by the operator is determined to be the incorrect component (step S509: No), the process proceeds to step S504, and a prescribed notification is issued. The correct component is determined based on the current operation flag and operation information. Specifically, the holding determination unit 232 determines the component associated with the operation corresponding to the current operation flag in the operation information as the correct component. If the current operation flag is 1, then the component 302a associated with operation 1-1 is determined to be the correct component.

[0071] In step S510, the grip determination unit 232 enables the grip state setting.

[0072] In step S511, the work completion determination unit 233 calculates the distance D2 between the part held by the operator and the semi-finished product. The method for calculating distance D2 is not particularly limited, but for example, it can be calculated as the distance between the detection position (center position, etc.) of the part and the detection position of the semi-finished product. Alternatively, it can be calculated as the IoU (the ratio of the intersection of the detection rectangle of the part and the detection rectangle of the semi-finished product to the union of the detection rectangles of the part and the semi-finished product (size ratio)). It can also be calculated as the ratio of the intersection of the detection rectangle of the part and the detection rectangle of the semi-finished product to the detection rectangle of one side of the semi-finished product (size ratio). If the semi-finished product is fixed (its position and orientation have not changed), then the semi-finished product can be omitted from detection, and distance D2 can be calculated based on the position of the part held by the operator.

[0073] Next, the job completion determination unit 233 determines whether the distance D2 calculated in step S511 is below the threshold Th2 (step S512). Generally, when components are installed on a semi-finished product, the distance between them decreases. Therefore, through the processing in step S512, it is possible to determine whether the current job is completed. If it is determined that the distance D2 is below the threshold Th2 (step S512: Yes), in other words, if it is determined that the current job is completed, the process proceeds to step S513. If it is determined that the distance D2 is greater than the threshold Th2 (step S512: No), in other words, if it is determined that the current job is not completed, Figure 5 The processing flow ends. The distance D2 is the threshold Th2. Below is an example of the conditions specified above.

[0074] In step S513, the job completion determination unit 233 updates the current job flag (1 increments by 1) to update the job being processed as the next job.

[0075] Next, the work completion determination unit 233 invalidates the holding state setting (step S514).

[0076] The components (types, shapes, sizes, etc.) that should be installed on the semi-finished product during the operation are always fixed and do not depend on the operator. Therefore, according to the first embodiment, by determining whether the component held by the operator is a component that should be installed on the semi-finished product during the operation, it is possible to determine with high accuracy whether the operation is being performed correctly. Moreover, by issuing a pre-defined notification when it is determined that the component held by the operator is not a component that should be installed on the semi-finished product during the operation, errors and omissions in the operation can be prevented.

[0077] Furthermore, by using the work completion determination unit 233, it is possible not only to determine whether the work was performed correctly, but also to determine whether the work was completed correctly. Moreover, if it is determined that the semi-finished product has been placed in the designated location before all work in the process is completed, a notification is also given, which can more reliably prevent the omission of work.

[0078] <Second Implementation>

[0079] The second embodiment of the present invention will be described.

[0080] The structure of the job determination system in the second embodiment is the same as that in the first embodiment. Figure 6 This is a block diagram illustrating a structural example of the PC200 according to the second embodiment. The PC200 of the second embodiment also includes an input unit 210, a storage unit 220, a control unit 230, and an output unit 240, similar to the first embodiment. Furthermore, the control unit 230 of the second embodiment also includes a component detection unit 231, a grip determination unit 232, a work completion determination unit 233, and a notification control unit 234, similar to the first embodiment. The control unit 230 of the second embodiment also includes a hand detection unit 235. In the second embodiment, processes other than those described later are the same as in the first embodiment.

[0081] The hand detection unit 235 acquires a work image from the input unit 210, detects the operator's hand from the acquired work image, and outputs the hand detection results (hand position, etc.) to the work completion determination unit 233. The hand detection unit 235 is an example of the hand detection unit of the present invention.

[0082] Figure 7 This is a flowchart illustrating an example of the processing flow in the second embodiment. PC200 executes this process repeatedly. Figure 7 The processing flow. Figure 7The processing cycle is not specifically limited, but it is repeated at the frame rate (e.g., 30fps) captured by camera 10. Figure 7 The processing flow.

[0083] exist Figure 7 In China, Figure 5 (First Embodiment) The same processing is added to... Figure 5 Same reference numerals. In Figure 7 In the flowchart, if it is determined in step S505 that the holding state setting is valid (step S505: yes), the process proceeds to step S701 instead of step S511.

[0084] In step S701, the work completion determination unit 233 determines whether the component held by the operator has been detected by the component detection unit 231 (whether the component held by the operator was detected in step S501). If it is determined that the component held by the operator has been detected by the component detection unit 231 (step S701: Yes), the process proceeds to step S511. If it is determined that the component held by the operator has not been detected by the component detection unit 231 (step S701: No), the process proceeds to step S702.

[0085] In step S702, the hand detection unit 235 acquires a work image from the input unit 210 and detects the operator's hand from the acquired work image. Next, the work completion determination unit 233 sets the position of the hand detected in step S702 as the position of the component held by the operator (step S703).

[0086] As described above, according to the second embodiment, when no part held by the operator is detected, the position of the detected hand is used as the position of the part held by the operator. Therefore, even if the part held by the operator is hidden by the operator's hand and is not detected, it is possible to determine whether the work has been completed.

[0087] <Third Implementation>

[0088] The third embodiment of the present invention will be described. The structure of the job determination system and the structure of the PC200 in the third embodiment are the same as those in the first embodiment.

[0089] Figure 8 This is a flowchart illustrating an example of the processing flow in the third embodiment. PC200 executes repeatedly. Figure 8 The processing flow. Figure 8 The processing cycle is not specifically limited, but it is repeated at the frame rate (e.g., 30fps) captured by camera 10. Figure 8 The processing flow.

[0090] exist Figure 8 In the middle, to and Figure 5 (First Embodiment) The same processing is added to... Figure 5 Same reference numerals. In Figure 8 In the flowchart, if it is determined in step S512 that the distance D2 is below the threshold Th2 (step S512: yes), the process proceeds to step S801 instead of step S513.

[0091] In step S801, the work completion determination unit 233 determines whether the relative position of the holding component to the semi-finished product (the component held by the operator) is a predetermined position. Generally, the relative position of the component to the semi-finished product is mostly predetermined when the component is installed on the semi-finished product. Therefore, by further processing in step S801, it is possible to determine with higher accuracy whether the current work is completed. If it is determined that the relative position of the holding component to the semi-finished product is a predetermined position (step S801: Yes), in other words, if it is determined that the current work is completed, the process proceeds to step S513. If it is determined that the relative position of the holding component to the semi-finished product is not a predetermined position (step S801: No), in other words, if it is determined that the current work is not completed, Figure 8 The processing flow is complete. The specified position of the holding component relative to the semi-finished product is an example of the specified conditions described in the first embodiment.

[0092] Furthermore, the relative orientation of the component to the semi-finished product is generally predetermined when installing components on a semi-finished product. Therefore, the work completion determination unit 233 can also determine whether the relative orientation of the holding component to the semi-finished product is a predetermined orientation (third determination). The fact that the relative orientation of the holding component to the semi-finished product is a predetermined orientation is also an example of the predetermined conditions described in the first embodiment.

[0093] If at least one of the following three checks is performed, it can be determined whether the current task is completed. Therefore, one of the three checks, two of the three checks, or all of the three checks can be performed. Performing two checks provides a more accurate determination of whether the current task is completed compared to performing only one check. Performing all three checks provides a more accurate determination of whether the current task is completed compared to performing only one check or performing only two checks.

[0094] • Determine whether the distance D2 is below the threshold Th2.

[0095] • Determine whether the relative position of the gripping components is the specified position.

[0096] • Determine whether the relative directions of the gripping components are the specified directions.

[0097] Furthermore, the margin can be used to determine whether the relative position of the gripping parts is a specified position and whether the relative direction of the gripping parts is a specified direction. In other words, if the difference between the relative position of the gripping parts and the specified position is below a threshold, the relative position of the gripping parts can be determined to be the specified position; if the difference is greater than the threshold, the relative position of the gripping parts can be determined to be not the specified position. Similarly, if the difference (angle difference, etc.) between the relative direction of the gripping parts and the specified direction is below a threshold, the relative direction of the gripping parts can be determined to be the specified direction; if the difference is greater than the threshold, the relative direction of the gripping parts can be determined to be not the specified direction.

[0098] Here, specific examples of all three cases for which the above determinations are made will be explained. Figure 9 (A) to (E) show an example of the state of parts and semi-finished products. Figure 9 Between (A) and (E), the position and orientation of the semi-finished products are the same. Figure 9 (A) shows the component correctly installed on the semi-finished product. Figure 9 In (B), the orientation of the component is... Figure 9 (A) Direction (the relative direction of the components is the specified direction). However, since the distance D2 is not below the threshold Th2, or the position of the component is not... Figure 9 The position of (A) (the relative position of the component is not the specified position) is therefore considered incomplete. Figure 9 In (C), the distance from D2 is below the threshold Th2, and the orientation of the component is... Figure 9 (A) direction, but the position of the component is not. Figure 9 The location (A) indicates the task is incomplete. Figure 9 In (D), the distance from D2 is below the threshold Th2, and the position of the component is... Figure 9 (A) is in position, but the orientation of the component is not. Figure 9 The direction is (A), therefore the task is deemed incomplete. Figure 9 In (E), the distance from D2 is below the threshold Th2, and the position of the component is... Figure 9 (A) position, and the orientation of the component is Figure 9 The direction of (A) indicates that the task is completed.

[0099] As described above, according to the third embodiment, by using conditions different from those in the first embodiment as the prescribed conditions for completing the task, the effects based on the first embodiment can be obtained. Furthermore, by including multiple conditions in the prescribed conditions, it is possible to determine with greater accuracy whether the task has been completed.

[0100] <Other>

[0101] The above embodiments are merely illustrative examples illustrating the structure of the present invention. The present invention is not limited to the specific embodiments described above, and various modifications can be made within the scope of its technical concept. The processing of embodiments 1 to 3 can also be appropriately combined.

[0102] <Postscript 1>

[0103] A job determination device (100, 200), characterized in that it has:

[0104] The image acquisition unit (101, 210) acquires an image of the operator holding the component and installing it onto the semi-finished product.

[0105] The component detection unit (102, 231) detects the image component obtained by the image acquisition unit.

[0106] The holding determination unit (103, 232) determines, based on the detection results of the component detection unit, whether the operator is holding a component, and whether the component held by the operator is the component that should be installed on the semi-finished product in the above-mentioned operation; and

[0107] The control unit (104, 234) is notified to perform control so that if the gripping determination unit determines that the part being gripped by the operator is not a part that should be installed on the semi-finished product in the operation, a prescribed notification is made.

[0108] <Appendix 2>

[0109] A job determination method, characterized by having:

[0110] Image acquisition step (S500): Acquire an image of the operator holding the component to install it onto the semi-finished product.

[0111] The component detection step (S501) detects components from the image obtained in the above image acquisition step;

[0112] The holding determination step (S506~S509) determines, based on the detection results in the component inspection step, whether the operator is holding the component, and whether the component held by the operator is the component that should be installed on the semi-finished product in the above operation; and

[0113] In the notification control step (S504), control is performed so that if, in the aforementioned holding determination step, it is determined that the part held by the operator is not a part that should be installed on the aforementioned semi-finished product in the aforementioned operation, a prescribed notification is made.

[0114] Explanation of reference numerals in the attached figures

[0115] 100: Work determination device

[0116] 101: Image acquisition unit; 102: Component detection unit; 103: Holding determination unit; 104: Notification control unit

[0117] 10: Camera 20: Notification Department

[0118] 200: PC (Work Determination Device)

[0119] 210: Input unit; 220: Storage unit; 230: Control unit; 240: Output unit

[0120] 231: Component Inspection Department; 232: Holding Judgment Department; 233: Work Completion Judgment Department

[0121] 234: Notification Control Department 235: Hand Inspection Department

[0122] 300: Work Image

[0123] 301: Workbench; 302: Component container; 302a~302i: Component

[0124] 303: Semi-finished product; 304: Operator; 305: Placement location for the next process.

[0125] 400: Job Information.

Claims

1. A job determination device, characterized in that, have: The image acquisition unit acquires an image of the operator holding the component and installing it onto the semi-finished product. A component detection unit detects components from the image obtained by the image acquisition unit; The holding determination unit determines whether the operator is holding the component based on the detection result of the component detection unit, and determines whether the component held by the operator is the component that should be installed on the semi-finished product in the operation. as well as The control unit is notified to perform control so that, if the gripping determination unit determines that the part being gripped by the operator is not a part that should be installed on the semi-finished product during the operation, a prescribed notification is issued. If the movement distance of a component detected by the component detection unit is greater than or equal to the first threshold, the gripping determination unit determines that the operator is gripping the component.

2. The job determination device according to claim 1, characterized in that, When multiple components are detected by the component detection unit, the gripping determination unit selects the component with the largest movement distance from the multiple components, and if the movement distance of the selected component is greater than or equal to the first threshold, it determines that the operator is gripping the component.

3. The job determination device according to claim 1 or 2, characterized in that, It also includes an information acquisition unit that acquires information indicating the correspondence between the operation and the semi-finished component to be installed in the operation. The holding determination unit determines whether the component held by the operator is a component that should be installed on the semi-finished product in the operation based on the detection result of the component detection unit and the information obtained by the information acquisition unit.

4. The job determination device according to claim 1 or 2, characterized in that, It also has a work completion determination unit. When the holding determination unit determines that the part held by the operator is a part that should be installed on the semi-finished product in the work, the work completion determination unit determines whether the work completion conditions are met based on the detection results of the part detection unit.

5. The job determination device according to claim 4, characterized in that, The specified conditions include the distance between the component held by the operator and the semi-finished product being below a second threshold.

6. The job determination device according to claim 4, characterized in that, The specified conditions include that the relative position of the component with respect to the semi-finished product is a specified position.

7. The job determination device according to claim 4, characterized in that, The specified conditions include that the relative orientation of the component with respect to the semi-finished product is a specified orientation.

8. The job determination device according to claim 4, characterized in that, The image acquisition unit acquires the image of the worker placing the semi-finished product in a designated location after completing one or more tasks. The component detection unit also detects the semi-finished product from the image acquired by the image acquisition unit. The task completion determination unit also determines whether the semi-finished product has been placed in the designated location based on the detection results of the component detection unit. The notification control unit also controls the placement of the specified notification when the semi-finished product is placed in the specified location before the work completion determination unit determines that all of the one or more work operations have been completed.

9. The job determination device according to claim 4, characterized in that, It also includes a hand detection unit that detects the operator's hand from the image acquired by the image acquisition unit. If the gripping determination unit determines that the part being gripped by the operator is a part that should be installed on the semi-finished product during the operation, and the part being gripped by the operator is not detected by the part detection unit, the operation completion determination unit will use the position of the hand detected by the hand detection unit as the position of the part being gripped by the operator.

10. A method for determining a job, characterized in that, have: The image acquisition step involves capturing an image of the operator holding the component and installing it onto the semi-finished product. The component detection step detects components from the image obtained in the image acquisition step; The holding determination step determines whether the operator is holding the component based on the detection results in the component detection step, and determines whether the component held by the operator is the component that should be installed on the semi-finished product in the operation. as well as The notification control step is implemented to ensure that, in the case where the gripping determination step determines that the component being gripped by the operator is not a component that should be installed on the semi-finished product during the operation, a prescribed notification is issued. In the gripping determination step, if the movement distance of the component detected by the component detection step is greater than or equal to the first threshold, it is determined that the operator is gripping the component.

11. A computer-readable storage medium storing a program for causing a computer to perform the steps of the job determination method of claim 10.

Images