Work support system

Abstract

To appropriately determine whether visual inspection work is correct or incorrect.SOLUTION: A system for supporting a visual inspection performed by a worker, who visually inspects an article to be inspected while moving the worker's fingers, includes an imaging device 20 that captures images of the worker, and a terminal device 10. The terminal device 10 is configured to extract finger skeletal coordinates of the worker from images captured by the imaging device 20, and to determine whether the visual inspection is being performed normally or not based on the finger skeletal coordinates.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a work support system that supports the visual inspection work of an article. [Background technology]

[0002] Patent Document 1 describes a system that extracts skeletal coordinates of a worker's fingers from an image captured by an imaging device and analyzes the work based on the skeletal coordinates. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-163293 A Summary of the Invention [Problem to be solved by the invention]

[0004] At manufacturing sites, etc., visual inspection of processed products is sometimes performed. Visual inspection is performed by workers moving the item to be inspected by hand while visually inspecting it. Depending on the way the item is moved, there may be parts that cannot be visually inspected, resulting in insufficient inspection. [Means for solving the problem]

[0005] The work support system that solves the above problem is a system that provides work support for visual inspection, which is performed by an operator visually inspecting an item to be inspected while moving it with his or her fingers, and is equipped with a photographing device that photographs the operator, and a processing unit, wherein the processing unit is configured to extract finger skeletal coordinates, which are the skeletal coordinates of the fingers, from the image captured by the photographing device, and to determine whether the visual inspection is being performed normally based on the finger skeletal coordinates. Effect of the Invention

[0006] The above-described work support system has an effect of being able to appropriately judge whether a visual inspection work is correct or incorrect. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating a schematic configuration of an embodiment of a work support system. [Diagram 2] FIG. 13 is a diagram showing a state of a visual inspection operation. [Diagram 3] FIG. 13 is a diagram showing an example of a setting mode of body skeleton coordinates. [Figure 4] FIG. 13 is a diagram showing an example of a setting mode of finger skeletal coordinates. [Diagram 5] 13 is a flowchart of an operation monitoring routine. [Figure 6] FIG. 2 is a diagram illustrating a configuration of a learning model used for work monitoring. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] An embodiment of the work support system will be described in detail below with reference to Figures 1 to 6. The work support system of this embodiment is a system that supports the work of visually inspecting an item to be inspected.

[0009] <About the Work Support System Configuration> First, the configuration of a work support system according to the present embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the work support system includes a terminal device 10 and an image capture device 20.

[0010] The functions of the terminal device 10 may be realized by any hardware or any combination of hardware and software. An example of the configuration of the terminal device 10 shown in FIG. 1 includes a main storage device 101, an auxiliary storage device 102, a CPU 103, an interface device 104, a display device 105, and an input device 106, and a bus B1 connecting them. The auxiliary storage device 102 stores a program for realizing the functions of the terminal device 10, and stores necessary files and data. When a program start instruction is received, the main storage device 101 reads out and stores the program from the auxiliary storage device 102. The CPU 103 executes various programs stored in the main storage device 101. The terminal device 10 realizes various functions according to the program executed by the CPU 103. The interface device 104 is used as an interface for connecting to an external communication line B2. The image capture device 20 and the notification device 21 are connected to the interface device 104 via the communication line B2. The display device 105 displays, for example, a GUI (Graphical User Interface) according to a program executed by the CPU 103. The input device 106 is used to allow a user or an administrator of the terminal device 10 to input various operation instructions related to the terminal device 10. In the work assistance system of this embodiment, the terminal device 10 corresponds to a processing unit.

[0011] The terminal device 10 is connected to the photographing device 20 and the notification device 21 via a communication line B2. The photographing device 20 is installed in a visual inspection work area and photographs a worker performing a visual inspection. In the present embodiment, the photographing device 20 is an photographing device capable of acquiring depth information of a subject, such as a stereo camera. Image data photographed by the photographing device 20 is transmitted to the terminal device 10 via the communication line B2. The notification device 21 is a device that notifies a worker of the correctness or incorrectness of a task through hearing or vision. Examples of the notification device 21 are a speaker, a buzzer, a monitor, and an electronic bulletin board.

[0012] <Visual inspection work> Next, an example of a visual inspection task supported by the work support system will be described with reference to FIG. 2. The workpiece W shown in FIG. 2 is an item to be visually inspected. Here, a machine part processed by a lathe is taken as an example of the workpiece W. In the case of FIG. 2, during visual inspection, the worker holds the end of the workpiece W with his left hand while supporting the lower side of the workpiece W with his right hand. Then, while rotating the workpiece W with his left hand, the worker visually inspects the entire side circumference of the workpiece W processed by the lathe to check for defects such as scratches. Note that the visual inspection may also be performed on other parts, such as painted or plated surfaces and welded parts.

[0013] <Body skeletal coordinates and finger skeletal coordinates> During the visual inspection, the image capturing device 20 transmits the captured image to the terminal device 10. The terminal device 10 extracts the worker's finger skeletal coordinates and body skeletal coordinates from the image received from the image capturing device 20. The finger skeletal coordinates are the skeletal coordinates of the worker's left and right fingers. The body skeletal coordinates are the skeletal coordinates of the parts of the worker's body other than the fingers.

[0014] An example of a setting mode of the body skeleton coordinates is shown in Fig. 3. In Fig. 3, the head, left and right shoulder joints, left and right elbow joints, left and right hands, and left and right hip joints of the worker are set as extraction points of the body skeleton coordinates. In this embodiment, the terminal device 10 extracts the body skeleton coordinates from the image as two-dimensional coordinates not including depth.

[0015] An example of how finger skeletal coordinates are set is shown in Fig. 4. In Fig. 4, the wrist, the joints of each finger, and the fingertips of each hand of the worker are set as extraction points of the finger skeletal coordinates. In this embodiment, the terminal device 10 extracts the finger skeletal coordinates from the image as three-dimensional coordinates including depth.

[0016] In this embodiment, a series of movements of the worker's body and fingers from the start to the end of the visual inspection work is divided into multiple stages in a time series. When the worker is performing the visual inspection work normally, the body movements of the worker are generally uniform. In other words, when the visual inspection work is performed normally, each body skeletal coordinate in each stage falls within a certain range.

[0017] Also, from the finger skeletal coordinates, the way in which the workpiece W is held can be confirmed. For example, from the finger skeletal coordinates, the distance between the fingertips of the left and right thumbs can be obtained. When the workpiece W is held correctly, the distance between the fingertips of the left and right thumbs is within a certain range. Also, from the transition of the finger skeletal coordinates of the left hand that rotates the workpiece W, the amount of rotation of the workpiece W can be obtained. Furthermore, in the case of this embodiment, the finger skeletal coordinates are three-dimensional coordinates. Therefore, the positional relationship between the workpiece W and the left and right hands as seen by the worker's eyes can also be confirmed from the finger skeletal coordinates. For example, when the worker's hand is located in front of the workpiece W as seen by the worker's eyes, and a part of the workpiece W is covered by the worker's hand and cannot be seen, this can be detected from the finger skeletal coordinates. In this way, based on the finger skeletal coordinates, it can be confirmed whether the worker is handling the workpiece W appropriately during visual inspection.

[0018] In the present embodiment, the distribution ranges of the body skeletal coordinates and the finger skeletal coordinates at each stage of the visual inspection work when the visual inspection work is performed normally are obtained in advance. In the following description, the distribution ranges of the body skeletal coordinates and the finger skeletal coordinates at each stage when the visual inspection work is performed normally are referred to as normal frames. If the work is not performed normally, the body skeletal coordinates and the finger skeletal coordinates deviate from the normal frame of the current stage. Therefore, if the normal frame of each stage is obtained in advance, it is possible to determine whether the visual inspection work is being performed normally or not depending on whether the body skeletal coordinates and the finger skeletal coordinates are within the normal frame of the current stage. In the following description, the ranges of the body skeletal coordinates and the finger skeletal coordinates that deviate from the normal frame are referred to as abnormal frames. In this case, the case where the body skeletal coordinates and the finger skeletal coordinates are within the abnormal frame is the case where the visual inspection work is not being performed normally.

[0019] In the following explanation, the order of a visual inspection work stage when the stages are arranged in chronological order is represented as the value of the frame number FN. The terminal device 10 is aware of the current work stage based on the value of the frame number FN. The terminal device 10 counts up the frame number FN each time the work stage progresses. When the count-up causes the value of the frame number FN to exceed the value indicating the final stage of the visual monitoring work, the terminal device 10 resets the value of the frame number FN to its initial value of "1."

[0020] <About work monitoring> Next, the task monitoring of visual inspection by the task support system of this embodiment will be described with reference to Fig. 5. Fig. 4 shows a flowchart of a task monitoring routine executed by the terminal device 10 for task monitoring. The terminal device 10 repeatedly executes this routine at every predetermined control period while task monitoring is being performed.

[0021] When starting this routine, the terminal device 10 first acquires an image captured by the photographing device 20 (S10). Next, the terminal device 10 extracts the skeletal coordinates of the worker's fingers and body from the acquired image (S12). Then, the terminal device 10 determines whether the extracted skeletal coordinates of the fingers and body are located within an abnormal frame corresponding to the current frame number FN (S14).

[0022] If the finger skeletal coordinates and the body skeletal coordinates are not located in the abnormal frame, i.e., if they are located in the normal frame (S14: NO), the terminal device 10 determines that the visual inspection work is being performed normally. In this case, the terminal device 10 counts up the value of the frame number FN (S16). Furthermore, the terminal device 10 resets the value of the abnormal frame counter CNT to "0" (S18). Then, the terminal device 10 ends the process related to the work monitoring in the current control cycle.

[0023] On the other hand, if the finger skeletal coordinates and the body skeletal coordinates are located within the abnormal frame (S14: YES), the terminal device 10 determines that the visual inspection work is not being performed normally. However, even if the finger skeletal coordinates and the body skeletal coordinates are temporarily located within the abnormal frame, if they immediately return to the normal frame, it is considered that the visual inspection work is being performed normally. Therefore, when it is confirmed that the finger skeletal coordinates and the body skeletal coordinates have deviated from the normal frame, the terminal device 10 reserves the determination that an abnormality has occurred in the work. Then, the terminal device 10 counts up the value of the abnormal frame counter CNT (S20). Then, the terminal device 10 determines whether the value of the abnormal frame counter CNT is equal to or greater than a predetermined threshold (S22). If the value of the abnormal frame counter CNT is less than the threshold (S22: NO), the terminal device 10 ends the process related to the work monitoring in the current control cycle.

[0024] On the other hand, if the value of the abnormal frame counter CNT is equal to or greater than the threshold value (S22: YES), the terminal device 10 determines that an abnormality has occurred in the visual inspection work. Then, the terminal device 10 instructs the notification device 21 to notify the worker of the occurrence of the abnormality (S24), and then ends the processing related to the work monitoring in the current control cycle.

[0025] The value of the abnormal frame counter CNT is reset to "0" when it is determined in S14 of Fig. 5 that the work is being performed normally. On the other hand, when it is determined in S14 of Fig. 5 that the work is not being performed normally, the value of the abnormal frame counter CNT is counted up. The value of the abnormal frame counter CNT thus manipulated represents the duration of the state in which it is determined that the visual inspection is not being performed normally.

[0026] <About judging whether visual inspection work is correct or not> Next, a description will be given of the judgment of the correctness of the visual inspection task based on the finger skeletal coordinates and the body skeletal coordinates performed in S14 of Fig. 5. The terminal device 10 performs the judgment in S14 using the learning model shown in Fig. 6. This learning model is learned by machine learning. The auxiliary storage device 102 of the terminal device 10 stores the learned learning model in advance.

[0027] The learning model in FIG. 6 is configured as a neural network having an input layer consisting of N nodes, an intermediate layer consisting of M nodes, and an output layer consisting of two nodes. The input values ​​X1, X2, X3, ..., XN of each node in the input layer are the number of frames FN, each finger skeletal coordinate, and each body skeletal coordinate. The output values ​​of the two nodes in the output layer are the likelihoods of a normal frame and an abnormal frame, respectively. The likelihood of a normal frame is the likelihood that the input finger skeletal coordinates and body skeletal coordinates are located within a normal frame corresponding to the input number of frames FN. The likelihood of a normal frame is the likelihood that the input finger skeletal coordinates and body skeletal coordinates deviate from a normal frame corresponding to the input number of frames FN.

[0028] Here, the input values ​​to each node in the middle layer are U1, U2, ..., UM, and the output values ​​of each node in the middle layer are Z1, Z2, ..., ZM. In addition, "i" is a natural number from "1" to "9", "j" is a natural number from "1" to "M", and "k" is a natural number from "1" to "L". The input value Uj of each node in the middle layer is calculated as the sum of the values ​​obtained by multiplying the input values ​​X1, X2, ..., X9 of the input layer by the weight Vij. The output value Zj of each node in the middle layer is calculated as the return value of the activation function F with the input value Uj of the node as an argument. An example of the activation function F is the sigmoid function. The output values ​​Y1, Y2, ..., YL of each node in the output layer are calculated as the sum of the values ​​obtained by multiplying the output values ​​Zj of each node in the middle layer by the weight Wjk.

[0029] Such a learning model is trained using a data set consisting of the number of frames FN during the visual inspection work, each finger skeletal coordinate, each body skeletal coordinate, and the likelihood of normal and abnormal frames as training data. Such a data set is created from images taken by the imaging device 20 at each stage during the visual inspection work. Images used to create the data set include images when the visual inspection work is being performed normally and images when an abnormality occurs. In the data set created from images when the work is being performed normally, the likelihood value of a normal frame is set to "1" and the likelihood value of an abnormal frame is set to "0". In addition, in the data set created from images when an abnormality occurs in the work, the likelihood value of a normal frame is set to "0" and the likelihood value of an abnormal frame is set to "1".

[0030] The terminal device 10 performs the following process for each data set to learn the learning model. The terminal device 10 inputs the number of frames FN, finger skeletal coordinates, and body skeletal coordinates in the data set to the neural network as input values ​​X1 to XN of the input layer, and calculates the output values ​​Y1 and Y2 of the neural network for the input. The terminal device 10 then corrects the values ​​of the weights Vij and Wjk using the backpropagation method so that the error between the calculated output values ​​Y1 and Y2 and the likelihood values ​​of normal frames and abnormal frames in the data set is reduced. The terminal device 10 repeats the correction process of the weights Vij and Wjk until the error is equal to or smaller than a predetermined value, thereby learning the neural network. Note that such learning of the learning model may be performed by a device other than the terminal device 10, and the learned learning model may be stored in the auxiliary storage device 102.

[0031] 5, the terminal device 10 inputs the value of the current frame number FN and the values ​​of the finger skeletal coordinates and the body skeletal coordinates extracted from the image into the trained learning model, and calculates the values ​​of the output values ​​Y1 and Y2. If the value of the output value Y2 is greater than the output value Y1 and is equal to or greater than a predetermined threshold, the terminal device 10 determines that the finger skeletal coordinates and the body skeletal coordinates are in an abnormal frame.

[0032] <Operation and Effects of the Embodiment> The operation and effects of this embodiment will be described. The terminal device 10 of the work support system of this embodiment extracts the finger skeletal coordinates and body skeletal coordinates of the worker from the image captured by the photographing device 20. Then, the terminal device 10 judges whether the visual inspection is being performed normally or not based on the extracted finger skeletal coordinates and body skeletal coordinates. If the terminal device 10 judges that the visual inspection is not being performed normally for a predetermined period of time or more, it judges that an abnormality has occurred in the visual inspection work. Then, the terminal device 10 notifies the worker of the occurrence of the abnormality via the notification device 21.

[0033] According to the work assistance system of the present embodiment described above, the following effects can be obtained. (1) The work support system of this embodiment includes an imaging device 20 that images a worker, and a terminal device 10. The terminal device 10 extracts the skeletal coordinates of the worker's fingers from the image captured by the imaging device 20, and determines whether the visual inspection is being performed correctly based on the skeletal coordinates. The visual inspection is performed by the worker visually inspecting the workpiece W, which is the item to be inspected, while moving the worker's fingers. Based on the skeletal coordinates, it is possible to check whether the worker is handling the workpiece W appropriately during such visual inspection. Therefore, the work support system of this embodiment has the effect of being able to appropriately determine whether the visual inspection work is correct or incorrect.

[0034] (2) The terminal device 10 extracts body skeletal coordinates, which are the skeletal coordinates of the parts of the worker's body other than the fingers, from the image taken by the photographing device 20. The terminal device 10 then determines whether the visual inspection is being performed correctly based on the body skeletal coordinates in addition to the finger skeletal coordinates. The body skeletal coordinates allow the movement of the worker's entire body to be confirmed. Therefore, by using the body skeletal coordinates in addition to the finger skeletal coordinates, the accuracy of determining whether the visual inspection is correct or incorrect can be improved.

[0035] (3) In the work support system of this embodiment, the finger skeletal coordinates are extracted as three-dimensional coordinates, and the body skeletal coordinates are extracted as two-dimensional coordinates. The accuracy of judging whether the visual inspection is correct or incorrect can be improved by obtaining the finger skeletal coordinates as three-dimensional coordinates and accurately grasping the relative position and movement of the fingers with respect to the workpiece W. In contrast, since the movement of parts of the worker other than the fingers during visual inspection is small, even if the body skeletal coordinates are three-dimensional coordinates, the improvement in the judgment accuracy is limited. On the other hand, when the body skeletal coordinates are three-dimensional coordinates, the amount of data used for judgment increases compared to when the body skeletal coordinates are two-dimensional coordinates, and the calculation load of the terminal device 10 increases. Therefore, the accuracy of judging whether the visual inspection is correct or incorrect can be improved while suppressing the increase in the calculation load of the terminal device 10.

[0036] (4) The terminal device 10 notifies the worker of an abnormality when a state in which it is determined that the visual inspection is not being performed normally continues for a predetermined period of time or more. Since the abnormality notification is only given when there is a high possibility that an abnormality has occurred in the work, the frequency of erroneous notifications can be reduced.

[0037] (5) The terminal device 10 performs the above-mentioned judgment using a learning model trained by machine learning. There is variation in the physique of workers and their movements during visual inspection. By using the above-mentioned learning model, it is possible to easily realize a judgment of the correctness of visual inspection that reflects such variation.

[0038] This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction. An abnormality notification may be issued when the visual inspection is judged to be abnormal a certain number of times, even if the number of times is not consecutive. Also, an abnormality notification may be issued when the visual inspection is judged to be abnormal even once.

[0039] The judgment using the finger skeletal coordinates and the judgment using the body skeletal coordinates may be performed separately. For example, it is judged whether or not a visual inspection is being performed based on the body skeletal coordinates. Then, if it is judged that a visual inspection is being performed, the correctness of the visual inspection is judged using the finger skeletal coordinates.

[0040] The body skeleton coordinates may be three-dimensional coordinates, and the finger skeleton coordinates may be two-dimensional coordinates. As the learning model used to determine whether the visual inspection is performed normally or not, a learning model having a configuration different from that shown in Fig. 6 may be used. Also, the above determination may be made without using a learning model learned by machine learning. [Explanation of symbols]

[0041] 10 Terminal Equipment 101 Main storage 102 Auxiliary storage 103 CPU 104 Interface device 105 Display device 106 Input Devices 20 Imaging Equipment 21 Notification device W Work (items subject to inspection)

Claims

1. A system for supporting visual inspection, which is performed by an operator visually inspecting an object to be inspected while moving the object with his / her fingers, comprising: The system includes an imaging device that images the worker, and a processing unit, The processing unit includes: extracting finger skeletal coordinates, which are skeletal coordinates of the fingers, from the image captured by the imaging device; Determining whether the visual inspection is performed normally based on the finger skeletal coordinates; A work support system that performs the above.

2. The work support system according to claim 1, wherein the processing unit extracts body skeletal coordinates, which are skeletal coordinates of parts of the worker's body other than the fingers, from the image captured by the photographing device, and makes the judgment based on the finger skeletal coordinates and the body skeletal coordinates.

3. 3. The task support system according to claim 2, wherein the finger skeletal coordinates are three-dimensional coordinates, and the body skeletal coordinates are two-dimensional coordinates.

4. The work support system according to claim 1 , wherein the processing unit notifies the worker of an abnormality when a state in which the visual inspection is determined to be not performed normally continues for a predetermined time or longer.

5. The task support system according to claim 1 , wherein the processing unit makes the determination using a learning model learned by machine learning.

Images