Work support system

Abstract

To make it possible to identify mis-delivery of components while preventing workability from deteriorating.SOLUTION: A terminal device 10 extracts the skeletal coordinates of an operator in an image that an imaging device 20 has captured, and based on the time series data of the skeletal coordinates, performs a determination process to determine whether the operator has carried component-boxes to a delivery entrance, and when it is determined by the determination process that the component-boxes have been carried in, performs a comparison process to compare the carried-in component-boxes with the delivery entrance to which the component-boxes have been carried.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a work support system for supporting a part carrying-in work. [Background technology]

[0002] In some cases, parts are delivered to a production factory or the like by placing a parts box in a designated entrance on a parts shelf. Patent Document 1 discloses a technique for preventing a parts box from being delivered to an entrance other than the designated one by using a reading device worn on the hand of a worker who delivers the parts. The reading device has a function for reading a QR code (registered trademark) attached to the parts box and a function for reading an RFID. The reading device then checks the QR code read from the parts box against the RFID read from an RF tag installed at each entrance on the parts shelf to determine whether or not the part box has been delivered in error. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6729498 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional technology, a worker must attach a reading device and read the QR code from the parts box, which makes it difficult to carry in parts. [Means for solving the problem]

[0005] The work support system that solves the above problem is a system that supports part delivery work in which a parts box containing parts is delivered to a part-specific delivery entrance provided on a parts shelf, and is equipped with a processing unit and a camera that photographs the parts shelf. The processing unit is configured to extract the skeletal coordinates of the worker in the image captured by the camera, and perform a determination process that determines whether the worker has delivered the parts box to the delivery entrance based on time-series data of the skeletal coordinates, and a comparison process that, if it is determined in the determination process that the parts box has been delivered, compares the delivered parts box with the delivery entrance through which the parts box was delivered.

[0006] The task assistance system can be configured to perform the determination process using a learning model learned by machine learning. Effect of the Invention

[0007] The above-described work support system has the effect of checking for erroneous delivery of parts while preventing deterioration of workability. [Brief description of the drawings]

[0008] [Figure 1] FIG. [Diagram 2] 1 is a diagram illustrating a schematic configuration of an embodiment of a work support system. [Diagram 3] FIG. 2 is a diagram showing an example of an image captured by a photographing device. [Figure 4] FIG. 13 is a diagram showing an example of a setting mode of skeleton coordinates. [Diagram 5] 4 is a flowchart showing a flow of processes executed in the work support system. [Figure 6] 1 is a diagram illustrating a configuration of a learning model used by the work support system for carrying-in determination. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the work support system will be described in detail with reference to FIGS. <About parts delivery> The parts carrying-in work supported by the work support system of this embodiment is carried out in a parts storage area of ​​a production factory, etc. In the parts storage area, parts shelves 30 shown in Fig. 1 are installed for each part delivered from the parts manufacturer to the production factory.

[0010] 1, parts shelf 30 has a carry-in entrance 31 and a collection entrance 32. Carry-in entrance 31 is an opening for carrying in parts boxes 34 that contain parts to be delivered from a parts manufacturer to a production factory. Collection entrance 32 is an opening for removing collection boxes 35 that contain parts to be returned from the production factory to the parts manufacturer.

[0011] A tag 33 indicating the part number of the part to be brought into the carry-in entrance 31 is attached to the parts shelf 30. This tag 33 also indicates the part number of the part to be collected through the collection opening 32 provided on the parts shelf 30. Similarly, a tag 36 indicating the part number of the part contained therein is attached to the parts box 34 and the collection box 35. The tags 33, 36 may indicate the part number in letters, or may indicate the part number by a bar code or a two-dimensional code. In this embodiment, the tags 36 of the parts box 34 and the collection box 35 may also indicate information such as the number of parts contained therein.

[0012] The parts shelf 30 may have a plurality of carry-in entrances 31 and collection entrances 32 corresponding to parts with different part numbers. In this case, the parts shelf 30 has a plurality of tags 33 indicating part numbers. A worker who brings in parts brings in a parts box 34 into the entrance 31 of the parts shelf 30 that corresponds to the part number. If there are parts to be returned, a return box 35 is placed at the return opening 32 of that parts shelf 30. In such a case, the worker brings in the parts box 34 and then returns the return box 35 from the return opening 32.

[0013] <Work support system configuration> Next, the configuration of the work support system of this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the work support system includes a terminal device 10, an image capturing device 20, and a notification device 21.

[0014] The functions of the terminal device 10 may be realized by any hardware or any combination of hardware and software. An example of the configuration of the terminal device 10 shown in FIG. 1 includes a main storage device 101, an auxiliary storage device 102, a CPU 103, an interface device 104, a display device 105, and an input device 106, and a bus B1 connecting them. The auxiliary storage device 102 stores a program for realizing the functions of the terminal device 10, and stores necessary files and data. When a program start instruction is received, the main storage device 101 reads out and stores the program from the auxiliary storage device 102. The CPU 103 executes various programs stored in the main storage device 101. The terminal device 10 realizes various functions according to the program executed by the CPU 103. The interface device 104 is used as an interface for connecting to an external communication line B2. The image capture device 20 and the notification device 21 are connected to the interface device 104 via the communication line B2. The display device 105 displays, for example, a GUI (Graphical User Interface) according to a program executed by the CPU 103. The input device 106 is used to allow a user or an administrator of the terminal device 10 to input various operational instructions relating to the terminal device 10 .

[0015] The photographing device 20 and the notification device 21 are installed in a parts storage area. The photographing device 20 photographs images of each parts shelf 30 installed in the parts storage area and the surrounding area. The notification device 21 is configured to be capable of issuing two types of notifications, audibly and visually, to workers working in the parts storage area, a normal notification and an abnormality notification.

[0016] <About skeletal coordinates> The photographing device 20 in the work support system of this embodiment is installed in a position where it can photograph each of the parts shelves 30 installed in the parts storage area and the surrounding area. The work support system of this embodiment supports the part loading work based on the image of the photographing device 20.

[0017] 3 shows an example of an image captured by the camera device 20. This image includes three parts shelves 30 and a worker W working in front of them holding a parts box 34. Note that although skeletal coordinates C1 to C9, which will be described later, are shown in the figure, these are not included in the actual image. The terminal device 10 extracts the skeletal coordinates C1 to C9 of the worker W from the image captured by the camera device 20 by image processing.

[0018] 4 shows an example of setting skeletal coordinates C1 to C9. In this example, the head (C1), left and right shoulder joints (C2, C3), left and right elbow joints (C4, C5), left and right hands (C6, C7), and left and right hip joints (C8, C9) of the worker W are set as extraction points of skeletal coordinates. In this embodiment, the skeletal coordinates C1 to C9 are two-dimensional coordinates, but the skeletal coordinates C1 to C9 may be three-dimensional coordinates when the image contains depth information of the subject, for example.

[0019] <About the processing of the work support system> The process executed by the work support system of this embodiment will be described with reference to Fig. 5. Fig. 5 shows the flow of the process executed by the terminal device 10 to support the carry-in work. The image of the parts storage area captured by the photographing device 20 is input to the terminal device 10. The "S" of S100, S110, ..., S240 in Fig. 5 represents the steps of the process executed by the terminal device 10.

[0020] When the terminal device 10 imports the photographed data of the photographing device 20 in S100, it performs worker authentication in S110. In worker authentication, the terminal device 10 judges whether or not a person is present in the image photographed by the photographing device 20. If a person is present in the image, the terminal device 10 judges whether or not the person is a pre-registered worker W who brings in parts. This judgment is made, for example, through face recognition or recognition based on skeleton coordinates C1 to C9. The worker authentication is established when a person is present in the image and it is determined that the person is a pre-registered worker W. If the worker authentication is established (S120: YES), the terminal device 10 proceeds to the process of S130. On the other hand, if the worker authentication is not established (S120: NO), the terminal device 10 returns the process to S100.

[0021] In S130, the terminal device 10 extracts the skeleton coordinates C1 to C9 of the worker W from the video. In the following S140, the terminal device 10 judges whether the worker W has carried in the parts box 34 to the entrance 31 based on the time series data of the skeleton coordinates C1 to C9. If the terminal device 10 judges that the parts box 34 has been carried in, it identifies the parts shelf 30 to which the parts box 34 has been carried in. The time series data of the skeleton coordinates C1 to C9 is the skeleton coordinates C1 to C9 of the worker W extracted from the video of the image capture device 20 arranged in time series. From such time series data of the skeleton coordinates C1 to C9, the actions performed by the worker W can be confirmed. If the terminal device 10 judges that the parts box 34 has not been carried in (S150: NO), it returns the process to S100. If the terminal device 10 judges that the parts box 34 has been carried in (S150: YES), it proceeds to S160.

[0022] In S160, the terminal device 10 analyzes the image of the camera device 20 and reads the product numbers of the entrance 31 and the parts box 34. Specifically, the terminal device 10 identifies the tag 33 of the parts shelf 30 having the entrance 31 through which the parts box 34 has been carried from the image of the camera device 20. The terminal device 10 then reads the product number information displayed on the tag 33 in the image. The terminal device 10 also reads the product number information displayed on the tag 36 of the parts box 34 carried into the entrance 31 from the image of the camera device 20. The terminal device 10 then determines whether the read product numbers of the entrance 31 and the parts box 34 match (S170). The terminal device 10 performs this determination at regular time intervals to avoid performing multiple determinations for one carry-in.

[0023] If the terminal device 10 determines that the part numbers of the entrance 31 and the parts box 34 match (S170: YES), the process proceeds to S180. In S180, the terminal device 10 causes the notification device 21 to issue a normal notification. The normal notification here means that the parts box 34 has been brought into the appropriate entrance 31. In contrast, if the terminal device 10 determines that the part numbers of the entrance 31 and the parts box 34 do not match (S170: NO), the process proceeds to S190. In S190, the terminal device 10 causes the notification device 21 to issue an abnormal notification. The abnormal notification here means that the parts box 34 has been brought into an inappropriate entrance 31.

[0024] After processing S180 or S190, the terminal device 10 advances the process to S200. In S200, the terminal device 10 determines whether or not the collection box 35 has been collected. For example, if the collection box 35 was present at the collection port 32 before the parts box 34 was carried in, but the collection box 35 is no longer present at the collection port 32 when the worker W leaves the parts shelf 30 thereafter, the terminal device 10 determines that collection has been performed. The terminal device 10 analyzes the image captured by the imaging device 20 to confirm whether or not the collection box 35 is present at the collection port 32.

[0025] Thereafter, when the terminal device 10 determines that the parts have not been collected (NO), it returns the process to S100. On the other hand, when the terminal device 10 determines that the parts have been collected (YES), it reads the information of the part number displayed on the tag 36 of the parts box 34 collected by the worker W in S210 by analyzing the video of the photographing device 20. Then, the terminal device 10 determines whether or not the part number of the collected box 35 collected by the worker W matches the part number of the parts box 34 that the worker W previously brought in in S220. Then, when the terminal device 10 determines that the part numbers match (YES), it causes the notification device 21 to perform a normal notification in S230. The normal notification here means that the worker W has collected the appropriate parts box 34. On the other hand, when the terminal device 10 determines that the part numbers do not match (NO), it causes the notification device 21 to perform an abnormality notification in S240. The abnormality notification here means that the worker W has collected the inappropriate parts box 34. After the process of S230 or S240, the terminal device 10 returns to the process of S100.

[0026] <Determination of parts box delivery> Next, with reference to Fig. 6 as well, the determination of whether or not a parts box 34 has been brought in, which is performed by the terminal device 10 in S140 of Fig. 5, will be described. The terminal device 10 determines whether or not a parts box 34 has been brought in, using the learning model shown in Fig. 6. In the following description, the number of parts shelves 30 by part number installed in the parts storage area is assumed to be "L-1". The entrances 31 of the parts shelves 30 are respectively referred to as entrance P1, entrance P2, ..., entrance P(L-1), and entrance PL.

[0027] The learning model in Fig. 6 is configured as a neural network having an input layer consisting of N nodes, an intermediate layer consisting of M nodes, and an output layer consisting of L nodes. The input values ​​X1, X2, ..., XN of each node in the input layer are time-series data of the skeleton coordinates C1 to C9 of the worker W. The output values ​​Y1, Y2, ..., YL of each node in the output layer are the likelihood of each type of the worker's motion. The motions of the worker W are classified into motions of carrying in the parts box 34 into each of the carry-in entrances P1 to P(L-1) and other motions.

[0028] Here, the input values ​​to each node in the middle layer are U1, U2, ..., UM, and the output values ​​of each node in the middle layer are Z1, Z2, ..., ZM. Also, "i" is a natural number from "1" to "N", "j" is a natural number from "1" to "M", and "k" is a natural number from "1" to "L". The input value Uj of each node in the middle layer is calculated as the sum of the values ​​obtained by multiplying the input values ​​X1, X2, ..., XN of the input layer by the weight Vij. The output value Zj of each node in the middle layer is calculated as the return value of the activation function F with the input value Uj of the node as an argument. An example of the activation function F is the sigmoid function. The output values ​​Y1, Y2, ..., YL of each node in the output layer are calculated as the sum of the values ​​obtained by multiplying the output values ​​Zj of each node in the middle layer by the weight Wjk.

[0029] Such a learning model is trained using a data set as training data, which is composed of time-series data of skeletal coordinates C1 to C9 when worker W performs various actions, and a one-dimensional array in which the likelihood of actions performed by worker W is 1 and other actions are 0. The data set is created by acquiring time-series data of skeletal coordinates C1 to C9 of worker W from images captured by the image capture device 20 when worker W performs various tasks.

[0030] The terminal device 10 performs the following process for each data set to learn the learning model. The terminal device 10 inputs the time series data of the skeleton coordinates C1 to C9 in the data set to the neural network as the input values ​​X1 to XN of the input layer, and calculates the output values ​​Y1 to YL of the neural network for the input. The terminal device 10 then corrects the values ​​of the weights Vij and Wjk using the error backpropagation method so that the error between the calculated output values ​​Y1 to YL and each value of the one-dimensional array indicating the type of movement in the data set is reduced. The terminal device 10 repeats the correction process of the weights Vij and Wjk until the error is equal to or smaller than a predetermined value, thereby learning the neural network. Note that such learning of the learning model may be performed by a device other than the terminal device 10, and the learned learning model may be stored in the auxiliary storage device 102.

[0031] 5, the terminal device 10 inputs the time-series data of the skeleton coordinates C1-C9 of the worker W extracted from the image captured by the image capture device 20 into the learning model as the values ​​of the input values ​​X1-XN. The terminal device 10 then calculates the output values ​​Y1-YL of the learning model for the input. The terminal device 10 then determines whether the worker W has brought in the parts box 34 and identifies the entrance 31 through which the parts box 34 has been brought in by finding the maximum value among the calculated output values ​​Y1-YL.

[0032] <Effects of the embodiment> The operation of the work support system of this embodiment will be described. When the terminal device 10 recognizes a worker in the image captured by the image capturing device 20, it performs worker authentication. Furthermore, when the terminal device 10 has successfully authenticated the worker, it starts extracting the skeleton coordinates C1-C9 of the worker W. Then, based on the time-series data of the skeleton coordinates C1-C9 extracted from the image, the terminal device 10 performs a determination process to determine whether or not the worker W has carried in the parts box 34 into the carry-in entrance 31 (S140 in FIG. 5).

[0033] Furthermore, when the terminal device 10 determines in the above-mentioned determination process that the parts box 34 has been carried in, it performs a matching process of matching the carried-in parts box 34 with the carry-in entrance 31 through which the parts box 34 was carried in (S170 in FIG. 5). The terminal device 10 reads, from the image of the imaging device 20, the part number displayed on the tag 33 of the parts shelf 30 where the carry-in entrance 31 is provided, and the part number displayed on the tag 36 of the parts box 34. The terminal device 10 then performs the matching process by determining whether the read part number of the carry-in entrance 31 and the part number of the parts box 34 match. In this embodiment, the terminal device 10 that performs the above-mentioned determination process and matching process corresponds to the processing unit.

[0034] The terminal device 10 notifies the worker W of the result of the collation process through the notification device 21. This enables the worker W to confirm whether the parts box 34 has been carried into the appropriate carry-in entrance 31 or not. Furthermore, the terminal device 10 determines whether or not the worker W has collected the collection box 35 from the collection port 32, based on the image captured by the photographing device 20. If the terminal device 10 determines that the collection box 35 has been collected, it reads the product number of the collection box 35 collected by the worker W from the image captured by the photographing device 20. The terminal device 10 then determines whether or not the read product number of the collection box 35 matches the product number of the parts box 34 previously brought in by the worker W. The terminal device 10 then notifies the worker W of the result of this determination via the notification device 21. This allows the worker W to confirm whether or not the correct collection box 35 has been collected.

[0035] According to the work assistance system of the present embodiment described above, the following effects can be obtained. (1) The terminal device 10 performs a judgment process and a verification process. The judgment process is a process of extracting the skeleton coordinates C1 to C9 of the worker W in the image captured by the imaging device 20, and judging whether or not the worker W has brought in the parts box 34 to the entrance 31 based on the time series data of the skeleton coordinates C1 to C9. The verification process is a process of verifying the brought-in parts box 34 and the entrance 31 through which the parts box 34 was brought in when it is judged in the judgment process that the parts box 34 has been brought in. These processes make it possible to confirm whether or not the worker W has brought in the parts box 34 to the appropriate entrance 31. The judgment process is performed based on the image captured by the imaging device 20, and the worker W does not need to perform any special work for the judgment process. Therefore, the work support system of this embodiment has the effect of making it possible to check for erroneous parts being brought in while suppressing deterioration of workability.

[0036] (2) The terminal device 10 reads the part number indicated on the tag 36 of the parts box 34 brought into the entrance 31 from the image captured by the camera device 20 and performs the matching process. Therefore, the part number of the parts box 34 in the matching process can be identified using the image from the camera device 20 used for judgment in the judgment process. Therefore, there is no need to add a separate device to identify the part number of the parts box 34 in the matching process.

[0037] (3) The terminal device 10 reads the part number displayed at the entrance 31 through which the parts box 34 was brought in from the image captured by the camera device 20 and performs the matching process. Therefore, the part number of the entrance 31 can be identified in the matching process using the image from the camera device 20 used for judgment in the judgment process. Therefore, it is not necessary to add a separate device to identify the part number of the entrance 31 in the matching process. Furthermore, rearrangement of the parts shelves 30 in the parts storage area and addition or change of part part numbers can be handled simply by replacing the tags 33 on the parts shelves 30.

[0038] (4) The work support system of this embodiment includes a notification device 21 that notifies the worker W of the result of the collation process. Therefore, the worker W can be notified on the spot of the erroneous delivery of the parts box 34 to an inappropriate delivery entrance 31.

[0039] (5) The terminal device 10 performs the determination process using a learning model learned by machine learning. Therefore, it is possible to easily build a system that performs the determination process of whether the parts box 34 has been carried into the carry-in entrance 31 based on the time-series data of the skeleton coordinates C1 to C9.

[0040] (6) The terminal device 10 determines whether the worker W has collected the collection box 35 from the collection port 32 based on the image captured by the photographing device 20. The terminal device 10 then reads the product number of the collection box 35 from the image captured by the photographing device 20 and compares it with the product number of the parts box 34 previously brought in by the worker W. This makes it possible to check whether the worker W has erroneously collected the collection box 35. Furthermore, the terminal device 10 notifies the worker W of the result of the above determination via the notification device 21. This makes it possible to notify the worker W of the erroneous collection of the collection box 35 on the spot.

[0041] <Other embodiments> This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction.

[0042] By configuring the work support system as follows, it becomes possible to automatically manage the stock quantity of each part. A data table that records the stock quantity of each part is stored in the auxiliary storage device 102 of the terminal device 10. When the terminal device 10 confirms the delivery of a parts box 34 or the collection of a collection box 35, it reads the information on the number of parts displayed on the tags 36 from the images captured by the camera device 20, and operates the stock quantity of parts in the data table.

[0043] As a learning model used to determine whether the parts box 34 has been delivered based on the time-series data of the skeleton coordinates, a learning model having a configuration different from that shown in Fig. 6 may be used. Also, it may be possible to determine whether the parts box 34 has been delivered without using a learning model learned by machine learning.

[0044] The determination of whether the collection box 35 has been collected in S200 of Fig. 5 may be performed in a manner different from the above-described manner. For example, similar to the determination of whether the parts box 34 has been carried in in S140 of Fig. 5, it may be determined whether the collection box 35 has been collected based on time-series data of the skeleton coordinates.

[0045] In the above embodiment, the part numbers of the parts contained in the parts box 34 or the collection box 35 were identified by reading the part number indications on the parts box 34 or the collection box 35 from the image captured by the imaging device 20. However, the part numbers may be identified by other methods. For example, an electronic tag recording part number information may be attached to the parts box 34 or the collection box 35, and the part number information may be obtained through wireless communication with the electronic tag.

[0046] When the part numbers of the parts to be brought in and collected by each worker W are determined, the part numbers of the parts box 34 and the collection box 35 that the worker W brings in and collects can be identified by the following method. A data table that records the correspondence between each worker W and the part numbers of the parts that the worker W brings in and collects is stored in advance in the auxiliary storage device 102. When authenticating the worker, the terminal device 10 identifies who the worker W is in the video and refers to the data table to obtain the part numbers of the parts that the worker W brings in and collects. Then, the terminal device 10 judges whether the delivery of the parts box 34 and the collection of the collection box 35 have been properly performed depending on whether the part numbers of the carry-in entrance 31 and the collection entrance 32 match the obtained part numbers.

[0047] In the above embodiment, the part numbers are identified by reading the part number indications of the entrance 31 / return port 32 from the image captured by the camera device 20, but the part numbers may be identified by other methods. For example, the part numbers can be identified as follows. A data table is stored in the auxiliary storage device 102 in advance, which records the positions of the entrance 31 and return port 32 of each parts shelf 30 in the image captured by the camera device 20 and the correspondence between the part numbers. When the terminal device 10 confirms the delivery of the parts box 34 or the collection of the return box 35, the terminal device 10 detects the positions of the entrance 31 / return port 32 where the delivery or collection was performed from the image captured by the camera device 20. The terminal device 10 then refers to the data table to identify the part numbers of the entrance 31 / return port 32.

[0048] In the above embodiment, the worker W is notified of the result of the determination as to whether the parts box 34 has been properly delivered / collected through the notification device 21, but the result of the determination may be recorded without notification. Even in such a case, it is possible to check for erroneous delivery or collection after the fact.

[0049] The work support system in the above embodiment judges whether or not both the delivery of the parts box 34 and the collection of the collection box 35 have been performed appropriately. The work support system may be configured to judge whether or not only the delivery of the parts box 34 has been performed appropriately. In that case, the processes of S200 to S240 in FIG. 5 are omitted. [Explanation of symbols]

[0050] 10 Terminal Equipment 101 Main storage 102 Auxiliary storage device 103 CPU 104 Interface device 105 Display device 106 Input Devices B1 Bus B2 Communication Line 20 Imaging Equipment 21 Notification device 30 Parts Shelf 31 Loading entrance 32 Collection port 33 Tags 34 Parts Box 35 Collection Box 36 Tags C1~C9 Skeleton coordinates

Claims

1. A system for supporting a part loading operation in which a part box containing parts is loaded into a part-specific loading entrance provided on a part shelf, comprising: A processing unit and an imaging device that images the component shelf, The processing unit includes: a determination process of extracting a skeleton coordinate of a worker in the image captured by the imaging device, and determining whether or not the worker has carried the parts box into the entrance based on time-series data of the skeleton coordinate; a verification process for verifying, when it is determined in the determination process that the parts box has been carried in, the carried-in parts box and the carry-in entrance through which the parts box was carried in; This is a work support system.

2. The parts box is provided with a label indicating the part number of the parts, The processing unit reads the indication of the parts box brought into the entrance from the image captured by the imaging device and performs the matching process. The work support system according to claim 1 .

3. The entrance is marked with a label indicating the part number of the part to be carried in, The processing unit reads the display of the part number of the entrance through which the part box is carried in from the image captured by the imaging device, and performs the matching process. The work support system according to claim 1 .

4. The work support system according to claim 1 , further comprising a notification device for notifying the worker of the result of the collation process.

5. The work support system according to claim 1 , wherein the judgment process is performed using a learning model learned by machine learning.

Images