Picking support system

Abstract

To provide a picking support system capable of accurately determining whether a worker has picked up a correct article from a predetermined area.SOLUTION: There is provided a picking support system that determines whether a worker has picked up a predetermined article from multiple articles held in a predetermined area, including: a camera that captures an image of a grasped article being grasped by a worker; a grasped article estimation unit that estimates the grasped article being grasped by the worker based on image data captured by the camera and a plurality of articles stored in advance; a grasped article determination unit that determines whether or not an estimated grasped article matches a predetermined article to be picked up; and a notification unit that notifies the worker when the grasped article does not match the predetermined article.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a picking support system that supports a picking operation when grasping and removing an item such as a material or a part from a predetermined area. [Background technology]

[0002] Patent Document 1 discloses a picking system for preventing the wrong item from being picked up when picking up a desired item from a cabinet storing items such as materials and parts. The system in Patent Document 1 includes a cabinet having multiple areas for holding items, a support system including a camera sensor that captures images of the multiple areas and a monitoring device that monitors the picking operation based on a signal from the camera sensor, and a picking management system that manages the picking operation.

[0003] In the system of Patent Document 1, a picking management system provides a display unit in each area of ​​a cabinet, and the display unit lights up to notify the worker of the area holding the next item to be picked, thereby encouraging the worker to pick the correct item. The support system also determines whether the worker has inserted their hand into the correct area among multiple areas of the cabinet. Specifically, the support system determines whether the frontage of the area displayed by the picking management system corresponds to the position of the worker's hand performing the picking. To make this determination, a monitoring device first measures position data by tracking the position of the picker's hand in a plane based on planar coordinates based on image data. Then, the depth of the hand's penetration at the hand position based on the position data is measured. This determines whether the hand has been inserted into the frontage of the area displayed by the picking management system. If the worker's hand is inserted into an area different from the displayed area, the support system is configured to notify the worker by outputting an abnormal signal or the like. Patent Document 2 discloses a picking support device that detects the movement of a screen caused by a worker touching the screen when picking out an item, and if the movement is detected at a position away from the item box containing the item to be picked, notifies the worker by turning on a warning light or emitting a warning sound, etc. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-48052 A [Patent Document 2] International Publication No. 2018 / 47355 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the system of Patent Document 1, based on the position data of the worker's hand and the data of the worker's hand penetration depth, the system determines whether the worker's hand is inserted into the correct area and notifies the worker, thereby preventing the worker from picking the wrong item. However, while the device of Patent Document 1 can determine whether the worker has inserted his or her hand into the correct area of ​​the cabinet, it is unclear whether the worker actually picked up the item held in that area after that. Therefore, for example, if the worker inserts his or her hand into that area to organize the items so that the next worker can easily use them, the system may erroneously determine that the worker has correctly picked up the item even though he or she did not actually pick up the item that should have been picked up.

[0006] Furthermore, the device described in Patent Document 1 requires a camera to be installed on or next to the cabinet to capture the hands of a worker standing in front of the cabinet. Furthermore, a depth camera must be installed to detect the depth of the hand's entry, and a traffic flow that does not interfere with the camera must be ensured. Similarly, the device described in Patent Document 2 requires a screen to be placed when the worker removes an item from the item box, and a projector to display instructions to the worker on the screen. These devices must be positioned so as not to interfere with the worker or peripheral equipment. Furthermore, the device described in Patent Document 1 requires reference coordinates based on image data to determine whether the worker has actually inserted their hand relative to the depth information of the hand relative to the cabinet area. However, if the cabinet's position changes, the reference coordinates must be reset. Thus, there is room for improvement in systems that determine whether a worker has picked up the correct item from multiple items stored in a specified area.

[0007] This invention has been made with an eye on the above-mentioned technical problems, and aims to provide a picking support system that can accurately determine whether a worker has picked up the correct item from a specified area. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the present invention provides a picking support system that determines whether a worker has picked up a specified item from a plurality of items held in a specified area, and is characterized by comprising: a camera that captures an image of the held item held by the worker; a held item estimation unit that estimates the held item held by the worker based on image data captured by the camera and the plurality of items that have been stored in advance; a held item determination unit that determines whether the estimated held item matches the specified item to be picked up; and a notification unit that notifies the worker if the held item does not match the specified item. [Effects of the Invention]

[0009] In an embodiment of the present invention, a picking support system captures an image of an item being held by a worker and determines whether the item being held is a predetermined item that the worker should pick up. If the item being held is different from the predetermined item, the system notifies the worker. That is, by confirming the item actually picked up by the worker, it is possible to reliably determine whether the worker has inserted their hand into the predetermined area for a purpose other than picking up an item, such as organizing items. Furthermore, by determining whether the item actually picked up by the worker matches the predetermined item, it is possible to accurately determine whether the worker has picked up the predetermined item. [Brief explanation of the drawings]

[0010] [Figure 1] 1A and 1B are explanatory diagrams for explaining the entire picking support system according to an embodiment of the present invention, in which FIG. 1A shows an overall view of the picking support system, and FIG. 1B is an explanatory diagram for explaining a cabinet and a display unit. [Figure 2] 1 is a block diagram illustrating a functional configuration of a picking support system according to an embodiment of the present invention. [Figure 3]4 is a flowchart illustrating an example of control executed by the picking support system according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present invention will be described below based on the embodiments shown in the drawings. Note that the embodiments described below are merely examples of specific embodiments of the present invention, and are not intended to limit the present invention.

[0012] FIG. 1(a) shows an overall view of a picking support system 1 according to an embodiment of the present invention. The picking support system 1 according to the embodiment of the present invention is applied to a cabinet 2, which is a work shelf for storing vehicle assembly parts and the like in a vehicle factory or the like. The picking support system 1 is configured to determine whether a worker 4 has correctly picked up a specific item 5 from a specific storage section 3 that stores the specific item 5 that the worker 4 is to pick up next, among multiple storage sections 3 in the cabinet 2 as shown in FIG. 1(b). Each storage section 3 in the cabinet 2 is assigned an identification number corresponding to the stored item 5. The picking support system 1 includes a camera 6, an instruction system 7, and an inspection system 8 as components for making the determination. The cabinet 2 corresponds to a specific area in the embodiment of the present invention.

[0013] The picking support system 1 may be used not only in vehicle factories but also in other factories where distribution, production, etc. For example, in distribution, the system may be used in warehouses, distribution centers, and delivery centers to pick up ordered products from designated shelves based on information about the products. Alternatively, in the production field, the system may be used in factories to pick up required materials or parts from designated shelves.

[0014] The camera 6 is configured to capture images of the multiple storage compartments 3 in the cabinet 2 and is attached to the worker 4. The camera 6 is attached to the worker 4 so as to capture images in the direction the worker 4 is looking. For example, as shown in FIG. 1( a), the camera 6 may be attached to the worker 4's ear with the capturing direction facing forward, or may be attached like glasses on the ear with the lens positioned between the worker 4's eyebrows. The camera 6 may be configured similarly to a conventionally known camera 6, such as an RGB camera, and is configured to acquire and output captured images and videos as image data. The camera 6 captures at least the state of the worker 4's hand when the worker 4 inserts his / her hand into the storage compartment 3 of the cabinet 2 and when the worker 4 removes his / her hand. The captured image data is then output to the inspection system 8. The camera 6 is also configured to temporarily store captured video data.

[0015] The instruction system 7 is configured to detect whether the worker 4 has inserted his / her hand into a predetermined storage section 3 and to notify the worker 4 of the next item 5 to be picked up. The instruction system 7 notifies the worker 4 so that the worker 4 can identify the predetermined storage section 3 containing the next item 5 to be picked up from among the multiple storage sections 3 of the cabinet 2. Such a device may, for example, be RFID (Radio Frequency Identification) using wireless data communication. For example, as shown in FIG. 2 , the instruction system 7 includes: an RFID tag 9 attached to each of the multiple storage sections 3 and having an IC chip containing information about the stored item 5; an RFID reader 10 for transmitting and receiving information to and from the RFID tag 9; a passage detection processing device 11 for processing the information received by the RFID reader 10; and a display unit 12 for indicating the predetermined storage section 3 containing the next item 5 to be picked up by the worker 4. When the instruction system 7 detects that the worker 4 has inserted his / her hand into the predetermined storage section 3, it outputs a detection signal to the inspection system 8. Thereafter, the display unit 12 attached as shown in FIG. 1(b) is turned on in the storage unit 3 that stores the specified item 5 that the worker 4 should pick up next.

[0016] The notification to the worker 4 by the instruction system 7 is not limited to this configuration, and may be configured to notify the worker 4 of the storage unit 3 that stores the item 5 to be taken out by displaying letters, numbers, symbols, etc. on the display unit 12 provided in the storage unit 3. Furthermore, when the instruction system 7 detects that the worker 4 has inserted his / her hand into a storage unit 3 in the cabinet 2, it turns off the display unit 12 of the specific storage unit 3 into which the worker 4 has inserted his / her hand. Alternatively, the display unit 12 turns off when the worker 4 presses a specific switch provided on the cabinet 2 or the like. Then, it outputs to the inspection system 8 that the hand has been inserted into the specific storage unit 3.

[0017] Based on the image data captured by the camera 6 and pre-stored data, the inspection system 8 determines that the item 5 picked up by the worker 4 is a predetermined item 5. To achieve this, the inspection system 8 is configured to include a passage detection signal receiving unit 13, an image data management unit 14, a worker detection unit 15, a held item determination unit 16, and an abnormality notification unit 17, as shown in FIG.

[0018] The passage detection signal receiving unit 13 receives a signal determined by the instruction system 7. As described above, the instruction system 7 detects that the hand of the worker 4 has been inserted into a predetermined storage unit 3 that stores the item 5 to be picked up. The passage detection signal receiving unit 13 receives a detection signal output from the instruction system 7 when it detects that the hand of the worker 4 has been inserted into the storage unit 3.

[0019] The image data management unit 14 instructs the camera 6 to send captured image data, etc. The camera 6 is attached to the worker 4 so as to capture what is in front of the worker 4's line of sight, and always captures an image when the worker 4 grasps an item 5 from the cabinet 2. The image data management unit 14 manages the image data captured by the camera 6 when the passage detection signal receiving unit 13 receives a detection signal that the worker 4 has inserted his / her hand into the storage unit 3 of the cabinet 2. In other words, the image data management unit 14 instructs the camera 6 to send image data immediately before the worker 4 inserts his / her hand into the storage unit 3 and image data immediately after the worker 4 removes his / her hand from the storage unit 3. The image data management unit 14 then manages the received image data and outputs it to the worker detection unit 15.

[0020] The worker detection unit 15 detects the hands of the worker 4 from the image data managed by the image data management unit 14. The worker detection unit 15 detects the hands of the worker 4 by, for example, comparing the hands of the worker 4 captured in the image data with a pre-stored AI model of the hands. The worker detection unit 15 identifies the hands of the worker 4 from the image data by having an artificial intelligence learn and analyze data such as the hand shapes of various workers 4 and the colors of gloves worn by the workers 4 in advance. For example, the unit uses deep learning to learn the characteristics of the hands of the worker 4, such as shape, size, number, brightness, and color. The unit then identifies the hands of the worker 4 in the image data from the characteristics of the worker's hands based on the learning.

[0021] After detecting the hand of worker 4 from the acquired image data in this manner, worker detection unit 15 cuts out the area around the detected hand from the image data. In other words, worker detection unit 15 masks the portion of the image data estimated to be worker 4's hand and the remaining portion, leaving a predetermined range around the hand. After processing in this manner, the image data before worker 4 inserts his / her hand into storage unit 3 and after removing his / her hand are compared. By comparing the image data before worker 4 inserts his / her hand into storage unit 3 and after removing his / her hand, worker detection unit 15 detects that worker 4 is picking up item 5.

[0022] When it is determined that the worker 4 is picking up an item 5, the held item determination unit 16 determines that the item 5 held by the worker 4 is a predetermined item 5 to be picked up. The held item determination unit 16 compares the item 5 held by the worker 4 with AI models of multiple items 5 previously stored. Like the AI ​​model of the hand, the AI ​​model of the item 5 is also pre-learned and analyzed by an artificial intelligence (AI) based on data such as the shape, color, attached parts, rigidity, and material of multiple items 5 stored in the storage section 3 of the cabinet 2. This makes it possible to identify whether the item 5 held by the worker 4 is a predetermined item 5 even if the entire item 5 is not captured in the image data. The held item determination unit 16 compares the identification number of the identified item 5 and determines whether the item 5 picked up by the worker 4 is a predetermined item 5. The held item determination unit corresponds to the held item estimation unit and the held item determination unit in the embodiment of the present invention.

[0023] The abnormality notification unit 17 notifies the worker 4 when it is determined by the held item determination unit 16 that the item 5 picked up by the worker 4 is not the specified item 5. The abnormality notification unit 17 is configured to notify the worker 4 that the wrong item 5 has been picked up, for example, by emitting an alarm sound, changing the color of the display unit 12, or playing an audio guide.

[0024] Next, an example of control executed by the picking support system 1 configured as described above will be described. FIG. 3 shows a flowchart executed when the worker 4 picks up a predetermined item 5 from a predetermined area of ​​the cabinet 2. As shown in FIG. 3, first, in step S1, the camera 6 attached to the worker 4 starts taking pictures. The camera 6 starts taking pictures when the worker 4 presses a predetermined switch attached to the camera 6, for example. The camera 6 continues taking pictures at least until the worker 4 completes the pick-up operation and leaves the cabinet 2.

[0025] In step S2, the instruction system 7 detects that the worker 4 has inserted his / her hand into the predetermined storage section 3 in the cabinet 2. After detection, the instruction system 7 outputs a detection signal to the inspection system 8, notifying the worker 4 that he / she has inserted his / her hand into the predetermined storage section 3 in the cabinet 2. If the worker inserts his / her hand into a storage section 3 different from the predetermined storage section 3, the display on the display unit 12 does not disappear, so the worker can know that he / she has inserted his / her hand into the wrong storage section 3. In some cases, the instruction system 7 is configured to determine whether the storage section 3 into which the worker 4 has inserted his / her hand matches the predetermined storage section 3 in which the item 5 to be picked up is stored. In such a case, the instruction system 7 may be configured to notify the worker 4 when there is no match, and to temporarily end this flowchart without executing the subsequent control.

[0026] In step S3, the inspection system 8 receives a detection signal notifying that the hand of the worker 4 has been inserted into the specified storage unit 3. After receiving the detection signal, the process proceeds to step S4. In step S4, the camera 6 is instructed to send image data captured before and after the hand of the worker 4 is inserted into the specified storage unit 3.

[0027] In step S5, the inspection system 8 stores the image data output from the camera 6. Then, the process proceeds to step S6, where the hand of the worker 4 is detected from the acquired image data. In step S6, as described above, the hand of the worker 4 is identified by comparing the acquired image data with an AI model of the hand of the worker 4 generated by machine learning.

[0028] After the hand of worker 4 is identified, the process proceeds to step S7. In step S7, the image data is masked except for a predetermined range centered on the hand of worker 4. As described above, in step S7, the image data is masked except for an area that allows the hand of worker 4 and the object 5 being held by worker 4 to be identified. In other words, a predetermined range centered on the hand of worker 4 is trimmed.

[0029] After masking unnecessary portions of the image data, the process proceeds to step S8, where the item 5 picked up by the worker 4 is identified. In step S8, based on an AI model of the multiple items 5 generated in advance by machine learning, it is determined which of the multiple items 5 the item 5 picked up by the worker 4 is. Note that if it is detected in step S8 that the worker 4 is not holding anything, it is presumed that the worker 4 has inserted his or her hand into the storage unit 3 for a purpose other than picking up a specific item 5, such as tidying up the storage unit 3. In that case, the process may be configured to end this flowchart once without executing the subsequent control.

[0030] After the item 5 picked up by the worker 4 is identified, the process proceeds to step S9. In step S9, the identification number of the item 5 picked up by the worker 4 is compared with the identification number of the specified item 5 that the worker 4 is to pick up. The process then proceeds to step S10, where it is determined whether the two identification numbers match.

[0031] If the two identification numbers match and the determination in step S10 is YES, the process proceeds to step S11. Since the two identification numbers match, it means that the worker 4 has correctly picked up the specified item 5, and therefore in step S11, the inspection by the inspection system 8 is temporarily terminated.

[0032] Conversely, if the two identification numbers do not match and the determination in step S10 is NO, the process proceeds to step S12. If the two identification numbers do not match, the worker 4 has picked up an item 5 that is different from the intended item 5. Therefore, in step S12, an abnormality notification is issued to inform the worker 4 that he or she has picked up the wrong item 5. After such notification is issued, the process proceeds to step S11, and the inspection by the inspection system 8 is temporarily terminated.

[0033] In the picking support system 1 according to the embodiment of the present invention, a camera 6 attached to the worker 4 captures an image of the cabinet 2. This allows for reliable capture of the cabinet 2, the worker's hands, and other aspects. This prevents the camera 6 from being difficult to position and reduces the workspace required to install the camera 6 while ensuring the worker's movement. Furthermore, an instruction system 7 detects when the worker 4's hand has been inserted into a storage unit 3, and outputs a detection signal indicating that the worker 4's hand has been inserted into the specified storage unit 3 to an inspection system 8. In response to this detection signal, the inspection system 8 acquires image data from the camera 6 when the worker 4's hand is removed from the specified storage unit 3. Based on this image data and a pre-stored AI model, it is determined whether the item 5 picked up by the worker 4 is the specified item 5 that should have been picked up. That is, after detecting the item 5 actually picked up by the worker 4, the camera 6 compares it with the item 5 that should have been picked up to determine whether it matches. This makes it possible to reliably determine whether the worker 4 inserted their hand into the storage unit 3 for purposes other than picking up the item 5, such as organizing the items 5. Furthermore, since it is determined whether the picked up item 5 matches the predetermined item 5, it is possible to accurately determine whether the worker 4 has picked up the correct predetermined item 5. Such a determination is made based on the captured image data and an AI model generated in advance by machine learning or the like, so that a highly accurate determination can be made. [Explanation of symbols]

[0034] 1. Picking support system 2 Cabinets 4. Workers 5 Goods 6. Camera 7. Instruction System 8 Inspection System 12 Display section 13 Image data management section 16 Grasped article determination unit 17 Abnormality notification section

Claims

[Claim 1] A picking support system that determines whether a worker has picked up a predetermined item from a plurality of items held in a predetermined area, a camera that captures an image of the object being held by the worker; a grasped object estimation unit that estimates the grasped object grasped by the worker based on image data captured by the camera and the plurality of objects stored in advance; a grasped article determination unit that determines whether the estimated grasped article matches the predetermined article to be picked up; and a notification unit that notifies the worker when the held object does not match the predetermined object. A picking support system characterized by:

Images