Methods, devices, and storage media for handling dropped items

By using an item drop detection module and monitoring image analysis, the problem of whether an item drop is abnormal has been solved, achieving automated processing, saving labor costs, and improving the efficiency of item picking operations.

CN116252331BActive Publication Date: 2026-11-17JINGDONG TECH HLDG CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202310273238.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-20
Publication Date
2026-11-17
Estimated Expiration
2043-03-20

AI Technical Summary

Technical Problem

Current technology cannot distinguish whether the drop of an item is abnormal, making it impossible to determine whether human intervention is needed, thus wasting human resources.

Method used

By analyzing the signals detected by the item drop detection module and the operating position information of the picking device, combined with the analysis of monitoring images, it is determined whether an item has fallen abnormally, and whether the drop point is correct, and appropriate action is taken.

Benefits of technology

It can distinguish between abnormal and normal drops, saving labor costs, improving work efficiency, and enhancing the automation of item picking operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116252331B_ABST
    Figure CN116252331B_ABST
Patent Text Reader

Abstract

This disclosure provides a method, apparatus, and medium for handling dropped items. The method includes: receiving a detection signal sent by an item drop detection module during the process of an item picking device picking and moving an item; determining whether an item has been abnormally dropped based on the detection signal and the operating position information of the item picking device; determining the drop point location of the item based on a monitoring image if an abnormal drop is determined; and determining whether the item has fallen to the correct location and performing corresponding processing based on the drop point location and the item's order information. This disclosure can detect whether an abnormal drop has occurred, filter out normal item drops, and save labor costs, improve work efficiency, and increase the automation level of item picking operations by determining whether the item has fallen to the correct location and performing corresponding processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of control technology, and in particular to a method, apparatus, and storage medium for handling dropped items. Background Technology

[0002] With the rapid development of e-commerce, logistics, and other industries, the application of automated equipment is increasing. For example, the use of robotic arms to replace manual labor in handling (grabbing, moving, and placing) items (industrial parts, goods, boxes, etc.) is becoming increasingly common. During the process of robotic arms grasping and moving items, problems such as unstable gripping and item drops often occur due to improper gripping point selection, item vibration, or difficulty in gripping the item. Taking item picking as an example, when the robotic arm grasps an item and places it at the target location, the item will normally fall to that location. However, if the item falls before reaching its target location, it is considered an abnormal event, requiring the triggering of an exception handling mechanism and manual intervention, consuming significant labor costs. If the item frequently falls to the correct position due to the inertia of the robotic arm's gripping motion, then manual intervention is not actually necessary; manual intervention is only required when it falls to the wrong position. Currently, there is no existing technology to distinguish whether an item's fall is abnormal and thus determine whether manual intervention is required. Summary of the Invention

[0003] In view of this, one technical problem to be solved by the present invention is to provide a method, apparatus and storage medium for handling dropped items.

[0004] According to a first aspect of this disclosure, a method for handling dropped items is provided, comprising: receiving a detection signal sent by an item drop detection module during the process of an item picking device picking and moving an item; wherein the item drop detection module is used to detect whether the item has fallen; determining whether the item has fallen abnormally based on the detection signal and the operating position information of the item picking device; determining the drop point location of the item based on a monitoring image if it is determined that the item has fallen abnormally based on the monitoring image; and determining whether the item has fallen to the correct location and performing corresponding processing based on the drop point location and the order information of the item.

[0005] Optionally, determining whether the item has fallen to the correct location and performing corresponding processing based on the drop point location and the item's order information includes: determining the item's source box and destination box, as well as the first placement location area of ​​the source box and the second placement location area of ​​the destination box, based on the item's order information; determining whether the drop point location is located within the first placement location area or the second placement location area; and determining whether the item has fallen to the correct location and performing corresponding processing based on the determination result.

[0006] Optionally, determining whether the item has fallen to the correct location based on the judgment result includes: if the location of the drop point is determined to be within the first placement location area or the second placement location area, determining that the item has fallen to the correct location, wherein the correct location includes: inside the source box or the destination box.

[0007] Optionally, if it is determined that the item has fallen into the source box, the item picking device is controlled to re-grab the item from the source box; if it is determined that the item has fallen into the destination box, the item picking task is confirmed to be completed.

[0008] Optionally, determining whether the item has fallen to the correct position based on the judgment result includes: if it is determined that the falling point is not located in the first placement location area and not located in the second placement location area, determining that the item has not fallen to the correct position and triggering an alarm.

[0009] Optionally, determining the location of the item's fall based on the monitoring image when it is determined that the item has fallen abnormally includes: sending a video cache acquisition instruction to the video cache device when it is determined that the item has fallen abnormally; acquiring the monitoring image cached by the video cache device that corresponds to the abnormal fall of the item; and analyzing and processing the monitoring image using a video analysis model to determine the item's fall trajectory and the location of the fall point.

[0010] Optionally, determining whether an item has been abnormally dropped based on the detection signal and the operating position information of the item picking device includes: determining whether the item has been dropped based on the detection signal; if it is determined that the item has been dropped, determining the current operating position of the item picking device when the detection signal is received; and determining whether the item has been abnormally dropped based on the current operating position.

[0011] Optionally, determining whether the item has been abnormally dropped based on the current operating position includes: determining that the item has been abnormally dropped if it is determined based on the current operating position that the item picking device has not reached the target operating position.

[0012] Optionally, the item picking device includes: a pneumatic picking device mounted on a robotic arm; the item drop detection module includes: a pressure sensor; and the detection signal includes: a pressure detection signal sent by the pressure sensor.

[0013] According to a second aspect of this disclosure, an item drop handling device is provided, comprising: a detection signal receiving module, configured to receive a detection signal sent by an item drop detection module during the process of an item picking device picking and moving an item; wherein the item drop detection module is configured to detect whether the item has fallen; an abnormal drop judgment module, configured to determine whether the item has fallen abnormally based on the detection signal and the operating position information of the item picking device; a drop point determination module, configured to determine the drop point position of the item based on a monitoring image when it is determined that the item has fallen abnormally; and an abnormal drop handling module, configured to determine whether the item has fallen to the correct position and perform corresponding processing based on the drop point position and the order information of the item.

[0014] According to a third aspect of this disclosure, an item drop handling apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method described above based on instructions stored in the memory.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions which are executed by a processor as described above.

[0016] The disclosed method, apparatus, and storage medium for handling dropped items determine whether an item has been abnormally dropped based on the detection signal sent by the item drop detection module and the operating position information of the item picking device, and determine the drop point location of the item based on the monitoring image; based on the drop point location and order information, determine whether the item has fallen to the correct position and perform corresponding processing; it can detect whether an abnormal drop has occurred, filter out normal item drops, and save labor costs, improve work efficiency, and increase the automation level of item picking operations by determining whether the item has fallen to the correct position and performing corresponding processing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of one embodiment of the method for handling dropped items according to this disclosure;

[0019] Figure 2This is a schematic diagram of the process for determining whether an item has been abnormally dropped, according to one embodiment of the item drop handling method of this disclosure;

[0020] Figure 3 This is a flowchart illustrating the process of determining the location of an item's drop point in one embodiment of the item drop handling method according to this disclosure.

[0021] Figure 4 This is a flowchart illustrating the process of determining whether an item has fallen to the correct position and performing corresponding processing in one embodiment of the item drop handling method according to this disclosure;

[0022] Figure 5 This is a schematic diagram illustrating an application scenario of an embodiment of the item drop handling method disclosed herein;

[0023] Figure 6 This is a schematic diagram of a module of an embodiment of the item drop handling device according to the present disclosure;

[0024] Figure 7 This is a schematic diagram of a module of another embodiment of the item drop handling device according to the present disclosure. Detailed Implementation

[0025] The present disclosure will now be described more fully with reference to the accompanying drawings, which illustrate exemplary embodiments of the present disclosure. The technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present disclosure. The technical solutions of the present disclosure will be described in various aspects below with reference to the various figures and embodiments.

[0026] The terms "first" and "second" in the following text are used only to describe the differences between the two and have no other special meaning.

[0027] Figure 1 This is a flowchart illustrating one embodiment of the item drop handling method according to the present disclosure, as follows: Figure 1 As shown:

[0028] Step 101: During the process of the item picking device grabbing and moving the item, a detection signal sent by the item drop detection module is received; wherein, the item drop detection module is used to detect whether the item has fallen.

[0029] In one embodiment, the item can be cargo, industrial parts, etc. The item picking device can be of various types, such as an existing pneumatic picking device mounted on a robotic arm. The pneumatic picking device includes a suction cup gripping unit that grips the item. The item drop detection module can also be of various types, such as an air pressure sensor that detects the working air pressure of the suction cup gripping unit. The detection signal includes an air pressure detection signal sent by the air pressure sensor. The working air pressure corresponding to the suction cup unit is determined by the air pressure detection signal, and based on this working air pressure, it can be determined whether the item has fallen. For example, if the working air pressure is lower than a pressure threshold, it is determined that the item has fallen.

[0030] Step 102: Based on the detection signal and the operating position information of the item picking device, determine whether an item has fallen abnormally.

[0031] Step 103: If it is determined that the item has fallen abnormally, determine the location of the item's fall point based on the monitoring image.

[0032] Step 104: Based on the drop location and the item's order information, determine whether the item has dropped to the correct location and take appropriate action.

[0033] In one embodiment, an item drop event during the process of an item picking device picking an item includes: the item picking device successfully picking up the item, starting to move after picking up the item, the item falling, the item continuing to move after falling, and the item coming to a stop, at which point the drop event ends. Item drops can be divided into normal drops and abnormal drops. An item drop event is a normal item drop event if the item picking device picks up the item and moves it to the correct placement position; an item drop event is an abnormal drop event if it occurs at any time before the item picking device starts moving to the correct placement position.

[0034] Figure 2 This is a schematic diagram illustrating the process of determining whether an item has been abnormally dropped, according to one embodiment of the item drop handling method of this disclosure. Figure 2 As shown:

[0035] Step 201: Determine whether the item has fallen based on the detection signal.

[0036] Step 202: If it is determined that an item has fallen, determine the current operating position of the item picking device when the detection signal is received.

[0037] Step 203: Determine whether the item has fallen abnormally based on the current running location.

[0038] There are several methods to determine whether an item has been abnormally dropped based on its current operating location. For example, if the current operating location indicates that the item picking device has not reached its target operating location, this can be used to determine whether an item has been abnormally dropped.

[0039] In one embodiment, a pressure sensor detects the working air pressure of the pneumatic picking device mounted on the robotic arm in real time during the item picking and moving process, and sends air pressure detection information. When an abnormal working air pressure is determined based on the air pressure detection signal, it is determined that an item has been dropped. The current operating position of the item picking device is determined when the air pressure detection signal is received. If, based on the current operating position, it is determined that the item picking device has not reached the target operating position, i.e., the item fell before the pneumatic picking device reached the normal placement point, it is determined that the item was dropped abnormally.

[0040] Figure 3 This is a flowchart illustrating the process of determining the location of an item's drop point according to one embodiment of the item drop handling method of this disclosure, as shown below. Figure 3 As shown:

[0041] Step 301: When it is determined that an item has fallen abnormally, a video cache retrieval command is sent to the video cache device.

[0042] Step 302: Obtain the surveillance image cached by the video caching device that corresponds to the abnormal fall of the item.

[0043] Step 303: Analyze and process the surveillance images using a video analysis model to determine the trajectory and location of the falling object.

[0044] In one embodiment, the video caching device caches the surveillance video captured by the camera. When it is determined that an object has fallen abnormally, a video cache retrieval command is sent to the video caching device. The video caching device caches a certain length of surveillance video in real time. After receiving the video cache retrieval command, the video caching device continues to cache surveillance video for a certain duration so that the cached surveillance video can record the entire process of the object falling, and then the caching of surveillance video ends. The video analysis model can be a variety of machine learning models. The video analysis model performs visual analysis processing on the surveillance video images, and after analysis using various existing visual analysis algorithms, outputs the object's falling trajectory and the object's falling point.

[0045] Figure 4 This is a flowchart illustrating the process of determining whether an item has fallen to the correct position and performing corresponding processing according to one embodiment of the item drop handling method of this disclosure, as shown below. Figure 4 As shown:

[0046] Step 401: Determine the source box and destination box of the item, as well as the first placement area of ​​the source box and the second placement area of ​​the destination box, based on the order information of the item.

[0047] Step 402: Determine whether the drop point is located within the first placement area or the second placement area. Based on the determination result, determine whether the item has fallen to the correct location and perform the corresponding processing.

[0048] In one embodiment, if the drop point is determined to be within a first placement location area or a second placement location area, the item is determined to have fallen into the correct location and corresponding processing is performed. The correct location includes either the source box or the destination box. If the item is determined to have fallen into the source box, the item picking device is controlled to retrieve the item from the source box again; if the item is determined to have fallen into the destination box, the item picking task is confirmed to be complete. If the drop point is determined to be outside the first and second placement location areas, the item is determined not to have fallen into the correct location and an alarm is triggered.

[0049] In one embodiment, such as Figure 5 As shown, a pressure sensor is used to detect the working air pressure of the pneumatic picking device installed on the robotic arm and send a pressure detection signal. A camera is used to collect monitoring video to monitor the trajectory and landing point of the falling item. An alarm device is used to alert relevant personnel to intervene when an abnormal item falls and requires manual intervention. This disclosed method for handling falling items is applied in a computer (e.g., a workstation computer), and the video buffer can be installed inside the computer.

[0050] Based on the item's drop location and order information (e.g., WCS (Warehouse Control System) order information), determine whether the item has dropped into the correct location. Determine the source and destination boxes based on the order information, as well as the first and second placement areas for the source boxes and destination boxes. Furthermore, other source and destination boxes, and their placement areas, can be determined based on the order information.

[0051] like Figure 5As shown, boxes 1 and 2 are source boxes, and boxes 3, 4, and 5 are destination boxes. Based on the item's drop location, the area number corresponding to that location is calculated: 1-5 indicate the item dropped into the corresponding destination box or source box, 0 indicates it dropped outside the box, and -1 indicates an uncertain situation. If the item is confirmed to have dropped into a source box or destination box, and if it has dropped into the correct source box, the item picking device will re-grab the item from the source box; if it has dropped into the correct destination box, the picking task is considered complete, and the next picking task can proceed automatically. Both of these situations are handled automatically by the system. If the drop location is not within the first or second placement area, i.e., the item has dropped into an incorrect source box, an incorrect destination box, outside the box, or is uncertain, an alarm will be triggered to prompt relevant personnel to intervene.

[0052] The disclosed method for handling dropped items uses sensor technology to detect abnormal item drops, filtering out normal drops; it uses a video analysis model to determine the item's trajectory and location, and improves processing speed through video caching; it determines whether manual intervention is needed based on the area where the drop location is located; if the item falls into the correct source bin, it is re-grabbed; if it falls into the correct destination bin, the grabbing task is considered successful, and the next grabbing task begins; this method improves the system's automation level, saves significant labor costs, increases work efficiency, and enhances the automation of item picking operations.

[0053] In one embodiment, such as Figure 6 As shown, this disclosure provides an item drop handling device 60, including a detection signal receiving module 61, an abnormal drop judgment module 62, a drop point determination module 63, and an abnormal drop handling module 64. The detection signal receiving module 61 receives detection signals sent by the item drop detection module during the process of the item picking device grabbing and moving an item; wherein, the item drop detection module is used to detect whether an item has fallen.

[0054] The abnormal drop judgment module 62 determines whether an item has been abnormally dropped based on the detection signal and the operating position information of the item picking device. The drop point determination module 63, if it determines that an item has been abnormally dropped, determines the drop point location based on the monitoring image. The abnormal drop processing module 64, based on the drop point location and the item's order information, determines whether the item fell to the correct location and performs corresponding processing.

[0055] In one embodiment, the abnormal drop processing module 64 determines the source box and destination box of the item based on the item's order information, as well as the first placement location area of ​​the source box and the second placement location area of ​​the destination box; the abnormal drop processing module 64 determines whether the drop point is located within the first placement location area or the second placement location area, and determines whether the item has fallen to the correct location based on the determination result and performs the corresponding processing.

[0056] For example, if the abnormal drop handling module 64 determines that the drop point is within the first placement location area or the second placement location area, it determines that the item has fallen to the correct location and performs corresponding processing. The correct location includes either the source box or the destination box. If the abnormal drop handling module 64 determines that the item has fallen into the source box, it controls the item picking device to re-grab the item from the source box; if it determines that the item has fallen into the destination box, it confirms that the item picking task is complete. If the abnormal drop handling module 64 determines that the drop point is not within the first or second placement location area, it determines that the item has not fallen to the correct location and triggers an alarm.

[0057] In one embodiment, when the drop point determination module 63 determines that an item has fallen abnormally, it sends a video cache acquisition instruction to the video cache device. The drop point determination module 63 acquires the monitoring image cached by the video cache device that corresponds to the abnormal drop of the item, and uses a video analysis model to analyze and process the monitoring image to determine the item's drop trajectory and drop point location.

[0058] The abnormal drop determination module 62 determines whether an item has been dropped based on the detection signal. If it determines that an item has been dropped, it determines the current operating position of the item picking device when the detection signal was received. The abnormal drop determination module 62 determines whether an item has been dropped abnormally based on the current operating position. For example, if the abnormal drop determination module 62 determines that the item picking device has not reached the target operating position based on the current operating position, it determines whether an item has been dropped abnormally.

[0059] In one embodiment, such as Figure 7 As shown, this disclosure provides an item drop handling device that may include a memory 702, a processor 701, a communication interface 703, and a bus 704. The memory 702 is used to store instructions, and the processor 701 is coupled to the memory 702. The processor 701 is configured to execute the above-described item drop handling method based on the instructions stored in the memory 702.

[0060] The memory 702 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 702 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 701 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the item drop handling method disclosed herein.

[0061] In one embodiment, this disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0062] The item drop handling method, apparatus, and storage medium in the above embodiments determine whether an item has fallen abnormally based on the detection signal sent by the item drop detection module and the operating position information of the item picking device, and determine the drop point location of the item based on the monitoring image; based on the drop point location and order information, determine whether the item has fallen to the correct position and perform corresponding processing; it can detect whether an abnormal drop has occurred, filter out normal item drops, and save a lot of labor costs, improve work efficiency, and increase the automation level of item picking operations by determining whether the item has fallen to the correct position and performing corresponding processing.

[0063] The methods and systems of this disclosure can be implemented in many ways. For example, they can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0064] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for handling dropped items, comprising: During the process of the item picking device grabbing and moving an item, a detection signal sent by the item drop detection module is received; wherein, the item drop detection module is used to detect whether the item has fallen; Based on the detection signal and the operating position information of the item picking device, it is determined whether the item has fallen abnormally; If it is determined that the item was dropped abnormally, the location of the drop point is determined based on the surveillance image; Based on the drop location and the item's order information, determine whether the item has dropped to the correct location and take corresponding actions, including: Based on the order information of the item, determine the source box and destination box of the item, as well as the first placement location area of ​​the source box and the second placement location area of ​​the destination box; Determine whether the drop point is located within the first placement area or the second placement area, and based on the determination result, determine whether the item has fallen to the correct position and perform corresponding processing.

2. The method as described in claim 1, wherein determining whether the item has fallen to the correct position based on the judgment result includes: If the drop point is determined to be within the first placement location area or the second placement location area, the item is determined to have dropped to the correct location, wherein the correct location includes: inside the source box or the destination box.

3. The method of claim 2, further comprising: If it is determined that the item has fallen into the source box, the item picking device is controlled to pick up the item again from the source box; If it is confirmed that the item has fallen into the destination box, the item picking task is confirmed to be complete.

4. The method as described in claim 1, wherein determining whether the item has fallen to the correct position based on the judgment result and performing corresponding processing includes: If it is determined that the drop point is not located in the first placement area and not in the second placement area, it is determined that the item has not fallen to the correct location and an alarm is triggered.

5. The method as described in claim 1, wherein determining the location of the item's fall point based on the surveillance image when it is determined that the item has fallen abnormally includes: When it is determined that the item has been abnormally dropped, a video cache retrieval command is sent to the video cache device; Acquire the surveillance images cached by the video caching device that correspond to the abnormal drop of the item; The video analysis model is used to analyze and process the surveillance images to determine the trajectory of the falling item and the location of the falling point.

6. The method of claim 1, wherein determining whether the item has been abnormally dropped based on the detection signal and the operating position information of the item picking device comprises: Based on the detection signal, it is determined whether the item has fallen; If it is determined that the item has been dropped, determine the current operating position of the item picking device when the detection signal is received; Determine whether the item has fallen abnormally based on its current location.

7. The method of claim 6, wherein determining whether the item has fallen abnormally based on the current operating position includes: If, based on the current operating position, it is determined that the item picking device has not reached the target operating position, it is determined that the item has been abnormally dropped.

8. The method of claim 1, wherein, The item picking device includes: a pneumatic picking device mounted on a robotic arm; The item drop detection module includes: a barometric pressure sensor; the detection signal includes: a barometric pressure detection signal sent by the barometric pressure sensor.

9. A device for handling dropped items, comprising: The detection signal receiving module is used to receive the detection signal sent by the item drop detection module during the process of the item picking device grabbing and moving the item; wherein, the item drop detection module is used to detect whether the item has fallen. An abnormal drop determination module is used to determine whether the item has been abnormally dropped based on the detection signal and the operating position information of the item picking device; The drop point determination module is used to determine the drop point location of the item based on the monitoring image when it is determined that the item has fallen abnormally. An abnormal drop handling module is used to determine whether an item has fallen to the correct location and to perform corresponding processing based on the drop point location and the item's order information. This includes: determining the source box and destination box of the item, as well as a first placement location area of ​​the source box and a second placement location area of ​​the destination box, based on the item's order information; determining whether the drop point location is located within the first placement location area or the second placement location area; and determining whether the item has fallen to the correct location and performing corresponding processing based on the determination result.

10. A device for handling dropped items, comprising: Memory; And a processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1-8 based on instructions stored in the memory.

11. A computer-readable storage medium that non-transitoryly stores computer instructions, which are executed by a processor according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Intelligent pickup method, device and equipment based on article falling and storage medium

    CN115578150A