Picking methods and apparatus, storage media and electronic equipment

By combining stereoscopic and planar images, the resource and time consumption during the robotic arm picking process is reduced, and the picking efficiency and accuracy are improved, thus solving the problem of time-consuming and resource-intensive target detection in existing technologies.

CN116524176BActive Publication Date: 2026-11-17JINGDONG TECH HLDG CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310474510.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2026-11-17
Estimated Expiration
2043-04-27

AI Technical Summary

Technical Problem

In existing technologies, robotic arms use image recognition technology to detect targets during each picking operation, which leads to a waste of time and computing resources and low efficiency.

Method used

Target detection is performed by combining stereo images and planar images. The objects to be picked are determined by comparing the first planar image and the second planar image, which reduces the acquisition of stereo images and target detection and improves picking efficiency.

Benefits of technology

Save resources and time during the picking process, ensure the accuracy of picking operations, identify unchanged items to be picked by comparing planar images, and maintain the credibility of information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116524176B_ABST
    Figure CN116524176B_ABST
Patent Text Reader

Abstract

This disclosure provides a picking method, apparatus, electronic device, and storage medium, relating to the field of computer technology. The method includes: acquiring a first stereoscopic image and a first planar image of a picking area; performing target detection based on the first stereoscopic image and the first planar image to obtain a plurality of first objects to be picked; determining a first target object from the plurality of first objects to be picked, and performing a picking operation on the first target object; acquiring a second planar image of the picking area again after the picking operation is completed; determining a second object to be picked from the plurality of first objects to be picked based on the first and second planar images, and then performing a picking operation again based on the second object to be picked; wherein the position of the second object to be picked in the first planar image is the same as its position in the second planar image. This method can achieve efficient and accurate multiple picking.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a picking method and apparatus, storage medium and electronic device. Background Technology

[0002] With the increasing automation in manufacturing, logistics and other scenarios, as well as the development of bionic technology, robotic arms are being used more and more widely. For example, in the field of warehousing and logistics, robotic arms can be used to pick items.

[0003] In related technologies, target detection is usually performed based on image recognition technology to identify the target object in each picking operation, and then the target object is picked. This method of performing target detection in each picking operation consumes a lot of time and computing resources and is inefficient.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a picking method, apparatus, electronic device, and storage medium to achieve efficient and accurate multiple picking.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, a picking method is provided, comprising: acquiring a first stereoscopic image and a first planar image of a picking area; performing target detection based on the first stereoscopic image and the first planar image to obtain a plurality of first objects to be picked; determining a first target object from the plurality of first objects to be picked, and performing a picking operation on the first target object; acquiring a second planar image of the picking area again after the picking operation is completed; determining a second object to be picked from the plurality of first objects to be picked based on the first planar image and the second planar image, and then performing a picking operation again based on the second object to be picked; wherein the position of the second object to be picked in the first planar image is the same as the position in the second planar image.

[0008] In one embodiment of this disclosure, determining a second object to be picked from a plurality of first objects to be picked based on a first planar image and a second planar image includes: determining a change region between the first planar image and the second planar image; determining the first objects to be picked contained in the change region in the first planar image as objects to be filtered; and removing the objects to be filtered from the plurality of first objects to be picked to obtain the second object to be picked.

[0009] In one embodiment of this disclosure, determining the change region between a first planar image and a second planar image includes: determining a first interpolation of the first planar image and a second interpolation of the second planar image; determining change map data corresponding to the first planar image and the second planar image based on the first interpolation and the second interpolation; and determining the change region based on the change map data.

[0010] In one embodiment of this disclosure, the method further includes: if no second object to be picked is obtained after removing the object to be filtered from a plurality of first objects to be picked, then a second stereoscopic image of the picking area is acquired again; target detection is performed based on the second stereoscopic image and the second planar image to obtain a third object to be picked, and then the picking operation is performed again based on the third object to be picked.

[0011] In one embodiment of this disclosure, the method further includes: determining the spatial picking position of each first object to be picked based on a first stereoscopic image and a first planar image; wherein performing a picking operation on a first target object includes: performing a picking operation based on the spatial picking position of the first target object; and performing a picking operation again based on a second object to be picked includes: determining a second target object from the second objects to be picked; and performing a picking operation based on the spatial picking position of the second target object.

[0012] In one embodiment of this disclosure, determining a first target object from a plurality of first objects to be picked includes: determining a first picking order of the plurality of first objects to be picked; determining a first target object based on the first picking order; and determining a second target object from a plurality of second objects to be picked includes: determining a second picking order of the second objects to be picked based on their relative order in the first picking order; and determining a second target object based on the second picking order.

[0013] In one embodiment of this disclosure, the picking operation is performed by a picking device; wherein, determining a first picking order of a plurality of first objects to be picked includes: obtaining the picking start position of the picking operation performed by the picking device; determining the distance between the spatial picking position of each first object to be picked and the picking start position; and determining the first picking order based on the distance.

[0014] According to another aspect of this disclosure, a picking device is provided, comprising: an acquisition module for acquiring a first stereoscopic image and a first planar image captured for a picking area; a detection module for performing target detection based on the first stereoscopic image and the first planar image to obtain a plurality of first objects to be picked; a picking module for determining a first target object from the plurality of first objects to be picked and performing a picking operation on the first target object; the acquisition module is further configured to acquire a second planar image captured again for the picking area after the picking operation is completed; a determination module for determining a second object to be picked from the plurality of first objects to be picked based on the first planar image and the second planar image; wherein the position of the second object to be picked in the first planar image is the same as the position in the second planar image; the picking module is further configured to perform a picking operation again based on the second object to be picked.

[0015] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the picking method described above.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the picking method described above by executing the executable instructions.

[0017] The picking method provided in the embodiments of this disclosure can first perform target detection based on a first stereoscopic image and a first planar image to accurately obtain multiple first objects to be picked. Then, a first target object is picked from the multiple first objects to be picked. After the picking is completed, a second planar image can be obtained, and a second object to be picked can be determined from the multiple first objects to be picked based on the first and second planar images, so as to perform the picking operation again. It can be seen that, on the one hand, in the whole process, after one picking is completed, the next object to be picked can be determined simply by comparing the acquired planar image with the previous planar image, without consuming resources and time to acquire stereoscopic images and perform target detection again, thus saving resources and time. On the other hand, the determined second object to be picked is the first object to be picked whose position has not changed after the previous picking operation. Therefore, the information used when picking a certain second object to be picked is still obtained from the previous target detection, thus maintaining reliability and ensuring the accuracy of the picking operation.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 A schematic diagram of an exemplary system architecture to which the picking method of embodiments of the present disclosure can be applied is shown;

[0021] Figure 2 A flowchart of a picking method according to an embodiment of the present disclosure is shown;

[0022] Figure 3 This illustrates a first object to be picked in a first planar image during a picking method according to an embodiment of the present disclosure;

[0023] Figure 4 A flowchart illustrating a picking method according to an embodiment of the present disclosure is shown, in which a picking operation is performed again based on a second object to be picked;

[0024] Figure 5 This illustration shows a flowchart of a picking method according to an embodiment of the present disclosure, in which a second object is determined from a plurality of first objects to be picked.

[0025] Figure 6 A flowchart illustrating the determination of a changing region in a picking method according to an embodiment of this disclosure is shown;

[0026] Figure 7 A block diagram of a picking device according to an embodiment of the present disclosure is shown; and

[0027] Figure 8 A structural block diagram of a picking computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0031] Figure 1 A schematic diagram of an exemplary system architecture to which the picking method of embodiments of the present disclosure can be applied is shown.

[0032] like Figure 1 As shown, the system architecture may include a server 101, a network 102, and a client 103. The network 102 serves as the medium for providing a communication link between the client 103 and the server 101. The network 102 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0033] In an exemplary embodiment, the client 103 that transmits data with the server 101 may include, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and smart wearable devices. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, and Windows systems.

[0034] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 101 can also be a server for a network platform, such as a trading platform, live streaming platform, social platform, or music platform, etc., which is not limited in this embodiment. The server can be a single server or a cluster of multiple servers, and the specific architecture of the server is not limited in this disclosure.

[0035] In an exemplary embodiment, the process by which server 101 implements the picking method may be as follows: server 101 acquires a first stereoscopic image and a first planar image for the picking area; server 101 performs target detection based on the first stereoscopic image and the first planar image to obtain a plurality of first objects to be picked; server 101 determines a first target object from the plurality of first objects to be picked and performs a picking operation on the first target object; after the picking operation is completed, server 101 acquires a second planar image for the picking area again; server 101 determines a second object to be picked from the plurality of first objects to be picked based on the first planar image and the second planar image, and then performs a picking operation again based on the second object to be picked; wherein the position of the second object to be picked in the first planar image is the same as the position in the second planar image.

[0036] In an exemplary embodiment, the client 103 may be located near a picking area containing objects to be picked (such as goods, products, etc.), and may be equipped with an image acquisition device. The client 103 may first use the image acquisition device to acquire stereoscopic and planar images of the picking area. The server 101 can then obtain the stereoscopic and planar images from the client 103 via the network 102, and begin executing the picking method provided in this disclosure. In an exemplary embodiment, the server 101 may control a picking device (such as an intelligent robot, robotic arm, etc.) to perform a picking operation on the identified first target object.

[0037] The method provided in the embodiments of this disclosure can be derived from, for example... Figure 1 The server 101 or client 103 shown is used for execution, but this disclosure is not limited thereto. For example, in some other embodiments, the client 103 may acquire stereoscopic and planar images from an image acquisition device located near the picking area via network 102 to begin executing the picking method provided in this disclosure; similarly, the client 103 may control the picking device to perform a picking operation on the identified first target object.

[0038] In addition, it should be noted that, Figure 1 The example shown is merely one application environment of the picking method provided in this disclosure. Figure 1 The number of servers 101, networks 102, and clients 103 shown is merely illustrative; any number of clients, networks, and servers can be used as needed.

[0039] To enable those skilled in the art to better understand the technical solutions of this disclosure, the following will describe in more detail the various steps of the picking method in the exemplary embodiments of this disclosure in conjunction with the accompanying drawings and examples.

[0040] Figure 2 A flowchart of a picking method according to an embodiment of the present disclosure is shown. In the following illustrative example, server 101 is used as the execution subject for illustration.

[0041] like Figure 2 As shown, the picking method provided in this embodiment may include the following steps.

[0042] Step S201: Obtain the first stereoscopic image and the first planar image for the picking area.

[0043] In this step, the picking area can contain multiple items to be picked, and can be a designated area in manufacturing, logistics, or other scenarios (such as a warehouse). In some practical applications, the picking area can contain one or more containers (bins, pallets, etc.) in which the items to be picked can be placed.

[0044] The first stereoscopic image can be an image that reflects the three-dimensional spatial coordinates of each point in the picking area. In some practical applications, a 3D (3-dimensional) camera (such as a 3D reality camera) can be used to acquire stereoscopic images. A 3D camera can also be called a depth camera, which can detect the depth of field in the shooting space. The image data acquired by the 3D camera can not only record the planar information presented by the image (such as (x,y) coordinates), but also the distance of each point in the image from the 3D camera. By combining the distance of each point from the 3D camera with its planar information, the three-dimensional spatial coordinates of each point in the stereoscopic image can be obtained.

[0045] The first planar image can be an image that reflects the two-dimensional planar coordinate information of each point in the picking area, or it can be an image acquired using a 2D (2-dimensional) camera (such as a digital camera). The first stereoscopic image and the first planar image can be acquired at the same location or at different locations.

[0046] Step S203: Target detection is performed based on the first stereoscopic image and the first planar image to obtain multiple first objects to be picked.

[0047] In this step, target detection can be performed using computer vision technology to obtain target detection results. These results can display multiple first objects to be picked in the first stereoscopic image and the first planar image.

[0048] In this disclosure, the objects mentioned (including the first object to be picked, the first target object, the second object to be picked, the second target object, and the third object to be picked) can all be complete items in the picking area, or they can all be a picking point area, where a picking point area can be considered as a part of a complete item. It should be understood that the physical meaning of "object" in different application scenarios may vary depending on the object detection algorithm, but the physical meaning of "object" should be consistent in the same application scenario.

[0049] Specifically, the process can begin by dividing the first planar image into multiple sub-regions (or contour lines). Sub-regions that may belong to the same item to be picked are then clustered to obtain a set of sub-regions corresponding to each object. This process can involve: determining corresponding points in the first 3D image and the first planar image; using the 3D spatial coordinates of each point in the first 3D image to determine the spatial coordinates of each point in the set of sub-regions corresponding to each object in the first planar image; then, using the spatial coordinates of each point, identifying continuous points to determine which sub-regions belong to the same item to be picked. This allows for the determination of information about each first item to be picked in the first 3D image (such as spatial coordinates and contour information) and in the first planar image (such as planar coordinates and contour information), thus obtaining the final target detection result. Furthermore, the depth information provided by the first 3D image can be used to screen and verify each point. For example, if the depth difference is too large or the distribution is found to be unreasonable, the sub-regions can be further segmented.

[0050] In addition, in some embodiments, the picking method may further include: determining the spatial picking position of each first object to be picked based on the first stereoscopic image and the first planar image.

[0051] The spatial picking position can be determined based on the information of the first object to be picked in the first stereo image from the target detection results. For example, the center point of the sub-region corresponding to the first object to be picked can be used as the spatial picking position, or the contour coordinate information of the sub-region corresponding to the first object to be picked can be used as the spatial picking position. Alternatively, the sub-regions corresponding to the same item can be used together as the basis for calculating the spatial picking position.

[0052] Figure 3 This illustrates a picking method according to an embodiment of the present disclosure, in which a first object to be picked is identified in a first planar image, such as... Figure 3 As shown, the first item to be picked is a complete item.

[0053] exist Figure 3 The image includes a first planar image 300 acquired for the picking area. The first planar image 300 includes the following five identified sub-regions: sub-region 301, sub-region 302, sub-region 303, sub-region 304 and sub-region 305. Sub-regions 301 and 302 correspond to the first object to be picked 306, sub-regions 303 and 304 correspond to the first object to be picked 307, and sub-region 305 corresponds to the first object to be picked 308.

[0054] Step S205: Determine the first target object from a plurality of first objects to be picked, and perform a picking operation on the first target object.

[0055] In some embodiments, step S205, "determining a first target object from a plurality of first objects to be picked", may further include: determining a first picking order of the plurality of first objects to be picked; and determining a first target object based on the first picking order.

[0056] In this process, the first object to be picked, which is currently the first in the first picking order, can be taken as the first target object. In some practical applications, during the target detection process in step S203, the detection confidence level for each first object to be picked can also be calculated, and the order of detection confidence levels from high to low can be determined as the first picking order.

[0057] In some embodiments, the picking operation may be performed by a picking device; the step of determining a first picking order of a plurality of first objects to be picked may further include: obtaining the picking start position of the picking operation performed by the picking device; determining the distance between the spatial picking position of each first object to be picked and the picking start position; and determining the first picking order based on the distance.

[0058] In this embodiment, the picking device can be located above multiple first objects to be picked. The order of distance from smallest to largest can be determined as the first picking order, so that the picking device first picks the surface objects among the multiple first objects to be picked, achieving a safe and efficient picking effect.

[0059] Furthermore, in some embodiments, when the spatial picking position of each first object to be picked is determined, the process of "performing a picking operation on the first target object" in step S205 can be: performing a picking operation based on the spatial picking position of the first target object.

[0060] The operation path for the picking equipment can be calculated based on the spatial picking position of the first target object and the picking start position of the picking equipment. For example, assuming a robotic arm for the picking equipment, the gripper in the robotic arm can contact the first target object along the trajectory of the operation path to perform picking.

[0061] Step S207: After the picking operation is completed, acquire a second planar image of the picking area.

[0062] Alternatively, a device that has previously acquired a first planar image can be used to acquire a second planar image, or a device located in the same position as the device that acquired the first planar image can be used to acquire a second planar image.

[0063] Step S209: Determine a second object to be picked from a plurality of first objects to be picked based on the first planar image and the second planar image, and then perform a picking operation again based on the second object to be picked; wherein, the position of the second object to be picked in the first planar image is the same as the position in the second planar image.

[0064] After the picking operation is completed (the first target object is picked out of the picking area), a second planar image can be captured for the picking area where the first target object has been picked out. Then, by comparing the second planar image with the first planar image, the object to be picked in the next operation (i.e., a second object to be picked) can be determined. The number of second objects to be picked can be one or more. The fact that the position of a second object to be picked is the same in both the first and second planar images indicates that the second object to be picked is the first object whose position in the first planar image did not change after the picking operation was performed on the first target object.

[0065] Figure 4 A flowchart illustrating a picking method according to an embodiment of the present disclosure, showing the picking operation performed again based on a second object to be picked, as follows: Figure 4 As shown, in some embodiments, when the spatial picking position of each first object to be picked is determined, step S209, "perform the picking operation again according to the second object to be picked", may further include the following steps.

[0066] Step S401: Determine the second target object from the second objects to be picked;

[0067] Step S403: Perform a picking operation based on the spatial picking position of the second target object.

[0068] Furthermore, in some embodiments, the step S401 of "determining the second target object from the second objects to be picked" may include: determining the second picking order of the second objects to be picked according to their relative order in the first picking order; and determining the second target object according to the second picking order.

[0069] In step S209, instead of the resource-intensive and time-consuming process of "acquiring a stereo image and performing target detection based on the stereo and planar images" to determine the second object to be picked, it is only necessary to acquire the planar image again and determine the first object to be picked whose position has not changed in the first planar image after the picking operation by comparing the first and second planar images. Then, the first object to be picked whose position has not changed is used as the second object to be picked in the new round of picking operations, and the picking operation is performed again. Since the second object to be picked is the first object to be picked whose position has not changed, the information determined in the previous target detection process (such as the spatial picking position) can be considered reliable, thus ensuring the accuracy of picking. By analogy, efficient and accurate multiple picking can be achieved.

[0070] The picking method provided in this disclosure allows for the accurate acquisition of multiple first objects to be picked based on a first stereoscopic image and a first planar image. Then, a first target object is picked from these first objects. After picking, a second planar image is acquired, and a second object to be picked is determined from the multiple first objects based on the first and second planar images, allowing for another picking operation. This process offers several advantages. First, after each picking operation, the next object to be picked can be determined simply by comparing the acquired planar image with the previous planar image, eliminating the need for acquiring stereoscopic images and re-performing object detection, thus saving resources and time. Second, the determined second object to be picked is a first object whose position has not changed since the previous picking operation. Therefore, the information used when picking a second object remains the same as that obtained from previous object detection, ensuring reliability and accuracy in the picking operation.

[0071] Figure 5 A flowchart illustrating the process of determining a second object to be picked from a plurality of first objects to be picked in a picking method according to an embodiment of the present disclosure is shown, as follows: Figure 5 As shown, in some embodiments, step S209, "determining a second object to be picked from a plurality of first objects to be picked based on a first planar image and a second planar image," may further include the following steps.

[0072] Step S501: Determine the area of ​​change between the first planar image and the second planar image.

[0073] Different methods can be selected to detect changing regions based on the actual situation.

[0074] Step S503: The first object to be picked, contained in the changing region of the first planar image, is determined as the object to be filtered.

[0075] Step S505: Remove the object to be filtered from the multiple first objects to be picked to obtain the second object to be picked.

[0076] Figure 6 A flowchart illustrating the determination of a changing region in a picking method according to an embodiment of this disclosure is shown, as follows: Figure 5 As shown, in some embodiments, step S501 may further include the following steps.

[0077] Step S601: Determine the first interpolation of the first planar image and the second interpolation of the second planar image.

[0078] Step S603: Determine the change map data corresponding to the first planar image and the second planar image based on the first interpolation and the second interpolation.

[0079] After obtaining the interpolation of two images, the change map data can be obtained by directly subtracting the two images, or the optical flow method can be used to characterize the changes between the two images and then output the "change map" as change map data.

[0080] Step S605: Determine the changed area based on the change map data.

[0081] Considering the potential disturbances between two shots, perturbation reduction processing can be applied to the change map data to obtain target change map data. The changed regions can then be determined based on this target change map data. This can be achieved by using morphological closing operations or image denoising methods to minimize subtle changes caused by perturbations in the "change map," thus reducing the perturbation. Finally, obtaining the changed regions from the "change map" can be done by directly using the change map, applying image smoothing operations, or using a Gaussian mixture model to estimate the probability of change, among other methods.

[0082] In some embodiments, the picking method may further include: if no second object is obtained after removing the object to be filtered from a plurality of first objects to be picked, then acquiring a second stereoscopic image of the picking area again; performing target detection based on the second stereoscopic image and the second planar image to obtain a third object to be picked, and then performing the picking operation again based on the third object to be picked.

[0083] After the initial picking process is completed (i.e., after the picking operation for the first target object is completed), a 2D camera can be used to acquire planar image data. Then, the newly captured planar image is compared with the first captured planar image (i.e., the first planar image) to detect areas of change. Based on the detected areas of change, affected original picking points (i.e., objects to be filtered) are filtered out to reduce the impact. If the area of ​​change is too large, for example, if it is impossible to find a first object to be picked that satisfies the condition that "its position in the first planar image is the same as its position in the second planar image," then it can be considered that the previous picking operation has completely affected the picking of the remaining objects. In this case, the picking process can be repeated according to the initial picking procedure. That is, a new stereoscopic image (i.e., the second stereoscopic image) and a new planar image can be acquired, and target detection can be performed again to determine the third object to be picked. The planar image used for re-target detection can be the planar image acquired when the area of ​​change was found to be too large, or it can be a new planar image acquired at this moment; this disclosure does not limit this.

[0084] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0085] Figure 7 A block diagram of a picking device 700 according to an embodiment of the present disclosure is shown; as follows: Figure 7 As shown, it includes:

[0086] According to another aspect of this disclosure, a picking device is provided, comprising: an acquisition module 701, configured to acquire a first stereoscopic image and a first planar image captured for a picking area; a detection module 702, configured to perform target detection based on the first stereoscopic image and the first planar image to obtain a plurality of first objects to be picked; a picking module 703, configured to determine a first target object from the plurality of first objects to be picked and perform a picking operation on the first target object; the acquisition module 701 is further configured to acquire a second planar image captured again for the picking area after the picking operation is completed; a determination module 704, configured to determine a second object to be picked from the plurality of first objects to be picked based on the first planar image and the second planar image; wherein the position of the second object to be picked in the first planar image is the same as the position in the second planar image; the picking module 703 is further configured to perform a picking operation again based on the second object to be picked.

[0087] The picking device provided in this disclosure can first perform target detection based on a first stereoscopic image and a first planar image to accurately obtain multiple first objects to be picked. Then, the first target object among the multiple first objects to be picked is picked. After the picking is completed, a second planar image can be acquired, and a second object to be picked can be determined from the multiple first objects to be picked based on the first and second planar images, so as to perform the picking operation again. It can be seen that, on the one hand, in the whole process, after one picking is completed, the next object to be picked can be determined simply by comparing the acquired planar image with the previous planar image, without consuming resources and time to acquire stereoscopic images and perform target detection again, thus saving resources and time. On the other hand, the determined second object to be picked is the first object to be picked whose position has not changed after the previous picking operation. Therefore, the information used when picking a certain second object to be picked is still obtained from the previous target detection, thus maintaining its reliability and ensuring the accuracy of the picking operation.

[0088] In some embodiments, the determining module 704 determines a second object to be picked from a plurality of first objects to be picked based on a first planar image and a second planar image, including: determining a change region between the first planar image and the second planar image; determining the first objects to be picked contained in the change region in the first planar image as objects to be filtered; removing the objects to be filtered from the plurality of first objects to be picked to obtain the second object to be picked.

[0089] In some embodiments, the determining module 704 determines the change region between the first planar image and the second planar image, including: determining a first interpolation of the first planar image and a second interpolation of the second planar image; determining change map data corresponding to the first planar image and the second planar image based on the first interpolation and the second interpolation; and determining the change region based on the change map data.

[0090] In some embodiments, if the determining module 704 removes the object to be filtered from a plurality of first objects to be picked but does not obtain a second object to be picked, the acquiring module 701 is further configured to acquire a second stereoscopic image of the picking area again; the detection module 702 is further configured to perform target detection based on the second stereoscopic image and the second planar image to obtain a third object to be picked; and the picking module 703 is further configured to perform a picking operation again based on the third object to be picked.

[0091] In some embodiments, the determining module 704 is further configured to determine the spatial picking position of each first object to be picked based on the first stereoscopic image and the first planar image; wherein the picking module 703 performs a picking operation on the first target object, including: performing a picking operation based on the spatial picking position of the first target object; and the picking module 703 performs a picking operation again based on the second object to be picked, including: determining the second target object from the second object to be picked; and performing a picking operation based on the spatial picking position of the second target object.

[0092] In some embodiments, the determining module 704 determines a first target object from a plurality of first objects to be picked, including: determining a first picking order of the plurality of first objects to be picked; determining the first target object according to the first picking order; and the determining module 704 determines a second target object from a plurality of second objects to be picked, including: determining a second picking order of the second objects to be picked according to their relative order in the first picking order; and determining the second target object according to the second picking order.

[0093] In some embodiments, the picking operation is performed by a picking device; wherein, the determining module 704 determines a first picking order of a plurality of first objects to be picked, including: obtaining the picking start position of the picking operation performed by the picking device; determining the distance between the spatial picking position of each first object to be picked and the picking start position; and determining the first picking order based on the distance.

[0094] Figure 7 Other aspects of the embodiments can be found in the other embodiments described above.

[0095] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0096] Figure 8 This diagram illustrates a structural block diagram of a picking computer device according to an embodiment of the present disclosure. It should be noted that the illustrated electronic device is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention.

[0097] The following reference Figure 8 To describe an electronic device 800 according to this embodiment of the present invention. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0098] like Figure 8As shown, the electronic device 800 is manifested in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including storage unit 820 and processing unit 810).

[0099] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 810 can perform actions such as... Figure 2 The method shown.

[0100] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include a read-only memory (ROM) 8203.

[0101] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0102] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0103] Electronic device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0104] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0105] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0106] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0109] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0110] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0111] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0112] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0113] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0114] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A picking method, characterized in that, include: Acquire a first stereoscopic image and a first planar image of the picking area; Target detection is performed based on the first stereoscopic image and the first planar image to obtain multiple first objects to be picked. A first target object is determined from the plurality of first objects to be picked, and a picking operation is performed on the first target object; After the picking operation is completed, a second planar image of the picking area is acquired again. A second object to be picked is determined from the plurality of first objects to be picked based on the first planar image and the second planar image, and then a picking operation is performed again based on the second object to be picked; wherein the position of the second object to be picked in the first planar image is the same as its position in the second planar image; the step of performing the picking operation again based on the second object to be picked includes: determining a second target object from the second objects to be picked; performing a picking operation based on the spatial picking position of the second target object, the spatial picking position of the second target object being determined based on the first stereoscopic image and the first planar image; The process of determining a second object to be picked from the plurality of first objects to be picked based on the first planar image and the second planar image includes: Determine the changing region between the first planar image and the second planar image; identify the first objects to be picked contained in the changing region of the first planar image as objects to be filtered; remove the objects to be filtered from the plurality of first objects to be picked to obtain the second objects to be picked; Determining the region of change between the first planar image and the second planar image includes: Determine a first interpolation value for the first planar image and a second interpolation value for the second planar image; determine change map data corresponding to the first planar image and the second planar image based on the first interpolation value and the second interpolation value; determine the change region based on the change map data.

2. The method of claim 1, wherein, The method further includes: If no second object is obtained after removing the object to be filtered from the plurality of first objects to be picked, then a second stereoscopic image of the picking area is acquired again. Target detection is performed based on the second stereoscopic image and the second planar image to obtain a third object to be picked, and then the picking operation is performed again based on the third object to be picked.

3. The method according to any of claims 1-2, characterized by, The method further includes: determining the spatial picking position of each first object to be picked based on the first stereoscopic image and the first planar image; The picking operation on the first target object includes: performing the picking operation based on the spatial picking position of the first target object.

4. The method of claim 1, wherein, Determining a first target object from the plurality of first objects to be picked includes: determining a first picking order of the plurality of first objects to be picked; and determining the first target object based on the first picking order. And, determining a second target object from the second objects to be picked includes: determining a second picking order of the second objects to be picked based on their relative order in the first picking order; and determining the second target object based on the second picking order.

5. The method of claim 4, wherein, The picking operation is performed by the picking equipment; Determining the first picking order of the plurality of first objects to be picked includes: Obtain the starting position of the picking operation performed by the picking device; Determine the distance between the spatial picking position of each first object to be picked and the picking start position; The first picking order is determined based on the distance.

6. A picking device, characterized in that include: The acquisition module is used to acquire a first stereoscopic image and a first planar image of the picking area; The detection module is used to perform target detection based on the first stereoscopic image and the first planar image to obtain multiple first objects to be picked. The picking module is used to determine a first target object from the plurality of first objects to be picked, and to perform a picking operation on the first target object; The acquisition module is also used to acquire a second planar image of the picking area after the picking operation is completed; The determining module is configured to determine a second object to be picked from the plurality of first objects to be picked based on the first planar image and the second planar image; wherein the position of the second object to be picked in the first planar image is the same as its position in the second planar image; The picking module is further configured to perform a picking operation again based on the second object to be picked, which includes: determining a second target object from the second object to be picked; and performing a picking operation based on the spatial picking position of the second target object, wherein the spatial picking position of the second target object is determined based on the first stereoscopic image and the first planar image. The determining module determines a second object to be picked from the plurality of first objects to be picked based on the first planar image and the second planar image, including: Determine the changing region between the first planar image and the second planar image; identify the first objects to be picked contained in the changing region of the first planar image as objects to be filtered; remove the objects to be filtered from the plurality of first objects to be picked to obtain the second objects to be picked; The determining module determines the region of change between the first planar image and the second planar image, including: Determine a first interpolation value for the first planar image and a second interpolation value for the second planar image; determine change map data corresponding to the first planar image and the second planar image based on the first interpolation value and the second interpolation value; determine the change region based on the change map data.

7. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the picking method as described in any one of claims 1 to 5.

8. An electronic device, comprising: include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the picking method as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Image recognition method and device, computer system and readable storage medium

    CN113780269A

  • Capturing packaging image via scanner

    US9505554B1