Depth Reconstruction System for Structured Light Camera

Abstract

The present invention provides a depth reconstruction system for a structured light camera, comprising: an image acquisition module for acquiring multiple images continuously collected of a target area within a continuous time, where the images are of multiple target objects stored on a material carrier; a movement judgment module for judging whether the target objects move within the continuous time according to the multiple images, and returning to the image acquisition module when the target objects move; and a depth reconstruction module for performing depth reconstruction according to the multiple images or a target image to generate a depth image of the target area. In the present invention, each time a picking robot picks a target item from the material carrier, images of multiple target objects stored on the material carrier are collected by a depth camera within a continuous time, and it is judged according to the multiple images whether the target item moves due to collapse during this time, and depth reconstruction is only performed when there is no movement, thus avoiding the failure of depth reconstruction.

Description

Background Art

[0002] Common projector technologies include DLP (Digital Light Processing), line laser combined with MEMS galvanometer or mechanical galvanometer, etc.

[0003] Among them, with the maturity of LED light source and DLP technology, DLP projectors have developed rapidly and become a widely used projection method. In 1987, TI invented the DMD device, which enabled the application of DLP digital light processing technology in the world and promoted the rise of DLP projectors. The DMD device is a binary pulse width modulation digital optical switch and is currently the most complex optical switch device in the world. Thousands of tiny square lenses are built on a hinge structure above static random access memory to form the DMD. Each lens can turn on and off the light of a pixel. The hinge structure allows the lens to tilt between two states, +10 degrees for "on" and -10 degrees for "off".

[0004] Based on DLP, line laser combined with MEMS galvanometer or mechanical galvanometer phase-shifting structured light 3D cameras can only perform depth reconstruction on static scenes, that is, it is assumed that there should be no changes in the acquired image sequence. However, in the actual industrial picking application scenario, when a robot picks out a product, the other products in the basket will collapse and move, and there will be a situation where depth reconstruction fails at this time. When detecting items on a conveyor belt, it is also often found that the movement of the conveyor belt or the sliding of the items causes movement and leads to depth reconstruction failure. Summary of the Invention

[0005] Aiming at the defects in the prior art, the purpose of the present invention is to provide a depth reconstruction system for a structured light camera.

[0006] The depth reconstruction system for a structured light camera provided by the present invention includes the following modules:

[0007] An image acquisition module for acquiring multiple images continuously collected of a target area within a continuous time, where the images are of multiple target objects stored on a material carrier;

[0008] A movement judgment module for judging whether the target object moves within the continuous time according to the multiple images, and when the target object moves, returning to the image acquisition module or removing the moving pixels in each image to generate a target image, and triggering the depth reconstruction module when the target object does not move;

[0009] A depth reconstruction module for performing depth reconstruction on the multiple images or the target image to generate a depth image of the target area.

[0010] Preferably, the image acquisition module includes the following units:

[0011] A reference image unit for storing a reference image of a target area, where the reference image is collected by a depth camera under floodlight, and the reference image includes a plurality of target objects placed on a material carrier;

[0012] A structured light image unit for storing multiple structured light images starting from the acquisition time of the reference image; the structured light images are collected when the depth camera projects structured light;

[0013] A binarization processing unit for binarizing the reference image and the structured light images to generate multiple temporally continuous images.

[0014] Preferably, the movement judgment module includes the following units:

[0015] A pixel extraction unit for extracting multiple pixel regions of the structured light image according to the binarized structured light image;

[0016] A pixel difference unit for generating a pixel value difference corresponding to each pixel by taking the difference between each pixel in the pixel region of each structured light image and the corresponding pixel in the reference image;

[0017] A changed pixel judgment unit for comparing the pixel value difference corresponding to each pixel with a preset pixel change threshold, and when the pixel value difference is greater than the preset pixel change threshold, determining that the pixel is a changed pixel;

[0018] An image movement judgment unit for obtaining the number of the changed pixels determined by the changed pixel judgment unit after multiple executions, and when the number of the changed pixels is greater than a preset pixel number threshold, determining that there is movement and triggering an image acquisition module, otherwise triggering a depth reconstruction module.

[0019] Preferably, when the depth camera includes a light receiving sensor and a projector, the depth reconstruction module includes the following modules:

[0020] A pixel coordinate calculation unit for calculating an absolute phase value corresponding to each pixel coordinate value on the light receiving sensor according to the structured light image, obtaining calibration information generated by pre-calibration, and calculating a corresponding pixel coordinate value on the projector through the absolute phase value and the calibration information;

[0021] A depth information generation unit for determining the distance between the optical center of the light receiving sensor in the depth camera and the coordinate of each object point along the optical axis direction of the light receiving sensor according to two pixel coordinate values matched by the light receiving sensor and the projector, that is, generating the depth information of each pixel;

[0022] A depth image generation unit for performing depth reconstruction or three-dimensional reconstruction based on the depth information of each pixel to generate the depth image or point cloud;

[0023] When the depth camera includes a main camera and a secondary camera, the depth reconstruction module includes the following modules:

[0024] A pixel coordinate calculation unit for calculating the absolute phase value corresponding to each pixel coordinate value on the main camera according to the structured light image, obtaining the calibration information generated by pre-calibration, and calculating the corresponding pixel coordinate value on the secondary camera through the calibration information and the absolute phase value;

[0025] A depth information generation unit for determining the distance between the optical center of the main camera and the coordinate of each object point along the optical axis direction of the main camera according to two matching pixel coordinate values of the main camera and the secondary camera, that is, generating the depth information of each pixel;

[0026] A depth image generation unit for performing depth reconstruction or three-dimensional reconstruction based on the depth information of each pixel to generate the depth image or point cloud.

[0027] Preferably, the image movement judgment unit includes the following parts:

[0028] A pixel quantity determination part for obtaining the changed pixels in each of the structured light images determined by the changed pixel judgment unit after multiple executions, and then determining the quantity of the changed pixels;

[0029] An image movement judgment part for obtaining a preset pixel quantity threshold, comparing the quantity of the changed pixels with the pixel quantity threshold, and when the quantity of the changed pixels is greater than the pixel quantity threshold, determining that the structured light image moves relative to the reference image, and when the quantity of the changed pixels is less than or equal to the pixel quantity threshold, determining that the structured light image does not move relative to the reference image;

[0030] An image movement processing part for triggering the image acquisition module when it is determined that there is movement and triggering the depth reconstruction module when each of the structured light images does not move relative to the reference image.

[0031] Preferably, the movement judgment module includes the following units:

[0032] A pixel extraction unit for extracting a plurality of pixel regions of the structured light image according to the binarized structured light image;

[0033] A pixel difference unit for generating a pixel value difference corresponding to each pixel by differentiating each pixel in the pixel region of each of the structured light images from the corresponding pixel of the reference image;

[0034] A variable pixel determination unit, configured to compare the pixel value difference corresponding to each pixel with a preset pixel change threshold, and when the pixel value difference is greater than the preset pixel change threshold, determine that the pixel is a variable pixel;

[0035] A target image generation unit, configured to delete the variable pixels in each of the structured light images to generate the target image.

[0036] Preferably, the image acquisition module includes the following units:

[0037] A reference image unit, configured to acquire a reference image of a target area, where the reference image is acquired by a depth camera under floodlight, and the reference image includes a material carrier and a plurality of target objects placed on the material carrier;

[0038] A structured light image unit, configured to acquire the acquisition time of the reference image, and then acquire multiple structured light images before or around the acquisition time, where the structured light images are acquired when the depth camera projects structured light;

[0039] A binarization processing unit, configured to perform binarization processing on the reference image and the structured light images to generate multiple images that are continuous in time.

[0040] Preferably, the pixel value difference is generated by subtracting the pixel value of each pixel in the pixel area of each structured light image from the pixel value of the corresponding pixel in the reference image.

[0041] Preferably, when performing binarization on the structured light image, compare the pixel value of each pixel in the structured light image with the pixel value of the corresponding pixel in a preset binarization reference image. When the pixel value of a pixel in the structured light image is greater than the pixel value of the corresponding pixel in the binarization reference image, mark the binarization result of the pixel as 255, otherwise mark it as 0.

[0042] Preferably, at least one reference image and multiple structured light images are included in the multiple images that are continuous in time.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] In the present invention, every time a picking robot picks a target item from a material carrier, images of a plurality of target objects placed on the material carrier are acquired by a depth camera in continuous time. According to multiple images, it is determined whether the target item has moved due to collapse during this time, and depth reconstruction is performed based on multiple images only when there is no movement, avoiding the failure of depth reconstruction and improving the reconstruction efficiency. Description of the Drawings

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present invention will become more obvious:

[0046] Figure 1 Schematic diagram of the modules of the depth reconstruction system for a structured light camera in an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of the modules of the image acquisition module in an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the modules of the target object movement judgment module in an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the modules of the target object movement judgment module in a variant embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the modules of the depth reconstruction module in an embodiment of the present invention;

[0051] Figure 6 Schematic diagram of the modules of the pixel number determination unit in an embodiment of the present invention;

[0052] Figure 7 Schematic diagram of the structure of an item picking robot applying the depth reconstruction method in an embodiment of the present invention;

[0053] Figure 8 Flowchart of the steps of the depth reconstruction method during item picking in an embodiment of the present invention;

[0054] Figure 9 Schematic diagram of the structure of the depth reconstruction device during item picking in an embodiment of the present invention; and

[0055] Figure 10 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed implementation manners

[0056] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made. These all belong to the protection scope of the present invention.

[0057] In the description of the present invention, the claims, and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0058] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0059] The technical solutions of the present invention and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below with reference to the drawings.

[0060] Figure 1 It is a schematic diagram of the modules of the depth reconstruction system for a structured light camera in an embodiment of the present invention. As Figure 1 shown, the depth reconstruction system 100 for a structured light camera provided by the present invention includes the following modules:

[0061] An image acquisition module 101, configured to acquire multiple images continuously collected of a target area within a continuous time, where the images are of multiple target objects stored on a material carrier.

[0062] In an embodiment of the present invention, the images include a reference image and a fringe structured light image; each pixel in the reference image has image information, that is, the gray value of each pixel is greater than or equal to 0, and only the area illuminated by the fringes in the fringe structured light image has image information.

[0063] In an embodiment of the present invention, the material carrier may be a material box, a conveyor belt, a palletizing table, etc.

[0064] Figure 2 It is a schematic diagram of the modules of the image acquisition module in an embodiment of the present invention. As Figure 2 shown, the image acquisition module includes the following units:

[0065] Reference image unit 1011, which is used to store a reference image of a target area. The reference image is collected when the depth camera projects floodlight, and the reference image includes a plurality of target objects placed on a material carrier.

[0066] Structured light image unit 1012, which is used to store multiple structured light images starting from the acquisition time of the reference image. The structured light images are collected when the depth camera projects structured light.

[0067] Binarization processing unit 1013, which is used to perform binarization processing on the reference image and the structured light images to generate multiple consecutive images in time.

[0068] In an embodiment of the present invention, the depth camera includes a floodlight projector, a structured light projector, and a camera module. The reference image is collected by the camera module when the floodlight projector projects floodlight or is collected by the camera module under natural light. The structured light images are collected by the camera module when the structured light projector projects structured light.

[0069] The structured light projector is used to project stripe structured light. The stripe structured light is preferably a Gray code image. The number of structured light images is 3 to 5, and the reference image is one.

[0070] In a variant of the present invention, the image acquisition module includes the following units:

[0071] Reference image unit 1011, which is used to acquire a reference image of a target area. The reference image is collected when the depth camera projects floodlight, and the reference image includes a material carrier and a plurality of target objects placed on the material carrier.

[0072] Structured light image unit 1012, which is used to acquire the acquisition time of the reference image, and then acquire multiple structured light images before or around the acquisition time. The structured light images are collected when the depth camera projects structured light.

[0073] Binarization processing unit 1013, which is used to perform binarization processing on the reference image and the structured light images to generate multiple consecutive images in time.

[0074] In a variant of the present invention, among several consecutive images, one is a reference image and the others are structured light images.

[0075] The depth camera includes a floodlight projector, a structured light projector, and a camera module. The reference image is collected by the camera module when the floodlight projector projects floodlight or is collected by the camera module under natural light. The structured light images are collected by the camera module when the structured light projector projects structured light.

[0076] The structured light projector is used to project stripe structured light; the stripe structured light is preferably a Gray code image. The number of the structured light images is 3 to 5, and the reference image is one.

[0077] The movement judgment module 102 is configured to judge whether the target object moves within the continuous time according to the multiple images, and when the target object moves, return to the image acquisition module or remove the moving pixels in each image to generate a target image, and trigger the depth reconstruction module when the target object does not move;

[0078] Figure 3 It is a schematic diagram of the module of the target object movement judgment module in the embodiment of the present invention. As Figure 3 shown, the movement judgment module includes the following units:

[0079] The pixel extraction unit 1021 is configured to extract a plurality of pixel regions of the structured light image according to the binarized structured light image;

[0080] In the embodiment of the present invention, when binarizing the structured light image, each pixel value of the structured light image is compared with the corresponding pixel value of a pre-set binarization reference image, and a binarization result image is generated. When a certain pixel value of the structured light image is greater than the corresponding pixel value of the binarization reference image, the binarization result of the pixel is marked as 255, otherwise it is marked as 0. Then, a plurality of pixel regions of the structured light image are extracted according to the pixel points with the pixel value of 255 in the binarization result image.

[0081] The pixel difference unit 1022 is configured to perform a difference between each pixel in the pixel region of each structured light image and the corresponding pixel of the reference image to generate a pixel value difference corresponding to each pixel;

[0082] The moving pixel judgment unit 1023 is configured to compare the pixel value difference corresponding to each pixel with a pre-set pixel change threshold, and when the pixel value difference is greater than the pre-set pixel change threshold, determine that the pixel is a moving pixel;

[0083] The image movement judgment unit 1024 is configured to obtain the number of the moving pixels determined by the moving pixel judgment unit after multiple executions, and when the number of the moving pixels is greater than a pre-set pixel number threshold, determine that there is movement and trigger the image acquisition module, otherwise trigger the depth reconstruction module.

[0084] In the embodiment of the present invention, the pixel region of the structured light image is the region illuminated after the light beam is projected;

[0085] Specifically, generating a pixel value difference corresponding to each pixel by taking the difference between each pixel in the pixel region of each of the structured light images and the corresponding pixel in the reference image means subtracting the pixel value of each pixel in the pixel region of each of the structured light images from the pixel value of the corresponding pixel in the reference image to generate the pixel value difference.

[0086] The pixel change threshold can be set to 3 to 7. The preset pixel quantity threshold can be set to 5000 to 10000, or can be determined according to the total number of pixels of the structured light image, such as set to 10% to 50% of the total number of pixels of the structured light image.

[0087] Figure 6 It is a schematic diagram of the module of the pixel quantity determination unit in the embodiment of the present invention. As Figure 6 shown, the image movement judgment unit includes the following parts:

[0088] The pixel quantity determination part 10241 is used to obtain the changed pixels in each of the structured light images determined by the changed pixel judgment unit for multiple executions, and then determine the quantity of the changed pixels;

[0089] The image movement judgment part 10242 is used to obtain the preset pixel quantity threshold, compare the quantity of the changed pixels with the pixel quantity threshold, and when the quantity of the changed pixels is greater than the pixel quantity threshold, determine that the structured light image moves relative to the reference image, and when the quantity of the changed pixels is less than or equal to the pixel quantity threshold, determine that the structured light image does not move relative to the reference image;

[0090] The image movement processing part 10243 is used to trigger the image acquisition module when it is determined that there is movement and trigger the depth reconstruction module when each of the structured light images does not move relative to the reference image.

[0091] Figure 4 It is a schematic diagram of the module of the target object movement judgment module in the variant example of the present invention. As Figure 4 shown, the movement judgment module includes the following units:

[0092] The pixel extraction unit 1021 is used to extract a plurality of pixel regions of the structured light image according to the binarized structured light image;

[0093] In the embodiment of the present invention, when binarizing the structured light image, each pixel value of the structured light image is compared with the corresponding pixel value of a pre-set binarization reference image, and a binarization result image is generated. When a certain pixel value of the structured light image is greater than the corresponding pixel value of the binarization reference image, the binarization result of this pixel is marked as 255, otherwise it is marked as 0. Then, multiple pixel regions of the structured light image are extracted according to the pixel points with pixel value 255 in the binarization result image.

[0094] The pixel difference unit 1022 is configured to generate a pixel value difference corresponding to each pixel by taking the difference between each pixel in the pixel region of each structured light image and the corresponding pixel in the reference image.

[0095] The variable pixel determination unit 1023 is configured to compare the pixel value difference corresponding to each pixel with a pre-set pixel variation threshold, and when the pixel value difference is greater than the pre-set pixel variation threshold, determine that this pixel is a variable pixel.

[0096] The target image generation unit 1025 is configured to delete the variable pixels in each structured light image to generate the target image.

[0097] In a variant of the present invention, the pixel region of the structured light image is the region illuminated after the light beam is projected.

[0098] The specific operation of generating a pixel value difference corresponding to each pixel by taking the difference between each pixel in the pixel region of each structured light image and the corresponding pixel in the reference image is to subtract the pixel value of the corresponding pixel in the reference image from the pixel value of each pixel in the pixel region of each structured light image to generate the pixel value difference.

[0099] The pixel variation threshold can be set to 3 to 7.

[0100] The depth reconstruction module 103 is configured to perform depth reconstruction based on multiple of the images or the target image to generate a depth image of the target region.

[0101] Figure 5 It is a schematic diagram of the modules of the depth reconstruction module in the embodiment of the present invention. As Figure 5 shown, when the depth camera includes a light receiving sensor and a projector, that is, the depth camera adopts a monocular structured light camera, the depth reconstruction module includes the following modules:

[0102] The pixel coordinate calculation unit 1031 is configured to calculate the absolute phase value corresponding to each pixel coordinate value on the light receiving sensor according to the structured light image, obtain the calibration information generated by pre-calibration, and calculate the corresponding pixel coordinate value on the projector through the absolute phase value and the calibration information.

[0103] A depth information generation unit 1032, configured to determine the distance between the optical center of the light receiving sensor in the depth camera and each object point coordinate along the optical axis direction of the light receiving sensor according to two pixel coordinate values that match the light receiving sensor and the projector, that is, generate the depth information of each pixel;

[0104] A depth image generation unit 1033, configured to perform depth reconstruction or three-dimensional reconstruction according to the depth information of each pixel to generate the depth image or point cloud;

[0105] In an embodiment of the present invention, the calibration information is calculated and generated when calibrating the depth camera. When performing depth reconstruction, the triangulation method is adopted.

[0106] In a variant embodiment of the present invention, when the depth camera includes a main camera and a secondary camera, that is, the depth camera adopts a binocular camera, the depth reconstruction module includes the following modules:

[0107] A pixel coordinate calculation unit 1031, configured to calculate the absolute phase value corresponding to each pixel coordinate value on the main camera according to the structured light image, obtain the calibration information generated by pre-calibration, and calculate the corresponding pixel coordinate value on the secondary camera through the calibration information and the absolute phase value;

[0108] A depth information generation unit 1032, configured to determine the distance between the optical center of the main camera and each object point coordinate along the optical axis direction of the main camera according to two pixel coordinate values that match the main camera and the secondary camera, that is, generate the depth information of each pixel;

[0109] A depth image generation unit 1033, configured to perform depth reconstruction or three-dimensional reconstruction according to the depth information of each pixel to generate the depth image or point cloud.

[0110] In a variant embodiment of the present invention, the calibration information is calculated and generated when calibrating the depth camera. When performing depth reconstruction, the triangulation method is adopted.

[0111] Figure 7 It is a schematic structural diagram of an item picking robot applying the depth reconstruction method in an embodiment of the present invention, as Figure 7 shown, the item picking robot provided by the present invention further includes:

[0112] A first unit and a second unit, configured to store and / or transport materials;

[0113] A depth camera 300, whose visual scanning area at least covers the first unit where the materials are stored or transported, configured to perform visual scanning on the materials, collect the depth images of the materials, and generate the pose information and storage location of the materials according to the depth images;

[0114] The robot unit 400, which is communicatively connected to the depth camera 300, is configured to receive the pose information and the storage location, determine the placement state of the target object based on the pose and the storage location, and pick up the target object according to the placement state.

[0115] In an embodiment of the present invention, the first unit may be set as the storage unit 200;

[0116] The storage unit 200 is configured to store the materials placed in a disorderly manner, and the materials are the target objects, such as any items like metal products, boxes, etc.;

[0117] The robot unit 400, which is communicatively connected to the depth camera 300, is configured to receive the pose information and the storage location, determine the placement state of the target object based on the pose and the storage location, and after picking up the target object, transfer it to the second unit.

[0118] The second unit may be set to transport or store the picked materials, such as a support frame that facilitates the neat arrangement of items.

[0119] The second unit may also be set with a transport unit, such that the robot unit 400 can move the target object on the support frame to the transport unit.

[0120] The depth camera 300 is disposed on the camera support 500 and is not shown due to the occlusion of the cross beam of the camera support 500.

[0121] Wherein, the robot unit 400 includes a processor. When the processor is configured to execute the steps of the depth reconstruction method during the item picking by executing executable instructions, each time the picking robot picks out a target item from the material carrier, images of a plurality of target objects stored on the material carrier are collected within a continuous time by the depth camera. It is determined whether the target item has moved due to collapse during this time based on the multiple images, and depth reconstruction is only performed based on the multiple images when there is no movement, avoiding the failure of depth reconstruction and improving the reconstruction efficiency.

[0122] Figure 8 It is a flowchart of the steps of the depth reconstruction method during item picking in an embodiment of the present invention. As Figure 8 shown, the depth reconstruction method during item picking provided by the present invention includes the following steps:

[0123] Step S1: Obtain multiple images continuously collected of a target area within a continuous time, where the images are the images of a plurality of target objects stored on the material carrier;

[0124] Step S2: Determine whether the target object moves within this continuous time according to the multiple images. When the target object moves, return to Step S1 or generate a target image after removing the moving pixels in each image. When the target object does not move, trigger Step S3;

[0125] Step S3: Perform depth reconstruction based on the multiple images or the target image to generate a depth image of the target area.

[0126] In an embodiment of the present invention, a depth reconstruction device during item picking is further provided, including a processor and a memory. The memory stores executable instructions of the processor. Wherein, the processor is configured to execute the steps of the depth reconstruction method during item picking by executing the executable instructions.

[0127] As above, in this embodiment, each time the picking robot picks a target item from the material carrier, images of multiple target objects stored on the material carrier are collected within a continuous time by a depth camera. It is determined whether the target item moves due to collapse during this time according to the multiple images, and depth reconstruction is only performed based on the multiple images when there is no movement, avoiding the failure of depth reconstruction and improving the reconstruction efficiency.

[0128] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, method, or program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "platform" here.

[0129] Figure 9 is a schematic structural diagram of the depth reconstruction device during item picking in an embodiment of the present invention. Refer to the following Figure 9 to describe the electronic device 600 according to this embodiment of the present invention. Figure 9 The electronic device 600 shown is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0130] As Figure 9 shown, the electronic device 600 is presented in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0131] Among them, the storage unit stores program code, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the part of the depth reconstruction method for item picking in the above description of this specification. For example, the processing unit 610 can execute steps as shown in Figure 1 as shown.

[0132] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0133] The storage unit 620 may also include a program / utilities 6204 having a set (at least one) of program modules 6205. Such program modules 6205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0134] The bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0135] The electronic device 600 may also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, a camera, a depth camera, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface 650. And the electronic device 600 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Figure 9 not shown, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0136] In an embodiment of the present invention, a computer-readable storage medium is further provided for storing a program, and when the program is executed, it implements the steps of the depth reconstruction method during item picking. In some possible implementation manners, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above-mentioned depth reconstruction method section for item picking in this specification.

[0137] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, each time the picking robot picks a target item from the material carrier, images of multiple target objects stored on the material carrier are collected continuously in time by controlling a depth camera, and it is judged whether the target item has moved due to collapse during this time based on multiple images. Only when there is no movement, depth reconstruction is performed based on multiple images, avoiding the failure of depth reconstruction and improving the reconstruction efficiency.

[0138] Figure 10 It is a schematic structural diagram of the computer-readable storage medium in an embodiment of the present invention. Refer to Figure 10 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described. It can be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited to this. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0139] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0140] A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0141] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0142] In an embodiment of the present invention, each time a picking robot picks a target item from a material carrier, images of a plurality of target objects stored on the material carrier are collected within a continuous time period by a depth camera. It is determined whether the target item has moved due to collapse during this time based on multiple images, and depth reconstruction is performed based on multiple images only when there is no movement, avoiding the failure of depth reconstruction and improving the reconstruction efficiency.

[0143] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments may be referred to each other. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0144] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depth reconstruction system, characterized in that, It includes the following modules: An image acquisition module, configured to acquire multiple temporally consecutive images of a target area, where the images are formed by imaging multiple target objects stored on a material carrier, including multiple structured light images and a reference image acquired under floodlight; A movement judgment module, configured to judge whether the target object moves within consecutive time according to the multiple images, and when the target object moves, return to the image acquisition module or remove the moving pixels in each image to generate a target image, and trigger the depth reconstruction module when the target object does not move; A depth reconstruction module, configured to perform depth reconstruction according to the multiple images or the target image to generate a depth image of the target area; The movement judgment module includes the following units: A pixel extraction unit, configured to extract multiple pixel regions of the structured light image according to the binarized structured light image; A pixel difference unit, configured to perform a difference between each pixel in the pixel region of each structured light image and the corresponding pixel in the reference image to generate a pixel value difference corresponding to each pixel; A changed pixel judgment unit, configured to compare the pixel value difference corresponding to each pixel with a preset pixel change threshold, and when the pixel value difference is greater than the preset pixel change threshold, determine that the pixel is a changed pixel; An image movement judgment unit, configured to obtain the number of the changed pixels determined by the changed pixel judgment unit after multiple executions, and when the number of the changed pixels is greater than a preset pixel number threshold, determine that there is movement and trigger the image acquisition module, otherwise trigger the depth reconstruction module.

2. The depth reconstruction system according to claim 1, wherein The image acquisition module includes the following units: A reference image unit, configured to store a reference image of the target area, where the reference image is acquired by a depth camera under floodlight, and the reference image includes multiple target objects stored on a material carrier; A structured light image unit, configured to store multiple structured light images starting from the acquisition time of the reference image; The structured light image is acquired when the depth camera projects structured light; A binarization processing unit, configured to perform binarization processing on the reference image and the structured light image to generate multiple temporally consecutive images.

3. The depth reconstruction system according to claim 2, wherein When the depth camera includes a light receiving sensor and a projector, the depth reconstruction module includes the following modules: A pixel coordinate calculation unit, configured to calculate an absolute phase value corresponding to each pixel coordinate value on the light receiving sensor according to the structured light image, obtain calibration information generated by pre-calibration, and calculate the corresponding pixel coordinate value on the projector through the absolute phase value and the calibration information; A depth information generation unit, configured to determine the distance between the optical center of the light receiving sensor in the depth camera and the coordinate of each object point along the optical axis direction of the light receiving sensor according to two matching pixel coordinate values of the light receiving sensor and the projector, that is, generate depth information of each pixel; A depth image generation unit, configured to perform depth reconstruction or three-dimensional reconstruction according to the depth information of each pixel to generate the depth image or point cloud; When the depth camera includes a main camera and a sub-camera, the depth reconstruction module includes the following modules: A pixel coordinate calculation unit, configured to calculate, according to the structured light image, an absolute phase value corresponding to each pixel coordinate value on the main camera, obtain calibration information generated by pre-calibration, and calculate, through the calibration information and the absolute phase value, a corresponding pixel coordinate value on the secondary camera; A depth information generation unit, configured to determine, according to two pixel coordinate values matched by the main camera and the secondary camera, a distance between the optical center of the main camera and each object point coordinate along the optical axis direction of the main camera, that is, generate depth information of each pixel; A depth image generation unit, configured to perform depth reconstruction or three-dimensional reconstruction according to the depth information of each pixel to generate the depth image or point cloud.

4. The depth reconstruction system according to claim 1, wherein The image movement judgment unit includes the following parts: A pixel quantity determination part, configured to obtain the changed pixels in each of the structured light images determined by the changed pixel judgment unit through multiple executions, and further determine the quantity of the changed pixels; An image movement judgment part, configured to obtain a preset pixel quantity threshold, compare the quantity of the changed pixels with the pixel quantity threshold, and when the quantity of the changed pixels is greater than the pixel quantity threshold, determine that the structured light image moves relative to the reference image, and when the quantity of the changed pixels is less than or equal to the pixel quantity threshold, determine that the structured light image does not move relative to the reference image; An image movement processing part, configured to return and trigger the image acquisition module when it is determined that there is movement, and trigger the depth reconstruction module when each of the structured light images does not move relative to the reference image.

5. The depth reconstruction system according to claim 2, wherein The movement judgment module includes the following units: A pixel extraction unit: configured to extract multiple pixel regions in the structured light image according to the binarized structured light image; A pixel difference unit, configured to perform a difference between each pixel in the pixel region of each structured light image and the corresponding pixel of the reference image to generate a pixel value difference corresponding to each pixel; A changed pixel judgment unit, configured to compare the pixel value difference corresponding to each pixel with a preset pixel change threshold, and when the pixel value difference is greater than the preset pixel change threshold, determine that the pixel is a changed pixel; A target image generation unit, configured to delete the changed pixels in each of the structured light images to generate the target image.

6. The depth reconstruction system according to claim 1, characterized in that The image acquisition module includes the following units: A reference image unit, configured to obtain a reference image of a target area, where the reference image is collected by a depth camera under floodlight, and the reference image includes a material carrier and a plurality of target objects placed on the material carrier; A structured light image unit, configured to obtain the acquisition time of the reference image, and further obtain multiple structured light images before or around the acquisition time, where the structured light images are collected when the depth camera projects structured light; A binarization processing unit, configured to perform binarization processing on the reference image and the structured light images to generate multiple images that are continuous in time.

7. The depth reconstruction system according to claim 1 or 5, characterized in that, The pixel value difference is generated by subtracting the pixel value of each pixel in the pixel region of each structured light image from the corresponding pixel of the reference image.

8. The depth reconstruction system according to claim 2, wherein, When binarizing the structured light image, each pixel value of the structured light image is compared with the corresponding pixel value of a preset binarization reference image. When a pixel value of the structured light image is greater than the corresponding pixel value of the binarization reference image, the binarization result of this pixel is marked as 255; otherwise, it is marked as 0.

9. The depth reconstruction system according to claim 1, characterized in that Among the multiple images that are temporally continuous, at least one reference image and multiple structured light images are included.

Images