A contactless express delivery device based on stereovision and a control method thereof

Abstract

The application belongs to the technical field of artificial intelligence, and proposes a contactless express delivery device based on stereovision and a control method thereof. A mechanical arm is arranged outside the box body, a clamp converter connected with multiple clamps is arranged at one end of the mechanical arm away from the outer wall of the box body, then the image recognition device installed on the mechanical arm is used to recognize the express package, the mechanical arm can sort the express package on the express shelf into the box according to the recognition result of the image recognition device, and the express package in the box is sent to the hand of the pickup person after being grabbed according to the recognition result of the image recognition device, thereby solving the problem that the whole process cannot be unmanned and the delivery process cannot be zero contact in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a contactless express delivery device and its control method based on stereo vision. Background Technology

[0002] With the continuous improvement of people's living standards, the increasing number of netizens, and the growing popularity of e-commerce platforms and online payment systems, online shopping has become the primary way of purchasing. This has greatly promoted the rapid development of the logistics and express delivery industry. Every day, express delivery receiving points receive and send a large number of express packages. Therefore, the logistics industry, especially the last mile of logistics, has many problems that need to be upgraded and transformed. The main solution to this problem is to use couriers to deliver or pick up goods. However, this method requires a lot of material, human and financial resources due to the large number of express delivery tasks every day.

[0003] The inventors discovered that current unmanned delivery vehicles require staff to manually input delivery information into the vehicle's system beforehand, and then manually place the items into individual compartments within the vehicle. Therefore, it is not truly unmanned, and the entire process is time-consuming and labor-intensive. Furthermore, each compartment can only hold a limited number of packages, resulting in wasted space. In addition, when delivering goods, existing unmanned delivery vehicles only open the corresponding compartment door after the pickup code is entered, requiring the recipient to manually retrieve the goods from the compartment and then manually close the door. This cannot avoid a large amount of direct contact between the recipient and the delivery vehicle during pickup, and also causes indirect contact between the recipients. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a contactless express delivery device and its control method based on stereoscopic vision. This solves the problems of existing technologies that cannot achieve fully unmanned operation and zero contact during the delivery process. At the same time, it places and records the express packages to be delivered in layers, which greatly improves the utilization rate of the cargo box in the express delivery vehicle and increases the number of express packages that can be delivered at one time.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a contactless express delivery device based on stereoscopic vision, employing the following technical solution:

[0006] A contactless express delivery device based on stereoscopic vision includes:

[0007] Box;

[0008] A robotic arm is attached to the outer wall of the housing;

[0009] A gripper converter is connected to the end of the robotic arm away from the outer wall of the housing, and the gripper converter is equipped with a variety of grippers.

[0010] An image recognition device is located at one end of the robotic arm near the gripper converter.

[0011] Furthermore, the lower end of the box is equipped with wheels, and the upper end is equipped with a box top cover.

[0012] Furthermore, a human-machine interface is also provided on the side wall of the enclosure.

[0013] Furthermore, various clamps include at least grippers and suction cups.

[0014] Furthermore, the robotic arm is a six-axis robotic arm, and the image recognition device is a real-sensing camera.

[0015] To achieve the above objectives, in a second aspect, the present invention also provides a control method for a contactless express delivery device based on stereoscopic vision, employing the following technical solution:

[0016] A control method for a contactless express delivery device based on stereoscopic vision, employing the contactless express delivery device based on stereoscopic vision as described in the first aspect; comprising:

[0017] Obtain global image information of the express delivery shelf using an image recognition device;

[0018] The grasping pose and size of the express package are determined based on global image information;

[0019] The type of clamp is determined based on the size of the express parcel, and the robotic arm is controlled to grab the express parcel from the shelf into the box according to the grabbing posture of the express parcel.

[0020] Furthermore, the image recognition device acquires global information of all parcels to be delivered on the express shelf, as well as point cloud data of the entire scene. It uses a trained volumetric grasping network model to perform grasping detection on the current scene, generates grasping poses, and selects appropriate gripper types based on the size of the parcels. The pixel coordinates of the parcels are converted into world coordinates for the robotic arm to recognize through a matrix. The robotic arm then sorts the parcels into boxes based on the world coordinate information.

[0021] Furthermore, the volume grasping network is a grasping detection point generation network, which includes a grasping mass head, an orientation head, and a width head, used to predict grasping mass, rotation, and opening width; the grasping mass head output size is 1×N. 3 The volume of the voxel is given by the quaternion representation of the orientation of the associated gripper, where each entry represents the predicted success probability of a gripper performed at the voxel center, the orientation head regresses to the quaternion representation of the orientation of the associated gripper candidate, and the wide head predicts the opening width of the gripper on each voxel.

[0022] Furthermore, firstly, a point cloud map of the entire scene is obtained. The volumetric grasping network receives the truncation symbol of the scene, which is represented by a distance function, and directly outputs the predicted grasping quality of each voxel in the queried 3D volume, as well as the associated grasping direction and opening width. Then, non-maximum suppression is applied using the grasping quality output, and invalid grasps are filtered out based on the distance function input.

[0023] Furthermore, after each express parcel is successfully picked up from the parcel shelf to the delivery box, the barcode of the parcel is scanned to generate a pickup code corresponding to the parcel, including the location of the parcel, the gripping posture and the clamp used, and the pickup code is sent to the recipient.

[0024] After the recipient enters the pickup code, the system retrieves the package information, including its location, gripping posture, and clamping device, to find the corresponding package and gripping method, and then retrieves the package.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1. In this invention, a robotic arm is installed outside the box, and a clamp converter with multiple clamps is installed at the end of the robotic arm away from the outer wall of the box. Then, an image recognition device installed on the robotic arm identifies the express package. The robotic arm can sort the express package from the express shelf to the box according to the recognition result of the image recognition device, and grab the express package in the box and deliver it to the recipient according to the recognition result of the image recognition device. This solves the problem of the inability to achieve full unmanned operation and zero contact in the delivery process in the prior art.

[0027] 2. The grasp detection point generation network utilized in this invention includes a grasp quality head, an orientation head, and a width head, used to predict grasp quality, rotation, and opening width. The grasp quality head outputs a volume of size 1×N³, where each entry represents the predicted success probability of grasping at the voxel center. The orientation head regresses to the quaternion representation of the orientation of the associated grasping candidate. The width head predicts the opening width of the gripper on each voxel, achieving accurate grasp prediction for express parcels and providing a guarantee for the fully unmanned operation of express parcels from the express warehouse or shelf to the recipient.

[0028] 3. This invention acquires global image information of the express delivery shelf through an image recognition device; determines the grasping posture and size of the express package based on the global image information; determines the type of clamp based on the size of the express package; and controls the robotic arm to grasp the express package from the express delivery shelf into the box based on the grasping posture of the express package. After each express package is successfully grasped from the express delivery shelf to the box, the barcode of the express package is scanned to generate a pickup code corresponding to the express package, the location of the express package, the grasping posture, and the clamp used. The pickup code is then sent to the recipient. When the recipient enters the pickup code, the location of the express package, the grasping posture, and the clamp used are retrieved based on the pickup code to find the corresponding express package and grasping method, and then the express package is grasped. This achieves a completely contactless process, effectively preventing direct or indirect contact between the courier and the package, between the recipient and the courier, and between the recipient and the delivery device.

[0029] 4. In this invention, the robotic arm autonomously grabs express parcels and places them into the box, then identifies the parcel information and delivers them, saving a significant amount of manpower and resources; the entire process is contactless, effectively preventing direct or indirect contact between couriers and parcels, between recipients and couriers, and between recipients and delivery devices, thus effectively preventing viral infections from contact; parcels are no longer limited to specific containers, but only need to be placed in the box, and the device can record their location, increasing the number of parcels that can be delivered each time and improving the efficiency of express delivery. Attached Figure Description

[0030] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0031] Figure 1 This is a schematic diagram of the structure of Embodiment 1 of the present invention;

[0032] Figure 2 This is a flowchart of the volume grabbing network performing grabbing detection in Embodiment 1 of the present invention;

[0033] Figure 3 This is a diagram illustrating the information storage format for the database when a package is successfully captured, as shown in Embodiment 1 of the present invention.

[0034] Figure 4 This is a diagram illustrating the information storage format for a package that is not successfully captured and saved to the database according to Embodiment 1 of the present invention.

[0035] Figure 5 This is a flowchart illustrating the overall process of express delivery as described in Embodiment 1 of the present invention.

[0036] The components include: 1. Human-machine interface; 2. Cargo box top cover; 3. Grippers; 4. Suction cups; 5. Fixture converter; 6. Image recognition device; 7. Robotic arm; and 8. Cargo box body. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0039] Example 1:

[0040] like Figure 1 As shown, this embodiment provides a contactless express delivery device based on stereo vision, including a human-computer interaction interface 1, a cargo box top cover 2, grippers 3, suction cups 4, a clamp converter 5, an image recognition device 6, a robotic arm 7, and a box body 8; the robotic arm 7 is connected to the outer wall of the box body 8; the clamp converter 5 is connected to the end of the robotic arm 7 away from the outer wall of the box body 8, and the clamp converter 5 is equipped with a variety of clamps; the image recognition device 6 is located at the end of the robotic arm 7 near the clamp converter 5.

[0041] Understandably, one end of the robotic arm 7 can be connected to the outer wall of the housing 8 via a turntable. When connected via the turntable, the robotic arm 7 can rotate 360° relative to the outer wall of the housing 8. Alternatively, one end of the robotic arm 7 can be directly fixed to the outer wall of the housing 8. The clamp converter 5 can be understood as a mounting device, such as a turntable that can mount other components. Various types of clamps are mounted at different positions on the converter 5. The selection of the clamp type based on the current gripping position and requirements is achieved through the rotation of the converter 5 itself or through the adjustment of the robotic arm. In this embodiment, the lower end of the housing 8 is equipped with wheels 9 to enable overall movement of the housing. The upper end of the housing 8... The container is equipped with a top cover 2. It is understood that the top cover 2 is connected to the upper part of the container via a controllable drive mechanism. The opening and closing of the top cover 2 can be controlled by controlling the drive mechanism according to different needs. The container 8 and other structural components constitute a type of unmanned delivery vehicle, with the container 8 serving as the cargo box. A human-machine interface 1 is also provided on the side wall of the container 8 for information input and display. Multiple grippers include at least grippers 3 and suction cups 4. The grippers 3 can be two-finger grippers or other multi-finger grippers, and the suction cups 4 are pneumatic suction cups. The robotic arm 7 can be a six-axis robotic arm. The image recognition device 6 can be a real-sensing camera, such as a RealSense D435i real-sensing camera.

[0042] The specific working process or principle of this embodiment is as follows:

[0043] The top cover 2 of the cargo box is opened, and the image recognition device 6 is invoked to obtain global information of all express parcels to be delivered on the express shelf, and point cloud data of the entire scene is obtained. The trained VGN (Volumetric Grasping Network) volumetric grasping network model is used to perform grasping detection on the current scene to generate the best grasping pose. According to the size of the express parcel, a suitable gripper is selected. Then, the pixel coordinates of the express parcel are converted into world coordinates that are recognized by the robotic arm 7 through a matrix. In this embodiment, the Robot Operating System (ROS) is used to handle all communication with the robotic arm and sensors. The coordinates are transmitted to the robotic arm through topic nodes. The robotic arm 7 sorts the express parcels into the box 8. When grasping, if it is a small bag or a small cardboard box, the gripper converter 5 can be driven to use a two-finger gripper to grasp it. If it is a large cardboard box or an irregular object that exceeds the opening width of the two-finger gripper, the suction cup 4 can be used for adsorption and grasping.

[0044] The VGN (Volumetric Grasping Network) used in this embodiment is a network with high accuracy for real-time object acquisition, detection, and grasping in cluttered scenes. The network follows a fully convolutional network (FCN) architecture. First, the perception module consists of three span convolutional layers with 16, 32, and 64 filters respectively, mapping the input volume V to a 64×53-dimensional feature map. The second part of the network consists of three convolutional layers interleaved with 2×bilinear upsampling, followed by three independent heads for predicting grasp quality, rotation, and aperture width. The grasp quality head outputs a volume of size 1×N³, where each entry Q represents the predicted success probability of grasping at the voxel center. The orientation head regresses to a quaternion representation of the orientation of the associated grasp candidate, where the quaternion is a 3D coordinate plus an angle. Finally, the width head predicts the aperture width of the gripper on each voxel. This network is trained end-to-end on ground truth mastery obtained from simulation experiments using the following loss function.

[0045]

[0046] in, q i ∈{0,1} represents target capture i Ground truth capture tags; It is the prediction width and ground truth label q Binary crossover loss between them; It is the prediction width and target width The mean square error between them.

[0047] like Figure 2 As shown, the depth-sensing camera acquires global information to obtain the point cloud map of the entire scene. The volumetric grasping network (VGN) receives the scene truncation symbol represented by the truncated signed distance function (TSDF) and directly outputs the predicted grasping quality of each voxel in the queried 3D volume, as well as the associated grasping direction and opening width. Then, non-maximum suppression (NMS) is applied using the grasping quality output, and invalid grasps are filtered out based on the TSDF input.

[0048] The actual detection process mainly includes: smoothing the grasping quality tensor using a 3D Gaussian kernel, which is beneficial for grasping in regions with high grasping quality; occluding voxels whose distance to the near surface is less than the defined finger depth; thresholding the grasping quality to mask voxels with low prediction scores, and applying NMS; using... Transform these candidate objects for crawling back to Cartesian coordinates, where, Define the voxel index to be captured; This is the formula for rigid transformation; The size of the capture voxel is the input. To determine the orientation of the voxel; The width of the opening at the voxel capture point; The opening width of the input grabber; apply more filters as needed for specific tasks.

[0049] In this embodiment, the Dex-net2.0 3D model dataset was used for training, and the FCN was optimized using the Adaptive Moment Estimation (Adam) optimizer with a learning rate of 3×10⁻⁶. -4 The learning time is 10 epochs, and the batch size is 32. Experiments show that the method can plan grasping within 10ms and remove 92% of targets in actual clutter removal experiments without explicit collision checking. Real-time capability enables closed-loop grasping planning, allowing the robot to handle disturbances, recover from errors, and provide greater robustness.

[0050] The advantages of using a network in this embodiment are: it is an end-to-end grasping synthesis method, generating 6 degrees of freedom grasping with a single forward pass; it performs well in highly chaotic scenes without explicit collision checking; it is extremely efficient when used with a GPU; based on the TSDF method, the model can be directly converted into a realistic robot setup without any additional adjustments; and it can be easily extended to different gripper geometries. The robotic arm configured in the device is a six-axis robotic arm with 6 degrees of freedom, capable of generating various suitable grasping poses for express packages, and can grasp express packages at various locations throughout the entire cargo box.

[0051] The robotic arm 7 is a six-axis robotic arm with six degrees of freedom. It can generate various suitable gripping postures for express delivery and can grab express packages from various locations within the entire cargo box.

[0052] While loading each express package onto the vehicle, the device uses the image recognition device 6 to scan the barcode information of the package to be delivered, and records the placement location, type of clamp used, and gripping posture. Simultaneously, it obtains the recipient's phone number and delivery address, generates a corresponding pickup code, and acquires the recipient's delivery address, estimates the delivery time, and sends an SMS notification to the recipient with the pickup code and estimated arrival time. The pickup code and barcode information are locked and stored in a MySQL database, and the data stored in the database is as follows: Figure 3 As shown.

[0053] Due to considerations of device convenience and ensuring its size and weight are suitable for road use, the equipped robotic arm 7 and grippers have size and weight limitations. The grippers have force feedback functionality; if the weight is within the bearing capacity, they can grasp the package; if the weight exceeds the bearing capacity, they will abandon the grasp. For the electric two-finger gripper, the maximum opening width can be set to 150mm, and the maximum weight it can bear can be set to 3kg. It is suitable for grasping cardboard boxes, with a gripping width within 150mm and a weight within 3kg. It is also suitable for grasping some bagged packages and some irregularly shaped packages within the size and weight allowable range. The suction cup 4 also has force feedback, suitable for some boxes, cartons, and some irregularly shaped packages with a flat surface. The suction cup 4 has a larger load-bearing capacity, allowing it to grasp packages weighing no more than 12kg. On-site verification showed that the combination of the gripper 3 and the suction cup 4 can grasp almost all common packages on the shelf. The only reason it cannot grasp packages is due to excessive weight, as many large packages are boxes with flat surfaces, which the suction cup 4 can handle. For packages whose size or weight does not meet the minimum grasping requirements, the device will use the image recognition device 6 to scan the barcode, generating information such as the reason for not grasping, the location of the object on the shelf, and the recipient's phone number. This information will then be fed back to staff to arrange alternative delivery methods, and the information will be stored in the database. Figure 4 As shown.

[0054] After all express parcels in the scene have been identified and detected, the robotic arm 7 will rotate 100 degrees in all directions to scan all positions of the entire express shelf for any undetected parcels. Once it is confirmed that there are no undetected parcels in any direction, the robotic arm 7 will retract to its initial position. After loading and scanning are completed, the cargo box top cover 2 will be closed, and delivery will begin. After the express parcel is delivered, the recipient will receive a text message notification and enter the obtained pickup code into the display interface of the human-machine interface 1.

[0055] The top cover 2 of the cargo box is opened, and the corresponding barcode information and location previously stored in the database are searched using the pickup code. Using a gripper and grasping posture, the robotic arm 7 grasps the express package based on the searched information and delivers it to the recipient. After successful delivery, the corresponding information for the delivered package is marked as "delivered" in the database for easy subsequent statistics, and then the next delivery can proceed. Because the grasping and detection have already been performed during the express sorting stage, and all key information has been recorded, there is no need for further detection at this stage. Instead, the previously stored information is directly retrieved, thus saving considerable time. The express package can be grasped immediately after the recipient enters the pickup code.

[0056] Example 2:

[0057] This embodiment provides a control method for a contactless express delivery device based on stereoscopic vision, employing the contactless express delivery device based on stereoscopic vision as described in Embodiment 1; including:

[0058] Obtain global image information of the express delivery shelf using an image recognition device;

[0059] The grasping pose and size of the express package are determined based on global image information;

[0060] The type of clamp is determined based on the size of the express parcel, and the robotic arm is controlled to grab the express parcel from the shelf into the box according to the grabbing posture of the express parcel.

[0061] In this embodiment, the image recognition device acquires global information of all parcels to be delivered on the express shelf, as well as point cloud data of the entire scene. It uses a trained volumetric grasping network model to perform grasping detection on the current scene, generates grasping poses, and selects appropriate fixture types based on the size of the parcels. The pixel coordinates of the parcels are converted into world coordinates for the robotic arm to recognize through a matrix. The robotic arm sorts the parcels into boxes based on the world coordinate information.

[0062] In this embodiment, the volume grasping network is a grasp detection point generation network. The grasp detection point generation network includes a grasp quality head, an orientation head, and a width head, which are used to predict grasp quality, rotation, and opening width. The grasp quality head outputs a volume of size 1×N3, where each entry represents the predicted success probability of grasping at the voxel center. The orientation head regresses to the quaternion representation of the orientation of the associated grasping candidate. The width head predicts the opening width of the gripper on each voxel.

[0063] In this embodiment, firstly, a point cloud map of the entire scene is obtained. The truncation symbol of the scene received by the volume capture network is represented by a distance function, and the predicted capture quality of each voxel in the queried 3D volume, as well as the associated capture direction and opening width, are directly output. Then, non-maximum suppression is applied using the capture quality output, and invalid captures are filtered out based on the distance function input.

[0064] In this embodiment, after each express parcel is successfully picked up from the express shelf to the box, the barcode of the express parcel is scanned to generate a pickup code corresponding to the express parcel, including the placement location, the gripping posture and the clamp used, and the pickup code is sent to the recipient.

[0065] After the recipient enters the pickup code, the system retrieves the package information, including its location, gripping posture, and clamping device, to find the corresponding package and gripping method, and then retrieves the package.

[0066] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A control method for a contactless express delivery device based on stereoscopic vision, characterized in that, This includes a contactless express delivery device based on stereoscopic vision, the device comprising: Box; A robotic arm is attached to the outer wall of the housing; A gripper converter is connected to the end of the robotic arm away from the outer wall of the housing, and the gripper converter is equipped with a variety of grippers. An image recognition device is disposed at one end of the robotic arm near the gripper converter; The control method includes: Obtain global image information of the express delivery shelf using an image recognition device; The grasping pose and size of the express package are determined based on global image information; The type of clamp is determined based on the size of the express parcel, and the robotic arm is controlled to grab the express parcel from the express shelf into the box according to the grabbing posture of the express parcel. The image recognition device acquires global information of all parcels to be delivered on the express shelf, as well as point cloud data of the entire scene. It uses a trained volumetric grasping network model to perform grasping detection on the current scene, generates grasping poses, and selects appropriate gripper types based on the size of the parcels. The pixel coordinates of the parcels are converted into world coordinates for the robotic arm to recognize through a matrix. The robotic arm sorts the parcels into boxes based on the world coordinate information. First, the point cloud map of the entire scene is obtained. The volume capture network receives the truncation symbol of the scene represented by the distance function and directly outputs the predicted capture quality of each voxel in the queried 3D volume, as well as the associated capture direction and opening width. Then, non-maximum suppression is applied using the capture quality output, and invalid captures are filtered out based on the distance function input. After each express parcel is successfully picked up from the parcel shelf to the delivery box, the barcode of the parcel is scanned to generate a pickup code corresponding to the parcel, the location of the parcel, the gripping posture and the clamp used, and the pickup code is sent to the recipient. After the recipient enters the pickup code, the system retrieves the location of the package, the gripping posture, and the clamping information based on the pickup code, finds the corresponding package and the gripping method, and then retrieves the package.

2. The control method for a contactless express delivery device based on stereoscopic vision as described in claim 1, characterized in that, The box is equipped with wheels at the bottom and a top cover at the top.

3. The control method for a contactless express delivery device based on stereoscopic vision as described in claim 1, characterized in that, A human-machine interface is also provided on the side wall of the enclosure.

4. The control method for a contactless express delivery device based on stereoscopic vision as described in claim 1, characterized in that, Multiple clamps include at least grippers and suction cups.

5. The control method for a contactless express delivery device based on stereoscopic vision as described in claim 1, characterized in that, The robotic arm is a six-axis robotic arm, and the image recognition device is a real-sensing camera.

6. The control method for a contactless express delivery device based on stereoscopic vision as described in claim 1, characterized in that, The volume grasping network is a grasping detection point generation network. This network includes a grasping mass head, an orientation head, and a width head, used to predict grasping mass, rotation, and opening width. The grasping mass head outputs a size of 1×N. 3 The volume of the voxel, where each entry represents the predicted success probability of a grasp performed at the voxel center, the orientation head regresses to a quaternion representation of the orientation of the associated grasping candidate, and the wide head predicts the opening width of the gripper on each voxel.

Images