Goods Tracking Method, Device, Electronic Device and Storage Medium

Abstract

The present invention provides a goods tracking method, device, electronic device and storage medium, which relates to the field of computer technology. The method includes: obtaining a goods image of goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the goods image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm; determining second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data; determining pose information of the goods to be inventoried at the next moment based on the second point cloud data; and tracking the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment, realizing real-time tracking of the goods to be inventoried, and further realizing accurate inventory of the goods to be inventoried, thereby improving the inventory efficiency of the goods.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a method, device, electronic device, and storage medium for tracking goods. Background Art

[0002] Intelligent warehousing is a link in the logistics process. The application of intelligent warehousing ensures the speed and accuracy of data input in all aspects of goods warehouse management, enables enterprises to timely and accurately grasp the real data of inventory, and reasonably maintain and control the inventory of enterprises. As a crucial item in warehousing logistics, goods inventory has long been carried out manually. This traditional inventory method is cumbersome, error-prone, difficult to recheck, time-consuming and laborious. Moreover, sometimes in the inventory of large warehouses, the inventory personnel need to rise to a relatively high place, and high-altitude operations pose a greater threat to the inventory personnel.

[0003] With the rapid development of computer technology and automation disciplines, in order to more efficiently improve the warehouse operation efficiency, an intelligent recognition technology based on active vision is introduced into warehouse inventory to replace manual inventory. However, the current shooting method with a fixed camera position and a fixed focal length has a low recognition accuracy for small labels and objects with irregularly placed goods, resulting in low inventory efficiency. Summary of the Invention

[0004] The present invention provides a method, device, electronic device, and storage medium for tracking goods to solve the problem of low inventory efficiency in the prior art.

[0005] The present invention provides a method for tracking goods, which is applied to a robotic arm and includes:

[0006] Obtaining a goods image of goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the goods image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm;

[0007] Based on the goods image and the first point cloud data, determining second point cloud data of the goods to be inventoried;

[0008] Based on the second point cloud data, determining pose information of the goods to be inventoried at the next moment;

[0009] Based on the pose information of the goods to be inventoried at the next moment, tracking the goods to be inventoried.

[0010] According to a method for tracking goods provided by the present invention, the determining the second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data includes:

[0011] Perform object detection on the cargo image to obtain the position information of the goods to be inventoried at the current moment;

[0012] Based on the position information and the first point cloud data, determine the second point cloud data of the goods to be inventoried.

[0013] According to a cargo tracking method provided by the present invention, the performing object detection on the cargo image to obtain the position information of the goods to be inventoried at the current moment includes:

[0014] Input the cargo image into the object detection network model to obtain the position information of the goods to be inventoried at the current moment output by the object detection network model; the object detection network model is trained based on sample cargo images and label number data and is used to detect the goods to be inventoried in the cargo image.

[0015] According to a cargo tracking method provided by the present invention, the based on the position information and the first point cloud data to determine the second point cloud data of the goods to be inventoried includes:

[0016] Align the first point cloud data with the cargo image corresponding to the position information to determine the position and range of the goods to be inventoried corresponding to the first point cloud data;

[0017] Based on the position and range, determine the second point cloud data of the goods to be inventoried.

[0018] According to a cargo tracking method provided by the present invention, the based on the second point cloud data to determine the pose information of the goods to be inventoried at the next moment includes:

[0019] Based on the second point cloud data, determine the pose information of the goods to be inventoried at the current moment;

[0020] Based on the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried, determine the pose information of the goods to be inventoried at the next moment; the relative displacement is obtained based on the position information of the goods to be inventoried at the current moment and the position information at the previous moment.

[0021] According to a cargo tracking method provided by the present invention, the based on the pose information of the goods to be inventoried at the next moment to track the goods to be inventoried includes:

[0022] Based on the pose information of the goods to be inventoried at the previous moment, the current moment and the next moment, track the movement trajectory of the goods to be inventoried;

[0023] Adjust the pose of the robotic arm based on the movement trajectory of the goods to be inventoried, and control the movement of the robotic arm to track the goods to be inventoried.

[0024] The present invention also provides a goods tracking device applied to a robotic arm, including:

[0025] An acquisition module, configured to acquire a goods image of the goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the goods image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm;

[0026] A goods segmentation module, configured to determine second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data;

[0027] A pose estimation module, configured to determine pose information of the goods to be inventoried at the next moment based on the second point cloud data;

[0028] A tracking module, configured to track the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the goods tracking method as described in any one of the above.

[0030] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the goods tracking method as described in any one of the above.

[0031] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the goods tracking method as described in any one of the above.

[0032] The goods tracking method, device, electronic device, and storage medium provided by the present invention obtain a goods image of the goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the goods image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm; determine second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data to achieve segmentation of the goods to be inventoried; then determine pose information of the goods to be inventoried at the next moment according to the second point cloud data to achieve pose estimation of the goods to be inventoried; then track the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment to achieve real-time tracking of the goods to be inventoried, thereby achieving accurate inventory of the goods to be inventoried and improving the inventory efficiency of the goods. Description of the Drawings

[0033] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0034] Figure 1 is one of the schematic flowcharts of the goods tracking method provided by the present invention;

[0035] Figure 2 is the second schematic flowchart of the goods tracking method provided by the present invention;

[0036] Figure 3 is the schematic structural diagram of the goods tracking device provided by the present invention;

[0037] Figure 4 is the schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0038] To make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0039] To better understand the embodiments of the present application, the relevant knowledge of the present application will be introduced first.

[0040] Intelligent warehousing is intelligent logistics jointly realized through informatization, the Internet of Things and mechatronics, thereby reducing warehousing costs, improving operation efficiency, and enhancing warehousing management capabilities. According to the warehousing environment characteristics of goods, it can generally be divided into indoor and outdoor; among them, the indoor environment is more complex, and according to the warehousing conditions of goods, it can be roughly divided into stereoscopic shelves, general shelves, and flat areas; the outdoor environment usually includes parking areas, goods loading and unloading areas, warehouses, and goods sorting areas, etc., to support the receipt, sorting, storage and shipment of goods. The outdoor environment also includes truck parking areas, parcel sorting areas, loading and unloading equipment, and distribution vehicles, etc., to support the sorting, packing and delivery of goods. Large stereoscopic shelves are often relatively tall in total height, with moderate cargo weights, and there is a situation of shelf occlusion. General shelf environments are more common, but due to the often different sizes and weights of goods, the efficiency of pure manual hand-held inventory is relatively low.

[0041] Currently, for the detection of goods at fixed positions, the main method is through fixed-position shooting; for the inventory of goods on three-dimensional multi-layer shelves, the main method is manual barcode scanning for inventory.

[0042] Traditional shooting methods based on fixed positions and fixed focal lengths are difficult to overcome the image blurring phenomenon caused by changes in the placement positions of goods. Based on the manual inventory method, there are certain safety issues on shelves with a height of 10 meters.

[0043] Based on the above problems, the present invention proposes a goods tracking method and designs a set of dynamic tracking detection specialists to solve the problems caused by fixed camera shooting. By installing a detection system on a multi-axis robotic arm, functions such as multi-angle and large-field-of-view goods pose detection and barcode recognition are realized, improving the inventory efficiency of warehouse goods. It plays an important role in the safety, high efficiency, and low cost of warehouse inventory. As an important part of warehouse intelligence, it has strong practical value.

[0044] The following combines Figure 1 - Figure 2 to describe the goods tracking method of the present invention.

[0045] Figure 1 is one of the flow schematic diagrams of the goods tracking method provided by the present invention. As Figure 1 shown, it is applied to a robotic arm. The method includes step 101 - step 104; among them,

[0046] Step 101, obtain the goods image of the goods to be inventoried collected by the camera sensor at the current moment and the first point cloud data corresponding to the goods image collected by the lidar; the camera sensor and the lidar are installed on the robotic arm.

[0047] It should be noted that the goods tracking method provided by the present invention is applicable to the scenario of intelligent warehouse inventory of goods. The execution subject of this method can be a goods inventory device, such as a robotic arm, or a control module in the goods inventory device for executing the goods tracking method.

[0048] Specifically, the camera sensor and the lidar are installed on the robotic arm, and the camera sensor is a depth camera. The camera sensor collects the goods image of the goods to be inventoried in real time. The lidar irradiates the environment with laser beams or structured light and records the reflection data in the environment. These reflection data are used to generate point cloud data. Through the calibration parameters of the camera sensor and the lidar, the goods image and the reflection data are aligned to obtain the first point cloud data corresponding to the goods image.

[0049] Calibration parameters refer to parameters such as calibration offset calibration, response linearity, sensitivity and gain, and zero drift. Among them, there may be a positional offset between the camera sensor and lidar on the robotic arm and the mechanical structure. Therefore, offset calibration is required, which can be achieved by recording the offsets between the camera sensor and lidar and the coordinate system of the robotic arm respectively for correction in subsequent applications; the responses of the camera sensor and lidar are linear, that is, the relationship between the output value and the input quantity should be a linear relationship. The calibration process can determine the linear range and linearity of the camera sensor and lidar respectively for correction in actual applications; the sensitivity and gain indicators of the camera sensor and lidar can reflect the sensitivity of the camera sensor and lidar to the input signal. The calibration process can determine the sensitivity and gain parameters of the camera sensor and lidar to ensure accuracy and consistency in different input ranges; the camera sensor and lidar may exhibit zero drift, that is, the output value changes when there is no input. The calibration process can measure and correct the zero drift of the camera sensor and lidar to maintain accuracy and stability.

[0050] Aligning the cargo image and reflection data includes hardware calibration, internal and external parameter calibration, image alignment, coordinate system conversion, and error correction; among them, hardware calibration ensures the hardware alignment between the camera sensor and lidar, which means installing the camera sensor and lidar in the same coordinate system so that they have the same position and orientation; internal parameter calibration is to determine the internal parameters of the camera sensor, such as focal length, optical center, etc., which can be achieved by calibrating the camera sensor using a calibration board or other objects of known size; at the same time, external parameter calibration is to determine the relative position and attitude relationship between the camera sensor and lidar sensor. External parameter calibration can be performed using hand-eye calibration or a known rigid body model; when aligning the cargo image and reflection data, image alignment is often required, which can be achieved by detecting specific feature points or markers in the cargo image and matching them with the corresponding points in the depth image (the image composed of reflection data). By matching the corresponding points in the cargo image and reflection data, the corresponding relationship between them can be established; after aligning the cargo image and reflection data, coordinate system conversion is usually required to represent them in the same coordinate system, which involves converting the pixel coordinates of the reflection data into three-dimensional coordinates and corresponding the reflection data with each pixel point in the cargo image; in addition, there may be some errors during the alignment process, such as image distortion, noise of the lidar, etc. Therefore, some error corrections are required after alignment to improve the accuracy and quality of the alignment.

[0051] Step 102: Based on the cargo image and the first point cloud data, determine the second point cloud data of the goods to be inventoried.

[0052] Specifically, the first point cloud data represents the point cloud data corresponding to the cargo image collected by the lidar, including the goods to be inventoried. The second point cloud data represents only the point cloud data corresponding to the image of the goods to be inventoried, that is, the goods to be inventoried are segmented from the cargo image to obtain the point cloud data of the goods to be inventoried. According to the cargo image and the first point cloud data, the second point cloud data of the goods to be inventoried can be determined.

[0053] Step 103: Based on the second point cloud data, determine the pose information of the goods to be inventoried at the next moment.

[0054] Specifically, the pose information includes position information, direction information, and angle information. According to the second point cloud data of the goods to be inventoried at the current moment, the pose information of the goods to be inventoried at the next moment can be predicted to determine the pose information of the goods to be inventoried at the next moment.

[0055] Step 104: Track the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment.

[0056] Specifically, according to the pose information of the goods to be inventoried at the next moment, by controlling the movement of the robotic arm, the camera sensor at the end of the robotic arm is aligned with the surface of the goods, and the strategy of gradually changing the shooting pose is adopted to continuously try to detect and identify the goods to be inventoried until the detection result of the goods to be inventoried is accurately obtained. The judgment of the detection result of the goods to be inventoried can be achieved through the Intersection over Union (IOU). The IOU quantifies the overlap degree between the detection box and the true annotation box, and can directly reflect the accuracy of object detection, so as to realize the tracking and inventory of the goods to be inventoried, and improve the accuracy and efficiency of goods inventory.

[0057] The goods tracking method provided by the present invention obtains the cargo image of the goods to be inventoried collected by the camera sensor at the current moment and the first point cloud data corresponding to the cargo image collected by the lidar; the camera sensor and the lidar are installed on the robotic arm; based on the cargo image and the first point cloud data, the second point cloud data of the goods to be inventoried is determined to realize the segmentation of the goods to be inventoried; then according to the second point cloud data, the pose information of the goods to be inventoried at the next moment is determined to realize the pose estimation of the goods to be inventoried; then based on the pose information of the goods to be inventoried at the next moment, the goods to be inventoried are tracked to realize the real-time tracking of the goods to be inventoried, and then the accurate inventory of the goods is realized, improving the inventory efficiency of the goods.

[0058] Optionally, the specific implementation manner of the above step 102 includes:

[0059] (1) Perform object detection on the cargo image to obtain the position information of the goods to be inventoried at the current moment.

[0060] Specifically, first, preprocess the cargo image collected by the camera sensor. The preprocessing methods include at least one of the following: erosion, dilation, and binarization. Then, use the object detection network model to detect the goods to be inventoried in the cargo image. The detection result is represented in the form of a detection box, and the detection box can represent the position information of the goods to be inventoried at the current moment, that is, the position coordinates of each pixel point of the goods to be inventoried in the detection box. Among them, the object detection network model can be a YOLO detection model or other detection models, and no specific limitation is imposed on this.

[0061] (2) Based on the position information and the first point cloud data, determine the second point cloud data of the goods to be inventoried.

[0062] Specifically, according to the position information of the goods to be inventoried and the first point cloud data of the cargo image, the second point cloud data of the goods to be inventoried can be further determined.

[0063] Optionally, the object detection of the cargo image to obtain the position information of the goods to be inventoried at the current moment includes:

[0064] Input the cargo image into the object detection network model to obtain the position information of the goods to be inventoried at the current moment output by the object detection network model. The object detection network model is trained based on sample cargo images and label number data and is used to detect the goods to be inventoried in the cargo image.

[0065] Specifically, input the cargo image into the object detection network model. The object detection network model matches the features of the goods to be inventoried with the features of the goods to be inventoried at the initial moment to obtain the most similar region in the current moment's cargo image to the initial moment's cargo image, and updates the detection box of the goods to be inventoried according to the matching result to ensure that the detection box can accurately reflect the position information of the goods to be inventoried, thereby updating the position information of the goods to be inventoried, realizing the positioning and classification of the goods to be inventoried, obtaining the position information of the goods to be inventoried at the current moment output by the object detection network model, improving the detection accuracy of the position information of the goods to be inventoried, and thus being able to improve the inventory accuracy and efficiency of the goods to be inventoried.

[0066] Optionally, the determining the second point cloud data of the goods to be inventoried based on the position information and the first point cloud data includes:

[0067] (a) Align the first point cloud data with the cargo image corresponding to the position information to determine the position and range of the goods to be inventoried in the first point cloud data.

[0068] Specifically, by aligning the pixel points at each position in the image of the goods to be inventoried corresponding to the position information with the first point cloud data at the corresponding position, the position and range of the goods to be inventoried corresponding to the aligned first point cloud data can be determined.

[0069] In practice, feature points corresponding to each pixel are detected in the image of the goods to be inventoried, and the feature points are matched with the corresponding points in the first point cloud data. By matching the corresponding points in the image of the goods to be inventoried and the first point cloud data, the corresponding relationship between the pixel points and the point cloud data can be established, and the alignment of the first point cloud data and the image of the goods to be inventoried can be achieved. After aligning the image of the goods to be inventoried and the first point cloud data, it is usually necessary to perform coordinate system transformation on the first point cloud data, that is, convert the first point cloud data into three-dimensional coordinates, so that the image of the goods to be inventoried and the first point cloud data are represented in the same coordinate system, and the first point cloud data is corresponding to the pixels in the image of the goods to be inventoried.

[0070] (b) Based on the position and range, determine the second point cloud data of the goods to be inventoried.

[0071] Specifically, according to the position and range of the goods to be inventoried corresponding to the first point cloud data, the goods to be inventoried are segmented from the first point cloud data to obtain the second point cloud data of the goods to be inventoried, realizing the detection and segmentation of the goods to be inventoried, and improving the detection accuracy and efficiency of the goods to be inventoried.

[0072] Optionally, the specific implementation manner of the above step 103 includes:

[0073] (1) Based on the second point cloud data, determine the pose information of the goods to be inventoried at the current moment.

[0074] Specifically, according to the second point cloud data of the goods to be inventoried, a residual network (resnet network) is used to determine the pose information of the goods to be inventoried at the current moment, that is, the second point cloud data of the goods to be inventoried is input into the residual network, and the pose information of the goods to be inventoried output by the residual network can be obtained; among them, the residual network is trained through a large amount of sample data, so as to improve the accuracy and robustness of the pose information.

[0075] (2) Based on the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried, determine the pose information of the goods to be inventoried at the next moment; the relative displacement is obtained based on the position information of the goods to be inventoried at the current moment and the position information at the previous moment.

[0076] It should be noted that the position information of the goods to be inventoried at the current moment and the position information at the previous moment can be obtained by using the object detection network model, and then the relative displacement of the goods to be inventoried is determined according to the position information at the current moment and the position information at the previous moment.

[0077] Specifically, according to the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried, the pose information at the next moment is deduced using the kinematic equation; wherein, the kinematic equation is used to deduce the pose information of the goods to be inventoried according to the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried.

[0078] Optionally, based on the pose information of the goods to be inventoried at the next moment, tracking the goods to be inventoried includes:

[0079] 1) Tracking the motion trajectory of the goods to be inventoried based on the pose information of the goods to be inventoried at the previous moment, the pose information at the current moment, and the pose information at the next moment.

[0080] Specifically, according to the pose information of the goods to be inventoried at the previous moment, the pose information at the current moment, and the pose information at the next moment, combined with the dynamic model, the motion trajectory of the goods to be inventoried can be predicted and tracked; wherein, the dynamic model is used to track the motion trajectory of the goods to be inventoried.

[0081] It should be noted that during the process of tracking the motion trajectory, situations such as target occlusion, scale change, and illumination change may occur. To improve the robustness of tracking, appearance model update, motion model prediction, and multi-scale tracking are adopted; wherein, appearance model update refers to updating the goods to be inventoried according to the new image of the goods to be inventoried to adapt to the changes of the goods to be inventoried over time. That is, features are extracted from the new image of the goods to be inventoried to describe the appearance of the goods to be inventoried, the newly extracted features are matched with the existing appearance model, and according to the result of feature matching, the parameters or representations of the appearance model are updated.

[0082] In motion model prediction, what is predicted is the motion state of the goods to be inventoried within a future time period, such as position, speed, and pose, etc. The purpose of motion model prediction is to be able to more accurately predict the future behavior of the target in tasks such as target tracking or path planning, so as to adopt corresponding control strategies or make decisions. The historical motion data of the goods to be inventoried can be obtained through methods such as camera sensors, lidar measurements, or trajectory recordings. The historical motion data includes information such as the position, speed, and pose of the goods to be inventoried. According to the historical motion data, the parameters in the motion model are estimated through methods such as parameter estimation or model training. For example, the least squares method, maximum likelihood estimation, optimization algorithms, or machine learning algorithms are used to achieve this. Using the estimated motion model parameters, the future motion state of the target is predicted, realizing state prediction using the motion model based on the current observation data and the known historical motion data.

[0083] At each scale, use the object detection network model to detect and locate the goods to be inventoried, and then use the sliding window or region proposal method to perform object detection at each scale.

[0084] Optionally, a termination criterion can also be set to determine whether the goods to be inventoried need to terminate tracking, or to re-detect the goods to be inventoried under certain feature conditions; where the termination criterion is to determine whether the goods to be inventoried belong to the goods in the database, and the feature conditions refer to the abnormal detection or unrecognizability of the goods to be inventoried.

[0085] 2) Based on the movement trajectory of the goods to be inventoried, adjust the pose of the robotic arm and control the movement of the robotic arm to track the goods to be inventoried.

[0086] Specifically, according to the movement trajectory of the goods to be inventoried, adjust the pose of the robotic arm through a motion planning algorithm and send a motion command to the robotic arm to control the movement of the robotic arm. By changing the pose of the robotic arm, the camera sensor on the robotic arm can be aligned with the object surface to track the goods to be inventoried.

[0087] Figure 2 is the second schematic flowchart of the goods tracking method provided by the present invention, as Figure 2 shown, the method includes steps 201 - step 208; where

[0088] Step 201, obtain the goods image of the goods to be inventoried collected by the camera sensor at the current moment and the first point cloud data corresponding to the goods image collected by the lidar; the camera sensor and the lidar are installed on the robotic arm.

[0089] Step 202, input the goods image into the object detection network model to obtain the position information of the goods to be inventoried at the current moment output by the object detection network model.

[0090] Step 203, align the first point cloud data and the goods image corresponding to the position information to determine the position and range of the goods to be inventoried corresponding to the first point cloud data.

[0091] Step 204, based on the position and range, determine the second point cloud data of the goods to be inventoried.

[0092] Step 205, input the second point cloud data of the goods to be inventoried into the residual network to obtain the pose information of the goods to be inventoried at the current moment output by the residual network.

[0093] Step 206, based on the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried, determine the pose information of the goods to be inventoried at the next moment; the relative displacement is obtained based on the position information of the goods to be inventoried at the current moment and the position information at the previous moment.

[0094] Step 207: Based on the pose information of the goods to be inventoried at the previous moment, the pose information at the current moment, and the pose information at the next moment, track the movement trajectory of the goods to be inventoried.

[0095] Step 208: Based on the movement trajectory of the goods to be inventoried, adjust the pose of the robotic arm and control the movement of the robotic arm to track the goods to be inventoried.

[0096] The goods tracking method provided by the present invention installs a visual goods detection system (i.e., a camera sensor and a lidar) on a multi-axis robotic arm to determine the spatial pose of the goods to be inventoried. According to the movement trajectory of the goods to be inventoried in space, the pose of the multi-axis robotic arm is adjusted in real time, and the movement of the robotic arm is controlled to track the goods to be inventoried, solving the problems of safety and efficiency in the complex warehouse environment. At the same time, the pose information obtained through the residual network is more accurate and robust. In terms of saving human resources and realizing intelligent and unmanned intelligent logistics, it can effectively solve the multi-angle recognition and detection of goods in the warehouse environment, so as to realize the inventory of goods, and has great theoretical and practical value.

[0097] Next, the goods tracking device provided by the present invention will be described. The goods tracking device described below can be mutually corresponding and referred to the goods tracking method described above.

[0098] Figure 3 is a schematic structural diagram of the goods tracking device 300 provided by the present invention, as Figure 3 shown, the goods tracking device 300 includes: an acquisition module 301, a goods segmentation module 302, a pose estimation module 303, and a tracking module 304; wherein,

[0099] The acquisition module 301 is configured to acquire the goods image of the goods to be inventoried collected by the camera sensor at the current moment and the first point cloud data corresponding to the goods image collected by the lidar; the camera sensor and the lidar are installed on the robotic arm;

[0100] The goods segmentation module 302 is configured to determine the second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data;

[0101] The pose estimation module 303 is configured to determine the pose information of the goods to be inventoried at the next moment based on the second point cloud data;

[0102] The tracking module 304 is configured to track the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment.

[0103] The goods tracking method provided by the present invention obtains the goods image of the goods to be inventoried collected by the camera sensor at the current moment and the first point cloud data corresponding to the goods image collected by the lidar; the camera sensor and the lidar are installed on the robotic arm; based on the goods image and the first point cloud data, the second point cloud data of the goods to be inventoried is determined to realize the segmentation of the goods to be inventoried; then, according to the second point cloud data, the pose information of the goods to be inventoried at the next moment is determined to realize the pose estimation of the goods to be inventoried; and then, based on the pose information of the goods to be inventoried at the next moment, the goods to be inventoried are tracked, thereby realizing the accurate inventory of the goods to be inventoried and improving the inventory efficiency of the goods.

[0104] Optionally, the goods segmentation module 302 is specifically configured to:

[0105] Perform object detection on the goods image to obtain the position information of the goods to be inventoried at the current moment;

[0106] Based on the position information and the first point cloud data, determine the second point cloud data of the goods to be inventoried.

[0107] Optionally, the goods segmentation module 302 is further configured to:

[0108] Input the goods image into the object detection network model to obtain the position information of the goods to be inventoried at the current moment output by the object detection network model; the object detection network model is trained based on sample goods images and label number data and is used to detect the goods to be inventoried in the goods image.

[0109] Optionally, the goods segmentation module 302 is further configured to:

[0110] Align the first point cloud data with the goods image corresponding to the position information to determine the position and range of the goods to be inventoried corresponding to the first point cloud data;

[0111] Based on the position and range, determine the second point cloud data of the goods to be inventoried.

[0112] Optionally, the pose estimation module 303 is specifically configured to:

[0113] Based on the second point cloud data, determine the pose information of the goods to be inventoried at the current moment;

[0114] Based on the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried, determine the pose information of the goods to be inventoried at the next moment; the relative displacement is obtained based on the position information of the goods to be inventoried at the current moment and the position information at the previous moment.

[0115] Optionally, the tracking module 304 is specifically configured to:

[0116] Track the movement trajectory of the goods to be inventoried based on the pose information of the goods to be inventoried at the previous moment, the pose information at the current moment, and the pose information at the next moment;

[0117] Adjust the pose of the robotic arm based on the movement trajectory of the goods to be inventoried, and control the movement of the robotic arm to track the goods to be inventoried.

[0118] Figure 4 FIG. is a schematic physical structure diagram of an electronic device provided by the present invention. As Figure 4 shown, the electronic device 400 may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 complete mutual communication through the communication bus 440. The processor 410 may call logic instructions in the memory 430 to execute a goods tracking method, which includes: acquiring a goods image of the goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the goods image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm; determining second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data; determining pose information of the goods to be inventoried at the next moment based on the second point cloud data; and tracking the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment.

[0119] In addition, when the logic instructions in the above-mentioned memory 430 are implemented in the form of a software functional unit and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the goods tracking method provided by each of the above methods. The method includes: obtaining a goods image of the goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the goods image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm; determining second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data; determining pose information of the goods to be inventoried at the next moment based on the second point cloud data; and tracking the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment.

[0121] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the goods tracking method provided by each of the above methods. The method includes: obtaining a goods image of the goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the goods image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm; determining second point cloud data of the goods to be inventoried based on the goods image and the first point cloud data; determining pose information of the goods to be inventoried at the next moment based on the second point cloud data; and tracking the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0123] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A goods tracking method, characterized in that, Applied to a robotic arm, including: Obtain a cargo image of the goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the cargo image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm; Based on the cargo image and the first point cloud data, determine the second point cloud data of the goods to be inventoried, including: perform object detection on the cargo image to obtain the position information of the goods to be inventoried at the current moment; align the first point cloud data with the cargo image corresponding to the position information to determine the position and range of the goods to be inventoried corresponding to the first point cloud data; based on the position and range, determine the second point cloud data of the goods to be inventoried; Based on the second point cloud data, determine the pose information of the goods to be inventoried at the next moment, including: based on the second point cloud data, determine the pose information of the goods to be inventoried at the current moment; based on the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried, determine the pose information of the goods to be inventoried at the next moment; the relative displacement is obtained based on the position information of the goods to be inventoried at the current moment and the position information at the previous moment; Based on the pose information of the goods to be inventoried at the next moment, track the goods to be inventoried.

2. The goods tracking method according to claim 1, characterized in that, The performing object detection on the cargo image to obtain the position information of the goods to be inventoried at the current moment includes: Input the cargo image into an object detection network model to obtain the position information of the goods to be inventoried output by the object detection network model at the current moment; the object detection network model is trained based on sample cargo images and label number data and is used to detect the goods to be inventoried in the cargo image.

3. The goods tracking method according to claim 2, characterized in that, The tracking the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment includes: Based on the pose information of the goods to be inventoried at the previous moment, the current moment and the next moment, track the movement trajectory of the goods to be inventoried; Based on the movement trajectory of the goods to be inventoried, adjust the pose of the robotic arm and control the movement of the robotic arm to track the goods to be inventoried.

4. A goods tracking device, characterized in that, Applied to a robotic arm, including: An acquisition module, configured to obtain a cargo image of the goods to be inventoried collected by a camera sensor at the current moment and first point cloud data corresponding to the cargo image collected by a lidar; the camera sensor and the lidar are installed on the robotic arm; A cargo segmentation module, configured to determine the second point cloud data of the goods to be inventoried based on the cargo image and the first point cloud data, including: perform object detection on the cargo image to obtain the position information of the goods to be inventoried at the current moment; align the first point cloud data with the cargo image corresponding to the position information to determine the position and range of the goods to be inventoried corresponding to the first point cloud data; based on the position and range, determine the second point cloud data of the goods to be inventoried; A pose estimation module, configured to determine the pose information of the goods to be inventoried at the next moment based on the second point cloud data, including: determining the pose information of the goods to be inventoried at the current moment based on the second point cloud data; determining the pose information of the goods to be inventoried at the next moment based on the pose information of the goods to be inventoried at the current moment and the relative displacement of the goods to be inventoried; the relative displacement is obtained based on the position information of the goods to be inventoried at the current moment and the position information at the previous moment. A tracking module, configured to track the goods to be inventoried based on the pose information of the goods to be inventoried at the next moment.

5. An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the goods tracking method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, it implements the goods tracking method according to any one of claims 1 to 3.

7. A computer program product, including a computer program, characterized in that, When the computer program is executed by the processor, it implements the goods tracking method according to any one of claims 1 to 3.

Images