Method, device and system for detecting the number of stacked objects

Abstract

The present invention provides a method, device and system for detecting the number of objects in stacked objects, and relates to the field of image processing technology. The method for detecting the number of objects in stacked objects includes: obtaining a depth image of the stacked objects; wherein the depth image carries the depth distance corresponding to each pixel point, and the stacked objects are obtained by stacking multiple target objects of the same size; based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image, determining the object layer corresponding to each target pixel point; determining the physical length and physical width corresponding to each target pixel point based on the depth image, and determining the number of target objects contained in the stacked object based on the object layer and the physical length and physical width corresponding to each target pixel point. The present invention improves the accuracy of cargo quantity detection of stacked goods.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and system for detecting the number of stacked objects. Background Art

[0002] With the rapid development of the automation industry, palletizing robots have become an irreplaceable and important equipment and means in the advanced manufacturing industry. Palletizing robots can simultaneously pick up or place multiple tightly packed stacked goods of the same size from the transfer pallet, thereby replacing manual work to complete the warehousing or outbound work of goods and improving production efficiency. However, since automatic palletizing often involves industrial robots dropping or discarding packages, as well as cargo tail boxes, it not only affects the efficiency of warehousing or outbound operations on the assembly line, but also easily causes economic losses to the company. Therefore, it is very necessary to confirm the quantity of goods when goods are in or out of the warehouse.

[0003] At present, with the development of computer vision technology, 3D point cloud data is mainly used to calculate the total volume of stacked goods, and then the number of goods in the stacked goods is calculated based on the total volume of the stacked goods and the volume of a single item. However, as the number of layers of goods is stacked, the cumulative volume error becomes larger, and the error of the calculated total volume of goods is large, resulting in inaccurate calculated number of goods. Therefore, the existing stacked goods quantity detection technology still has the problem of low accuracy. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a method, device and system for detecting the number of stacked objects, which can effectively avoid the problem of large number statistical errors caused by directly measuring the volume of goods, and improve the accuracy of goods quantity detection of stacked goods.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting the number of stacked objects, comprising: acquiring a depth image of the stacked objects; wherein the depth image carries a depth distance corresponding to each pixel, and the stacked object is obtained by stacking multiple target objects of the same size; based on the depth distance corresponding to each target pixel covered by the stacked object in the depth image, determining the object layer corresponding to each target pixel; determining the physical length and physical width corresponding to each target pixel based on the depth image, and determining the number of target objects contained in the stacked object based on the object layer corresponding to each target pixel and the physical length and physical width.

[0007] Further, an embodiment of the present invention provides a first possible implementation of the first aspect, wherein the depth image is acquired based on an image sensor; the step of determining the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image includes: obtaining a reference surface distance between the plane where the stacked object is located and the image sensor; determining the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point, the reference surface distance and the actual height of the target object.

[0008] Further, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the step of determining the physical length and physical width corresponding to each target pixel based on the depth image includes: performing image denoising and target segmentation on the depth image to obtain a mask image of the stacked object; performing depth data calibration on the depth distance corresponding to each target pixel covered by the stacked object in the mask image to obtain a calibrated distance corresponding to each target pixel; and determining the physical length and physical width corresponding to each target pixel based on the calibrated distance.

[0009] Further, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the step of determining the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point, the reference surface distance, and the actual height of the target object includes:

[0010] Based on the calibration distance corresponding to each target pixel point covered by the stacked object in the depth image, the reference surface distance, the actual height of the target object and the object layer calculation formula, the object layer corresponding to the target pixel point is determined; wherein the object layer calculation formula is:

[0011]

[0012] layer i is the object layer corresponding to the i-th target pixel, dis plane is the reference surface distance, dis i is the calibration distance corresponding to the i-th target pixel, H good is the actual height of the target object.

[0013] Further, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the step of determining the number of target objects contained in the stacked object based on the object layers corresponding to each of the target pixel points and the physical length and physical width includes: determining the cross-sectional area of ​​each object layer based on the object layers corresponding to each of the target pixel points and the physical length and physical width; determining the number of target objects contained in the stacked object based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object.

[0014] Further, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the step of determining the cross-sectional area of ​​each object layer based on the object layer corresponding to each target pixel point and the physical length and the physical width comprises: determining the cross-sectional area of ​​each object layer based on the object layer corresponding to each target pixel point, the physical length, the physical width and an area calculation formula; wherein the area calculation formula is:

[0015]

[0016] S k is the cross-sectional area of ​​the k-th object layer, m is the number of target pixels corresponding to the k-th object layer, is the physical length corresponding to the i-th target pixel in the k-th layer, is the physical width corresponding to the i-th target pixel point in the k-th layer, n is the number of target pixels whose corresponding object layer is the k+1-th layer, r is the total number of object layers of the stacked object, and t is the number of target pixels whose object layer is the r-th layer.

[0017] Furthermore, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the step of determining the number of target objects contained in the stacked object based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object comprises: determining the number of objects in each object layer based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object; and determining the number of target objects contained in the stacked object based on the sum of the number of objects in each object layer.

[0018] In a second aspect, an embodiment of the present invention further provides an object quantity detection device for stacked objects, comprising: an image acquisition module, used to acquire a depth image of the stacked objects; wherein the depth image carries the depth distance corresponding to each pixel point, and the stacked object is obtained by stacking multiple target objects of the same size; an object layer determination module, used to determine the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image; a quantity determination module, used to determine the physical length and physical width corresponding to each target pixel point, and determine the number of target objects contained in the stacked object based on the object layer to which each target pixel point belongs and the physical length and physical width.

[0019] In a third aspect, an embodiment of the present invention provides an object quantity detection system for stacked objects, comprising: a conveyor belt, an image sensor and a controller; the image sensor is arranged directly above a target position on the conveyor belt; the conveyor belt is used to transport stacked objects; the image sensor is used to obtain a depth image of the stacked objects when the stacked objects are transported to the target position, and send the depth image to the controller; the controller comprises a processor and a storage device; a computer program is stored on the storage device, and the computer program, when executed by the processor, executes the method as described in any one of the first aspects.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in any one of the above-mentioned first aspects are executed.

[0021] The embodiment of the present invention provides a method, device and system for detecting the number of objects in a stacked object. The method includes: first obtaining a depth image (carrying the depth distance corresponding to each pixel) of a stacked object (obtained by stacking multiple target objects of the same size); then determining the object layer corresponding to each target pixel based on the depth distance corresponding to each target pixel covered by the stacked object in the depth image; finally determining the physical length and physical width corresponding to each target pixel based on the depth image, and determining the number of target objects contained in the stacked object based on the object layer and the physical length and physical width corresponding to each target pixel. In this method, by obtaining a depth image of the stacked object, determining the object layer corresponding to each target pixel covered by the stacked object and the physical length and physical width corresponding to each target pixel based on the depth image, the number of target objects in the stacked object can be quickly calculated, effectively avoiding the problem of large number statistical errors caused by directly measuring the volume of the goods, and improving the accuracy of the detection of the number of goods in the stacked goods.

[0022] Other features and advantages of the embodiments of the present invention will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned techniques of the embodiments of the present invention.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flow chart of a method for detecting the number of stacked objects provided by an embodiment of the present invention is shown;

[0026] Figure 2 A depth image of a stacked object provided by an embodiment of the present invention is shown;

[0027] Figure 3 A smooth image of a stacked object provided by an embodiment of the present invention is shown;

[0028] Figure 4 An image to be segmented of a stacked object provided by an embodiment of the present invention is shown;

[0029] Figure 5 A mask diagram of stacked objects provided by an embodiment of the present invention is shown;

[0030] Figure 6 A graph showing the result of object quantity detection provided by an embodiment of the present invention is shown;

[0031] Figure 7 A flow chart of cargo quantity detection provided by an embodiment of the present invention is shown;

[0032] Figure 8 A schematic structural diagram of a device for detecting the number of stacked objects provided by an embodiment of the present invention is shown;

[0033] Fig. 9 A schematic diagram of the structure of a controller provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be described below in conjunction with the drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0035] At present, robot palletizers are widely used in chemical, beverage, food, beer, plastic and other production enterprises, mainly used for handling various cartons, bags, cans, beer boxes, bottles and other packaged finished products. Robot palletizers realize automatic storage or outbound delivery of products by placing or picking up multiple items of the same specifications from the transfer pallet. Since robots will carry multiple goods of the same specifications at the same time when carrying goods, in order to accurately calculate the outbound or inbound quantity, it is necessary to detect the quantity of stacked goods carried by the robot. At present, 2D or 3D vision algorithms are mainly used to detect the quantity of goods. However, when goods of the same size and specifications are placed on the same pallet, it is difficult to segment and identify them by relying on 2D vision algorithms. 3D vision algorithms generally use 3D point cloud gridding to calculate the unit volume of goods and calculate the number of goods based on the standard volume of a single cargo. This method is simple and has a small amount of calculation, but as the number of cargo layers is stacked, the volume error accumulates and becomes larger, resulting in inaccurate statistics on the number of goods.

[0036] Considering that the existing technology for detecting the quantity of stacked goods still has the problem of low accuracy, in order to improve this problem, the embodiments of the present invention provide a method, device and system for detecting the quantity of stacked objects, which can be used to improve the accuracy of detecting the quantity of stacked goods. The embodiments of the present invention are described in detail below.

[0037] This embodiment provides a method for detecting the number of stacked objects. The method can be applied to electronic devices such as controllers or computers connected to image sensors. Figure 1 The method for detecting the number of stacked objects shown in the flowchart mainly includes the following steps S102 to S106:

[0038] Step S102, obtaining a depth image of the stacked object.

[0039] The above-mentioned depth image carries the depth distance corresponding to each pixel point, and the above-mentioned stacked object is obtained by stacking multiple target objects of the same size. The above-mentioned stacked object is a multi-layer object formed when multiple objects of the same size are tightly placed on the same tray. In order to accurately detect the number of target objects included in the stacked object, an image sensor can be used to obtain a depth image of the stacked object, and the depth image carries the depth distance corresponding to each pixel point (that is, the distance between the image sensor and the surface of the stacked object). The above-mentioned image sensor can be a sensor that can capture the depth image of the stacked object, such as a solid-state laser radar or a 3D camera (also known as a depth camera).

[0040] Step S104: determining the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image.

[0041] The number of objects placed in each object layer of the stacked objects may be different. When carrying the stacked objects, in order to ensure the stability of the object placement, the number of target objects in the bottom layer is usually larger, and the number of target objects in the top layer is less than or equal to the number of target objects in the bottom layer. Since the depth distance is the distance between each pixel point on the surface of the stacked object and the image sensor, the object layer corresponding to each target pixel point can be obtained according to the distance between each target pixel point on the surface of the stacked object and the image sensor, and the actual height of the target object.

[0042] Step S106, determining the physical length and physical width corresponding to each target pixel based on the depth image, and determining the number of target objects included in the stacked object based on the object layer and the physical length and physical width corresponding to each target pixel.

[0043] When the image sensor acquires an image, it samples the physically continuous stacked objects into discrete pixels. Each pixel covered by the stacked objects in the depth image corresponds to a three-dimensional regular quadrangular prism with a specific size. The actual physical length and physical width corresponding to each pixel can be determined based on the internal parameters of the image sensor. The cross-sectional area of ​​each object layer of the stacked object can be determined based on the physical layer corresponding to each target pixel and the physical length and physical width corresponding to each target pixel. The number of target objects contained in each layer of the stacked object can be determined based on the cross-sectional area of ​​each object layer and the size of the target object, thereby obtaining the number of all target objects included in the stacked object.

[0044] The object quantity detection method for the above-mentioned stacked objects provided in the present embodiment can quickly calculate the number of target objects in the stacked objects by acquiring a depth image of the stacked objects and determining the object layer corresponding to each target pixel point covered by the stacked objects and the physical length and physical width corresponding to each target pixel point based on the depth image, thereby effectively avoiding the problem of large number statistical errors caused by directly measuring the volume of goods, and improving the accuracy of goods quantity detection of stacked goods.

[0045] In order to improve the accuracy of stacked goods quantity detection, this embodiment provides an implementation method for determining the physical length and physical width corresponding to each target pixel point based on the depth image, which can be specifically performed with reference to the following steps 1 to 3:

[0046] Step 1: Perform image denoising and object segmentation on the depth image to obtain a mask image of the stacked object.

[0047] Since the depth images collected by the image sensor usually have holes and noise, see Figure 2 The depth image of the stacked object shown in the figure, the smaller the depth distance corresponding to the pixel point in the depth image, the darker the color of the pixel point, and the median filter is used to filter the depth of the pixel point. Figure 2 The depth image in is processed by filling holes and removing noise to obtain a smooth image, see Figure 3 The smoothed image of the stacked objects shown in the figure, after the depth image is denoised by median filtering, the holes in the image are filled and the noise is removed.

[0048] In order to reduce the impact of debris and platforms where stacked objects are located (such as conveyor belts or cargo pallets) on target segmentation and extraction, the smoothed image obtained above can be subjected to through-filtering processing based on the distance between the image sensor and the plane where the target object is located, and the pixels within the selected range are retained. For example, when the reference plane distance between the image sensor and the plane where the stacked objects are located is x, in order to filter out the interference of other debris or ground on target extraction, the pixels in the depth image within the distance range of (0, x) from the image sensor can be retained (the values ​​of the retained pixels are set to white pixels 255), and the pixels in the depth image whose distance values ​​from the image sensor are greater than or equal to x are removed (the values ​​of the removed pixels are set to black pixels 0). See, for example, Figure 4 The image to be segmented of the stacked object shown in the figure is filtered through the smoothed image according to the installation height of the image sensor and the height of the conveyor belt or pallet where the stacked objects are located, so as to remove the debris in the image, and the object image corresponding to the pixel points in the depth image whose depth distance is less than or equal to the reference surface distance (the distance between the conveyor belt or pallet where the target object is located and the image sensor) is set as white pixel points and retained, and the object image corresponding to the pixel points in the depth image whose depth distance is greater than the reference surface distance is set as black pixel points. Figure 4 The white pixels in the image are the images of objects whose distance between the surface and the image sensor is less than or equal to the reference surface distance. Figure 4 The black pixels in the figure are the filtered out areas of no interest.

[0049] The image sensor is usually set just above the stacked object. The stacked object to be detected is usually in the center of the image sensor's sensing range and has the largest area. The image to be segmented is extracted and fitted with the minimum circumscribed rectangular frame (the minimum rectangle with the maximum horizontal coordinate, minimum horizontal coordinate, maximum vertical coordinate, and minimum vertical coordinate of each vertex of the given two-dimensional shape) to obtain the mask image of the target object. See, for example, Figure 5 The mask map of the stacked objects shown, Figure 5 The white pixels in the image are the images of the stacked objects, and the black pixels are the background.

[0050] Step 2: Perform depth data calibration on the depth distance corresponding to each target pixel covered by the stacked object in the mask image to obtain the calibration distance corresponding to each target pixel.

[0051] The mask image of the stacked object segmented as above is obtained. The depth distance line segment between the surface of the stacked object and the image sensor in the depth image may have a certain angle with the optical axis of the image sensor, and the obtained depth distance still has a certain error. In order to improve the calculation accuracy of the physical size corresponding to each target pixel, the depth distance corresponding to each target pixel of the stacked object in the above mask image (that is, the distance from the area on the surface of the stacked object corresponding to each target pixel to the image sensor) is calibrated according to the camera intrinsic parameters of the image sensor to eliminate the error of the depth distance of each target pixel.

[0052] Obtain the pixel coordinates of each target pixel point of the stacked object segmented in the depth image, and calculate the pixel coordinates of each target pixel point of the stacked object (i.e. Figure 5 The depth distance of each white pixel in the stacked object is calibrated to obtain the calibration distance of each target pixel of the stacked object; wherein the first calculation formula is:

[0053]

[0054] dis i is the calibration distance of the i-th target pixel covered by the stacked object, is the depth distance corresponding to the ith target pixel point covered by the stacked object (the distance between each target pixel point on the surface of the stacked object carried in the depth image and the image sensor), (x, y) is the pixel coordinates of the ith target pixel point covered by the stacked object, CX is the lateral offset of the optical axis of the image sensor in the image coordinate system, CY is the longitudinal offset of the optical axis of the image sensor in the image coordinate system (the origin of the image coordinate system is the center point of the image) (the unit of the offset can be pixel), and FX is the focal length of the image sensor.

[0055] Step 3: Determine the physical length and physical width corresponding to each target pixel based on the calibration distance.

[0056] Each white pixel in the mask image of the stacked object can be regarded as a cargo unit. The collection of cargo units forms the image of the stacked object. Each pixel can represent a regular quadrangular prism with a specific size. According to the triangle principle, the actual length and actual width (i.e., the actual physical length and physical width) of each white pixel in the surface area of ​​the target object can be calculated.

[0057] Based on the calibration distance corresponding to each target pixel point covered by the stacked object and the second calculation formula, the physical length of the surface area of ​​the stacked object corresponding to each target pixel point is determined; wherein the second calculation formula is:

[0058]

[0059] Based on the calibration distance corresponding to each target pixel point covered by the stacked object and the third calculation formula, the physical width of the surface area of ​​the stacked object corresponding to each target pixel point is determined; wherein the third calculation formula is:

[0060]

[0061] W i is the physical length of the i-th target pixel covered by the stacked object in the depth image, α is the horizontal field of view of the image sensor, and F w is the row pixel resolution of the depth image (the number of pixels per row), L i is the physical width of the i-th target pixel covered by the stacked object in the depth image, β is the vertical field of view of the image sensor, and F h is the column pixel resolution of the depth image (the number of pixels per column), i is the calibration distance corresponding to the i-th target pixel of the target object.

[0062] In order to accurately determine the object layer corresponding to each target pixel, this embodiment provides an implementation method for determining the object layer corresponding to each target pixel based on the depth distance corresponding to each target pixel covered by the stacked object in the depth image. Specifically, the following steps (1) to (2) may be referred to for execution:

[0063] Step (1): Obtain the reference plane distance between the plane where the stacked objects are located and the image sensor.

[0064] The reference plane distance is the vertical distance between the plane where the stacked objects are located and the image sensor. When the stacked objects are located above the tray, the reference plane distance is the vertical distance between the upper surface of the tray and the image sensor.

[0065] Step (2): Based on the depth distance corresponding to each target pixel point, the reference surface distance and the actual height of the target object, determine the object layer corresponding to each target pixel point.

[0066] Based on the calibration distance, reference surface distance, actual height of the target object and object layer calculation formula corresponding to each target pixel covered by the stacked object in the depth image, the object layer corresponding to the target pixel is determined; wherein the object layer calculation formula is:

[0067]

[0068] layer i is the object layer corresponding to the i-th target pixel, dis plane is the reference surface distance, dis i is the calibration distance corresponding to the i-th target pixel, H good is the actual height of the target object. The stacked object is an object with a certain physical height, and the reference surface distance dis plane is the vertical distance between the plane where the stacked object is located and the image sensor. The calibration distance dis i is the vertical distance between each target pixel on the surface of the stacked object and the image sensor, through the dis plane -dis i The actual physical height corresponding to each target pixel on the surface of the stacked object can be calculated (i.e., the actual height of each cargo unit corresponding to the regular quadrangular prism). By calculating the ratio of the actual physical height corresponding to each target pixel to the actual height of the target object, the object layer to which each target pixel belongs can be calculated. For example, if the actual height of the target object is a, when the actual physical height corresponding to the i-th target pixel of the stacked object is dis plane -dis i =a, Determine that the object layer corresponding to the i-th target pixel point is the first layer (that is, the bottom layer object next to the tray or conveyor belt is the first layer object); when the actual physical height dis corresponding to the i-th target pixel point of the stacked object plane -dis i =2a, Determine that the object layer corresponding to the i-th target pixel is the second layer.

[0069] In order to accurately calculate the number of target objects included in the stacked objects, this embodiment provides an implementation method for determining the number of target objects included in the stacked objects based on the object layer to which each target pixel point belongs and the physical length and physical width. Specifically, the following steps 1) to 2) may be referred to for execution:

[0070] Step 1): Based on the object layer to which each target pixel belongs and the physical length and physical width, the cross-sectional area of ​​each object layer is determined.

[0071] The number of target pixels corresponding to each object layer is determined based on the object layer corresponding to each target pixel. The cross-sectional area of ​​each object layer is determined based on the object layer, physical length, physical width and area calculation formula corresponding to each target pixel; wherein the area calculation formula is:

[0072]

[0073] S kis the cross-sectional area of ​​the k-th object layer, m is the number of target pixels corresponding to the k-th object layer, is the physical length corresponding to the i-th target pixel in the k-th layer, is the physical width corresponding to the i-th target pixel in the k-th layer, n is the number of target pixels whose corresponding object layer is the k+1-th layer, r is the total number of object layers of the stacked objects, and t is the number of target pixels whose object layer is the r-th layer. is the physical length corresponding to the i-th target pixel in the k+1th layer, is the physical width corresponding to the i-th target pixel in the k+1th layer, is the physical length corresponding to the i-th target pixel in the r-th layer, is the physical width corresponding to the i-th target pixel in the r-th layer.

[0074] The depth image only includes the pixels on the upper surface of the stacked objects, and the upper target objects in the stacked objects will block the lower target objects. Therefore, when calculating the cross-sectional area of ​​each object layer in the stacked objects, it is necessary to obtain the area of ​​the lower object layer based on the area of ​​the upper object layer. For example, when the total number of stacked layers r of the stacked objects is 3, the cross-sectional area of ​​the first object layer is Cross-sectional area of ​​the second object layer Cross-sectional area of ​​the third object layer

[0075] When the total number of stacked layers r is 2, the cross-sectional area of ​​the first layer is Cross-sectional area of ​​the second object layer

[0076] in, is the physical length corresponding to the i-th target pixel point in the first object layer (the number of target pixels corresponding to the first object layer is m), is the physical width corresponding to the i-th target pixel of the first object layer, is the physical length corresponding to the i-th target pixel point of the second object layer (the number of target pixels corresponding to the second object layer is n), is the physical width corresponding to the i-th target pixel point in the second object layer, is the physical length corresponding to the i-th target pixel point in the third object layer (the number of target pixels in the third object layer is t), It is the physical width corresponding to the i-th target pixel of the third object layer.

[0077] Step 2): Based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object, determine the number of target objects included in the stacked object.

[0078] Based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object, the number of objects in each object layer is determined. p k is the number of target objects contained in the kth object layer, S k is the cross-sectional area of ​​the k-th object layer (i.e., the upper surface area of ​​the k-th object layer), W good is the actual length of the target object, H good is the actual width of the target object.

[0079] Determine the number of target objects contained in the stacked object based on the sum of the number of objects in each object layer. Wherein, r is the total number of object layers of the stacked object. For example, when the stacked object includes 3 object layers, that is, the total number of object layers is 3, the total number of target objects contained in the stacked object is When the stacked object includes two object layers, that is, the total number of object layers is two, the calculation formula for the total number of target objects included in the stacked object is:

[0080] In a specific embodiment, by using the above-mentioned object quantity detection method for stacked objects, Figure 2 By detecting and calculating the stacked objects in the depth image, the quantity of goods included in the stacked objects can be calculated. Figure 6 As shown in the object quantity detection result diagram, the number of goods included in the stacked objects (i.e., the number of target objects) is 7. The detection result diagram is a result output method of the above-mentioned object quantity detection method for stacked objects, and the number of goods included in the stacked objects is output through the text in the image.

[0081] The object quantity detection method for the above-mentioned stacked objects provided in the present embodiment can quickly calculate the number of target objects included in each object layer by calculating the cross-sectional area of ​​each object layer of the stacked objects. According to the sum of the number of target objects included in each object layer, the total number of target objects included in the stacked objects can be calculated in real time, thereby realizing fast and accurate detection of the number of objects in the stacked objects, reducing calculation errors, and effectively avoiding the problem of errors in number statistics caused by directly measuring the volume of the stacked objects, thereby improving the efficiency and robustness of detection of the number of objects in the stacked objects.

[0082] Based on the above-mentioned embodiment, this embodiment provides an example of applying the above-mentioned method for detecting the number of objects in stacked objects to detect the number of goods in unloading and stacking goods, see Figure 7 The cargo quantity detection flow chart shown in the figure can be specifically performed with reference to the following steps S702 to S710:

[0083] Step S702: Acquire a depth image of the unloaded and palletized goods, and pre-process the depth image.

[0084] The preprocessing of the depth image includes: using median filtering to fill in data holes and remove noise from the depth image, then performing through filtering based on the reference surface distance between the image sensor and the plane where the unloading and stacking goods are located, and then using contour extraction and minimum circumscribed rectangular frame fitting to perform target segmentation to obtain the preprocessed depth image (i.e., the mask map of the unloading and stacking goods).

[0085] Step S704: performing data calibration on the pre-processed depth image, and calculating the cargo unit size of each target pixel point covered by the unloading and stacking cargo based on the calibrated depth image.

[0086] The above-mentioned cargo unit size includes physical length, physical width and physical height. The pixel coordinates of the segmented unloading and palletizing cargo are obtained, and the depth distance of each pixel point of the unloading and palletizing cargo is calibrated according to the internal reference of the image sensor. When the image sensor acquires an image, physically continuous objects are sampled into discrete pixels. Therefore, each depth data in the mask image of the unloading and palletizing cargo is regarded as a cargo unit to form a cargo unit set, and each cargo unit represents a three-dimensional cube with specific dimensions. According to the triangle principle and the calibrated depth data, the actual physical dimensions of the length, width and height occupied by each depth pixel point (that is, each cargo unit) can be calculated, that is, the physical length, physical width and physical height.

[0087] Step S706: Based on the physical height of each target pixel and the actual height of a single cargo, determine the cargo layer corresponding to each target pixel.

[0088] Step S708: determining the upper surface area of ​​each cargo layer based on the cargo layer corresponding to each target pixel point and the physical length and physical height of each target pixel point.

[0089] Step S710: Determine the total quantity of goods included in the unloading and palletizing of goods based on the upper surface area of ​​each cargo layer and the actual length and actual width of a single cargo.

[0090] Corresponding to the method for detecting the number of stacked objects provided in the above embodiment, an embodiment of the present invention provides a device for detecting the number of stacked objects, see Figure 8The structure diagram of a device for detecting the number of stacked objects is shown, and the device includes the following modules:

[0091] The image acquisition module 81 is used to acquire a depth image of the stacked object; wherein the depth image carries the depth distance corresponding to each pixel point, and the stacked object is obtained by stacking multiple target objects of the same size.

[0092] The object layer determination module 82 is used to determine the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image.

[0093] The quantity determination module 83 is used to determine the physical length and physical width corresponding to each target pixel, and determine the number of target objects included in the stacked object based on the object layer to which each target pixel belongs and the physical length and physical width.

[0094] The above-mentioned object quantity detection device for stacked objects provided in the present embodiment can quickly calculate the number of target objects in the stacked objects by acquiring a depth image of the stacked objects and determining the object layer corresponding to each target pixel point covered by the stacked objects and the physical length and physical width corresponding to each target pixel point based on the depth image, thereby effectively avoiding the problem of large number statistical errors caused by directly measuring the volume of goods, and improving the accuracy of goods quantity detection of stacked goods.

[0095] In one embodiment, the depth image is acquired based on an image sensor; the object layer determination module 82 is further used to obtain the reference surface distance between the plane where the stacked object is located and the image sensor; based on the depth distance corresponding to each target pixel point, the reference surface distance and the actual height of the target object, the object layer corresponding to each target pixel point is determined.

[0096] In one embodiment, the quantity determination module 83 is further used to perform image denoising and target segmentation on the depth image to obtain a mask image of the stacked object; perform depth data calibration on the depth distance corresponding to each target pixel point covered by the stacked object in the mask image to obtain the calibration distance corresponding to each target pixel point; and determine the physical length and physical width corresponding to each target pixel point based on the calibration distance.

[0097] In one embodiment, the object layer determination module 82 is further used to determine the object layer corresponding to the target pixel point based on the calibration distance corresponding to each target pixel point covered by the stacked object in the depth image, the reference surface distance, the actual height of the target object and the object layer calculation formula; wherein the object layer calculation formula is:

[0098]

[0099] layeri is the object layer corresponding to the i-th target pixel, dis plane is the reference surface distance, dis i is the calibration distance corresponding to the i-th target pixel, H good is the actual height of the target object.

[0100] In one embodiment, the quantity determination module 83 is further used to determine the cross-sectional area of ​​each object layer based on the object layer and the physical length and physical width corresponding to each target pixel point; and to determine the number of target objects contained in the stacked object based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object.

[0101] In one embodiment, the quantity determination module 83 is further used to determine the cross-sectional area of ​​each object layer based on the object layer, physical length, physical width and area calculation formula corresponding to each target pixel point; wherein the area calculation formula is:

[0102]

[0103] S k is the cross-sectional area of ​​the k-th object layer, m is the number of target pixels corresponding to the k-th object layer, is the physical length corresponding to the i-th target pixel in the k-th layer, is the physical width corresponding to the i-th target pixel in the k-th layer, n is the number of target pixels whose corresponding object layer is the k+1-th layer, r is the total number of object layers of the stacked objects, and t is the number of target pixels whose object layer is the r-th layer.

[0104] In one embodiment, the quantity determination module 83 is further used to determine the number of objects in each object layer based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object; and determine the number of target objects contained in the stacked object based on the sum of the number of objects in each object layer.

[0105] The object quantity detection device for the above-mentioned stacked objects provided in the present embodiment can quickly calculate the number of target objects included in each object layer by calculating the cross-sectional area of ​​each object layer of the stacked objects. According to the sum of the number of target objects included in each object layer, the total number of target objects included in the stacked objects can be calculated in real time, thereby realizing fast and accurate detection of the number of objects in the stacked objects, reducing calculation errors, and effectively avoiding the problem of errors in number statistics caused by directly measuring the volume of the stacked objects, thereby improving the efficiency and robustness of detection of the number of objects in the stacked objects.

[0106] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0107] Corresponding to the method and device provided in the above-mentioned embodiment, an embodiment of the present invention further provides a system for detecting the number of stacked objects, the system comprising: a conveyor belt, an image sensor and a controller.

[0108] The conveyor belt is used to transport stacked objects.

[0109] The image sensor is arranged just above the target position on the conveyor belt. The image sensor is used to obtain a depth image of the stacked objects when the stacked objects are transported to the target position, and send the depth image to a controller including a processor and a storage device.

[0110] An embodiment of the present invention provides a controller, such as Fig. 9 As shown in the schematic diagram of the controller structure, the controller includes a processor 91 and a memory 92. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.

[0111] See also Fig. 9 The controller further includes: a bus 94 and a communication interface 93, and the processor 91, the communication interface 93 and the memory 92 are connected via the bus 94. The processor 91 is used to execute executable modules stored in the memory 92, such as computer programs.

[0112] The memory 92 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 93 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0113] The bus 94 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0114] Among them, the memory 92 is used to store programs, and the processor 91 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 91, or implemented by the processor 91.

[0115] The processor 91 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 91 or the instructions in the form of software. The above processor 91 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 92, and the processor 91 reads the information in the memory 92 and completes the steps of the above method in combination with its hardware.

[0116] An embodiment of the present invention provides a computer-readable medium, wherein the computer-readable medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the above embodiment.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned embodiment, and will not be repeated here.

[0118] The computer program product of the method, device and system for detecting the number of stacked objects provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0119] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0121] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0122] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting the number of stacked objects, characterized in that: include: Acquire a depth image of a stacked object; wherein the depth image carries a depth distance corresponding to each pixel point, and the stacked object is obtained by stacking multiple target objects of the same size; Determining the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image; Determine the physical length and physical width corresponding to each of the target pixels based on the depth image, and determine the number of target objects included in the stacked object based on the object layer corresponding to each of the target pixels and the physical length and physical width; The depth image is obtained based on an image sensor; the step of determining the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image comprises: obtaining the reference surface distance between the plane where the stacked object is located and the image sensor; determining the object layer corresponding to the target pixel point based on the calibration distance corresponding to each target pixel point covered by the stacked object in the depth image, the reference surface distance, the actual height of the target object and the object layer calculation formula; wherein the object layer calculation formula is: layer i is the object layer corresponding to the i-th target pixel, dis plane is the reference surface distance, dis i is the calibration distance corresponding to the i-th target pixel, H good is the actual height of the target object; The step of determining the physical length and physical width corresponding to each of the target pixels based on the depth image comprises: determining the physical length and physical width corresponding to each of the target pixels based on the calibration distance; The calculation formula of the physical length is: The calculation formula of the physical width is: Among them, W i is the physical length corresponding to the i-th target pixel covered by the stacked object in the depth image, α is the horizontal field of view of the image sensor, and F w is the row pixel resolution of the depth image, L i is the physical width of the i-th target pixel covered by the stacked object in the depth image, β is the vertical field of view of the image sensor, and F h is the column pixel resolution of the depth image, dis i is the calibration distance corresponding to the i-th target pixel; The step of determining the number of target objects contained in the stacked objects based on the object layer corresponding to each target pixel point and the physical length and physical width includes: determining the cross-sectional area of ​​each object layer based on the object layer corresponding to each target pixel point, the physical length, the physical width and an area calculation formula; wherein the area calculation formula is: S k is the cross-sectional area of ​​the k-th object layer, m is the number of target pixels corresponding to the k-th object layer, is the physical length corresponding to the i-th target pixel in the k-th layer, is the physical width corresponding to the i-th target pixel of the k-th layer, n is the number of target pixels whose corresponding object layer is the k+1-th layer, r is the total number of object layers of the stacked object, and t is the number of target pixels whose object layer is the r-th layer; Determining the number of target objects included in the stacked object based on the cross-sectional area of ​​each of the object layers and the actual length and the actual width of the target object; The method for determining the calibration distance corresponding to the target pixel point includes: performing image denoising and target segmentation on the depth image to obtain a mask image of the stacked object; performing depth data calibration on the depth distance corresponding to each target pixel point covered by the stacked object in the mask image to obtain the calibration distance corresponding to each target pixel point.

2. The method according to claim 1, characterized in that The step of determining the number of target objects included in the stacked objects based on the cross-sectional area of ​​each object layer and the actual length and actual width of the target object comprises: Determining the number of objects in each object layer based on the cross-sectional area of ​​each object layer and the actual length and the actual width of the target object; The number of target objects included in the stacked objects is determined based on the sum of the number of objects in each of the object layers.

3. A device for detecting the number of stacked objects, characterized in that: include: An image acquisition module, used to acquire a depth image of a stacked object; wherein the depth image carries a depth distance corresponding to each pixel point, and the stacked object is obtained by stacking multiple target objects of the same size; An object layer determination module, configured to determine the object layer corresponding to each target pixel point based on the depth distance corresponding to each target pixel point covered by the stacked object in the depth image; a quantity determination module, configured to determine a physical length and a physical width corresponding to each of the target pixels, and determine the number of target objects contained in the stacked object based on the object layer to which each of the target pixels belongs and the physical length and the physical width; The object layer determination module is used to obtain the reference surface distance between the plane where the stacked object is located and the image sensor; based on the calibration distance corresponding to each target pixel point covered by the stacked object in the depth image, the reference surface distance, the actual height of the target object and the object layer calculation formula, determine the object layer corresponding to the target pixel point; wherein the object layer calculation formula is: layer i is the object layer corresponding to the i-th target pixel, dis plane is the reference surface distance, dis i is the calibration distance corresponding to the i-th target pixel, H good is the actual height of the target object; The quantity determination module is used to perform image denoising and target segmentation on the depth image to obtain a mask image of the stacked object; perform depth data calibration on the depth distance corresponding to each target pixel point covered by the stacked object in the mask image to obtain a calibration distance corresponding to each target pixel point; and determine the physical length and physical width corresponding to each target pixel point based on the calibration distance; The calculation formula of the physical length is: The calculation formula of the physical width is: Among them, W i is the physical length corresponding to the i-th target pixel covered by the stacked object in the depth image, α is the horizontal field of view of the image sensor, and F w is the row pixel resolution of the depth image, L i is the physical width of the i-th target pixel covered by the stacked object in the depth image, β is the vertical field of view of the image sensor, and F h is the column pixel resolution of the depth image, dis i is the calibration distance corresponding to the i-th target pixel; Based on the object layer corresponding to each target pixel point, the physical length, the physical width and the area calculation formula, the cross-sectional area of ​​each object layer is determined; wherein the area calculation formula is: S K is the cross-sectional area of ​​the k-th object layer, m is the number of target pixels corresponding to the k-th object layer, is the physical length corresponding to the i-th target pixel in the k-th layer, is the physical width corresponding to the i-th target pixel of the k-th layer, n is the number of target pixels whose corresponding object layer is the k+1-th layer, r is the total number of object layers of the stacked object, and t is the number of target pixels whose object layer is the r-th layer; The number of target objects included in the stacked objects is determined based on the cross-sectional area of ​​each of the object layers and the actual length and the actual width of the target objects.

4. A system for detecting the number of stacked objects, characterized in that: include: conveyor belts, image sensors and controllers; The image sensor is arranged directly above the target position on the conveyor belt; The conveyor belt is used to transport stacked objects; The image sensor is used to obtain a depth image of the stacked object when the stacked object is transferred to the target position, and send the depth image to the controller; The controller includes a processor and a storage device; a computer program is stored in the storage device, and when the computer program is executed by the processor, the method according to any one of claims 1 to 2 is executed.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are executed.

Images