Real-time online cargo volume recognition method, system, and device, and medium

Abstract

The present application relates to a real-time online cargo volume recognition method, system, and device, and a medium. The method comprises: collecting, by means of a camera, a depth image and a color image of a set area comprising a cargo; introducing a median filter to remove noise pixels in the depth image, and aligning the color image with the filtered depth image on the basis of internal and external parameters of the camera to obtain an aligned color image; on the basis of color, light, and grayscale differences between the cargo and a non-cargo object, using a pre-trained YOLACT model to perform instance segmentation on the aligned color image, and generating a cargo mask; and performing edge fitting, surface extraction, and background separation of the cargo by means of the aligned color image, the filtered depth image and the cargo mask, so as to determine the volume of the cargo. Thus, the present application has high stability and robustness, reliable data can be obtained regardless of the placement location and the placement direction of a cargo, and the working efficiency can be improved.

Description

A method, system, device and medium for real-time online identification of cargo volume

[0001] This application claims priority to the Chinese patent application filed with the Patent Office of China on August 21, 2023, with application number 202311047066.1, and entitled “A method, system, device and medium for real-time online identification of cargo volume”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of cargo identification technology, and in particular to a method, system, device and medium for real-time online identification of cargo volume. Background Art

[0003] Sorting operations in distribution centers are gradually shifting from traditional manual sorting to automated sorting, but volume calculation remains a major challenge for the industry. Traditional line-scan structured light solutions are expensive, making commercial use difficult in small-scale logistics centers.

[0004] As a result, cargo sorting is largely based on couriers' intuitive vision, leading to unusually dense and crowded packages. However, humans' intuitive sense of object size is prone to errors. With the increasing volume of express delivery services, an effective cargo volume measurement solution is necessary. Accurately measuring cargo volume can help couriers better sort cargo and improve sorting efficiency.

[0005] There are some existing technologies that have made certain contributions to the above problems, such as:

[0006] "Binocular Endoscope Three-Dimensional Measurement Method Based on Deep Learning (English)" by Yu Hao et al., Frontiers of Information Technology & Electronic Engineering, Issue 4, 2022. This paper proposes a binocular three-dimensional (3D) measurement method based on deep learning. This method uses a 3D scanner to obtain 3D data of the target, simulates a binocular camera through 3D rendering software, and creates a simulated binocular image dataset to train a disparity estimation model for 3D measurement, thereby providing accurate and reliable dimensions.

[0007] "Individual Corn Plant Height Measurement Method Based on RGB-D Cameras," by Qiu Ruicheng et al., Transactions of the Chinese Society of Agricultural Machinery, 2017, S1, describes a method for measuring corn plant height using an RGB-D camera. This method first uses an EPI-based light field rendering algorithm to process color images and locate their center. The method then clusters the point cloud data to extract the plant, and uses a skeleton detection algorithm to extract the highest point of the central leaf. Finally, the height difference between the highest point above the ground and the central leaf is calculated to measure plant height. Shi Shuai et al. proposed a 3D surface reconstruction and body dimension measurement system based on a multi-view RGB-D camera. This system uses a Kinect depth camera to acquire free-walking point clouds from three different viewpoints (top, left, and right). A rectangular cuboid is used to obtain registration parameters, reconstructing three local point clouds. The distribution of point cloud projections in different directions is then used to determine the measurement location, allowing accurate measurement of key parameters such as body length, height, width, and abdominal circumference. The system also develops a polar coordinate transformation to improve the accuracy of abdominal circumference measurement.

[0008] "Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling," Elise Lachat et al., Remote Sensing, considers the applicability and accuracy of RGB-D cameras for close-range 3D modeling. They investigate the use of the recent Kinect v2 sensor for 3D reconstruction of small objects and discuss its advantages and disadvantages compared to the original sensor and photogrammetry.

[0009] In summary, the above methods are difficult to meet the actual requirements of fast, easy and simple measurement of cargo volume in terms of system complexity and measurement speed, and are difficult to be applied to the cargo sorting assembly line.

[0010] Summary of the Invention

[0011] (1) Technical issues to be resolved

[0012] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present application provides a method, system, device and medium for real-time online identification of cargo volume, which solves the technical problem in the prior art that it is difficult to quickly, easily and accurately identify the cargo volume.

[0013] (2) Technical solution

[0014] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0015] In a first aspect, embodiments of the present application provide a method for real-time online identification of cargo volume, comprising:

[0016] Capturing depth and color images of a set area containing the goods through a camera;

[0017] A median filter is introduced to remove noise pixels in the depth image. The color image and the filtered depth image are aligned by combining the camera's internal and external parameters to obtain an aligned color image.

[0018] Based on the color, illumination, and grayscale differences between goods and non-goods, the pre-trained YOLACT model is used to perform instance segmentation on the aligned color images and generate a goods mask.

[0019] The aligned color image, filtered depth image and cargo mask are used to perform cargo edge fitting, surface extraction and background separation, thereby determining the cargo volume.

[0020] Optionally, a median filter is introduced to remove noise pixels in the depth image, and the color image and the filtered depth image are aligned in combination with the camera's internal and external parameters. The aligned color image includes:

[0021] A median filter is introduced to remove noise pixels in the depth image to obtain a filtered depth image;

[0022] Use an RGB-D camera to shoot pre-set calibration plates at multiple locations to obtain calibration color images and calibration infrared images;

[0023] For calibration, the color and infrared images are used to calibrate the internal and external parameters. After excluding images that do not meet the relative position range of RGB to D and images whose calibration error differs by a certain threshold, the internal and external parameter matrix of the RGB-D camera is obtained.

[0024] According to Zhang Zhengyou's calibration principle, the relative positions of the pixels of the color image and the filtered depth image are aligned according to the obtained internal and external parameter matrices to obtain an aligned color image.

[0025] Optionally, based on the color, illumination, and grayscale differences between goods and non-goods, the pre-trained YOLACT model is used to perform instance segmentation on the aligned color image and generate a goods mask, and the following is also included:

[0026] Get the dataset pre-taken by the camera;

[0027] Use the Labelme tool to label some images in the dataset with target goods to form a coco format dataset, which will serve as the training set for the pre-set YOLACT model.

[0028] The remaining images in the dataset except for the training set are aligned and used as the test set of the pre-set YOLACT model;

[0029] The dataset includes depth images with a resolution of 640×480 and four-channel RGBA color images with a resolution of 1080×1920; the number of depth images and color images in the training set and test set is 1:1.

[0030] Optionally, based on the color, illumination, and grayscale differences between goods and non-goods, a pre-trained YOLACT model is used to perform instance segmentation on the aligned color image and generate a goods mask, including:

[0031] Train the YOLACT model based on the training set and generate corresponding weight parameters;

[0032] Use the generated weight parameters to perform product recognition and instance segmentation on the aligned color images to obtain the products and product masks;

[0033] Use a rectangular box to mark the detected goods and mark the corresponding confidence level on the rectangular box;

[0034] Use the generated product mask to label the detected products.

[0035] Optionally, using the aligned color image, the filtered depth image, and the cargo mask to perform cargo edge fitting, surface extraction, and separation from the background, thereby determining the cargo volume includes:

[0036] Extract the boundary line of the goods using the aligned color image and the filtered depth image;

[0037] Separate goods from background in aligned color images based on fitted edges,

[0038] Surface extraction is achieved using the obtained cargo mask;

[0039] The cargo mask is optimized based on the depth value of the extracted mask to obtain a new mask;

[0040] The cargo volume is determined based on the new mask, the filtered depth image, the intrinsic parameter matrix of the RGB-D camera, and the pixel information in the new mask.

[0041] Optionally, the cargo mask is optimized based on the depth value of the extracted mask, and the new mask obtained includes:

[0042] Initialize a numpy array with the same resolution as the depth map to store the new mask;

[0043] Extract all the depth values ​​of the masked part in the depth map and calculate the median of all the depth values ​​as the upper surface depth value of the cargo mask;

[0044] Get the new mask based on the numpy array and the upper depth value of the cargo mask.

[0045] Optionally, determining the cargo volume according to the new mask, the filtered depth image, the intrinsic parameter matrix of the RGB-D camera, and pixel information in the new mask includes:

[0046] Based on the new mask, the filtered depth image, and the intrinsic parameter matrix of the RGB-D camera, the actual distance between any two pixels in the filtered depth image is calculated.

[0047] Select two coordinate points in the longest continuous row in the new mask, calculate the positions of the two coordinate points in the world coordinate system, and calculate the distance between the two coordinate points;

[0048] Divide the distance between two coordinate points by the number of non-zero pixels between them to get the length of each pixel;

[0049] Get the area of ​​each pixel based on the length of each pixel;

[0050] Based on the filtered depth image and the intrinsic parameters of the depth camera, the volume of the cargo is calculated by combining the number of non-zero pixels in the new mask, the height of the non-zero pixels, and the area of ​​each pixel.

[0051] In a second aspect, an embodiment of the present application provides a real-time online cargo volume identification system, comprising:

[0052] An image acquisition module, configured to acquire a depth image and a color image of a set area containing the goods through a depth camera;

[0053] The noise filtering and image alignment module is used to introduce a median filter to remove noise pixels in the depth image, and align the color image and the filtered depth image with the camera's internal and external parameters to obtain an aligned color image;

[0054] The model call and mask generation module is used to perform instance segmentation on the aligned color images using the pre-trained YOLACT model based on the color, illumination, and grayscale differences between goods and non-goods, and generate a goods mask;

[0055] The cargo volume determination module is used to use the aligned color image, the filtered depth image, and the cargo mask to perform cargo edge fitting, surface extraction, and background separation, thereby determining the cargo volume.

[0056] In a third aspect, an embodiment of the present application provides a real-time online cargo volume identification device, comprising: at least one database; an RGB-D depth camera for capturing depth images and color images of a set area containing cargo and transmitting them to the database; and a memory communicatively connected to the at least one database; wherein the memory stores instructions executable by the at least one database, and the instructions are executed by the at least one database so that the at least one database can execute the real-time online cargo volume identification method as described above.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having computer-executable instructions stored thereon, which, when executed by a processor, implements the real-time online cargo volume identification method as described above.

[0058] (3) Beneficial effects

[0059] The beneficial effects of this application are: it implements depth map visualization based on an RGB-D camera and alignment of depth data to RGB data, RGB image cargo recognition and instance-level segmentation based on the Yolact model, and fitting of object edges and upper surfaces based on depth images and mask information. Ultimately, without making major modifications to the assembly line itself, it implements fast, simple, and accurate measurement of cargo volume. As a result, this application has high stability and robustness, and can obtain reliable data regardless of the location or orientation of the cargo, while also helping to improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] FIG1 is a flow chart of a method for real-time online cargo volume identification according to an embodiment of the present application;

[0061] FIG2 is a schematic diagram of a depth image and color image calibration process of a real-time online cargo volume recognition method provided by an embodiment of the present application;

[0062] FIG3 is a schematic diagram of a specific flow chart of step S2 of a method for real-time online identification of cargo volume provided in an embodiment of the present application;

[0063] FIG4 (a) is a depth image without median filtering of a real-time online cargo volume recognition method provided by an embodiment of the present application;

[0064] FIG4( b ) is a depth image after median filtering provided by the embodiment of FIG4( a );

[0065] FIG5 (a) is an unaligned color image of a real-time online cargo volume recognition method provided by an embodiment of the present application;

[0066] FIG5( b ) is an aligned color image provided by the embodiment of FIG5( a );

[0067] FIG6 is a color point cloud file generated after alignment in a real-time online cargo volume recognition method provided by an embodiment of the present application;

[0068] FIG7 is a schematic diagram of a specific process before step S3 of a method for real-time online identification of cargo volume provided by an embodiment of the present application;

[0069] FIG8 is a schematic diagram of cargo labeling according to a method for real-time online cargo volume identification provided by an embodiment of the present application;

[0070] FIG9 is a schematic diagram of a specific flow chart of step S3 of a method for real-time online identification of cargo volume provided in an embodiment of the present application;

[0071] FIG10 is a flowchart of a color image cargo recognition method based on a YOLACT model in a real-time online cargo volume recognition method provided in an embodiment of the present application;

[0072] FIG11 is a schematic diagram of cargo identification results of a real-time online cargo volume identification method provided in an embodiment of the present application;

[0073] FIG12 is a flowchart of cargo volume calculation based on aligned images and masks in a real-time online cargo volume recognition method provided by an embodiment of the present application;

[0074] FIG13 is a schematic diagram of a specific flow chart of step S4 of a method for real-time online identification of cargo volume provided in an embodiment of the present application;

[0075] FIG14( a ) is a schematic diagram of a mask before optimization of a real-time online cargo volume identification method provided in an embodiment of the present application;

[0076] FIG14( b ) is a schematic diagram of the optimized mask in the embodiment provided in FIG14( a ). DETAILED DESCRIPTION

[0077] In order to better explain the present application and facilitate understanding, the present application is described in detail below with reference to the accompanying drawings through specific implementation methods.

[0078] Prior to this, in order to facilitate understanding of the technical solution provided by this application, some concepts are introduced below.

[0079] Edge Fitting: Edge fitting involves extracting the boundaries of objects or regions from image or point cloud data. This can be achieved using various edge detection algorithms, such as Canny edge detection and the Sobel operator. These algorithms identify locations with significant changes in grayscale values ​​in an image or depth in a point cloud, thereby generating edge lines or sets of edge points.

[0080] Surface Extraction: Surface extraction involves extracting surface information from point cloud or voxel grid data. This can be achieved using various algorithms, such as point cloud registration, point cloud clustering, and triangulation. These algorithms convert discrete point cloud data into a continuous 3D surface representation, enabling object modeling and analysis.

[0081] Background Separation: Background separation involves separating foreground objects from the background in an image or point cloud. This can be achieved through methods such as background modeling, image segmentation, and threshold segmentation. These methods use pixel color, texture, depth, or other characteristics to separate foreground objects from the background for further analysis and processing.

[0082] As shown in Figure 1, an embodiment of the present application proposes a real-time online cargo volume identification method, including: first, using a camera to capture a depth image and a color image of a set area containing cargo; second, introducing a median filter to remove noise pixels in the depth image, and aligning the color image and the filtered depth image in combination with the camera's internal and external parameters to obtain an aligned color image; third, based on the color and illumination grayscale differences between cargo and non-cargo, using a pre-trained YOLACT model to perform instance segmentation on the aligned color image and generate a cargo mask; finally, using the aligned color image, the filtered depth image, and the cargo mask to perform edge fitting, surface extraction, and background separation of the cargo, thereby determining the cargo volume.

[0083] This application implements depth map visualization and alignment of depth data to RGB data based on an RGB-D camera, cargo identification and instance-level segmentation of RGB images based on the Yolact model, and fitting of object edges and top surfaces based on depth images and mask information. Ultimately, without making major modifications to the assembly line itself, it enables fast, simple, and accurate measurement of cargo volume. As a result, this application has high stability and robustness, and can obtain reliable data regardless of the cargo placement or orientation, while also helping to improve work efficiency.

[0084] To better understand the above technical solutions, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a clearer and more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0085] Specifically, the present application provides a real-time online cargo volume identification method, which includes:

[0086] S1. Capture a depth image and a color image of a set area containing goods through a camera.

[0087] This application uses a Kinect V2 RGB-D camera for shooting. The resolution of the depth map and infrared image obtained is 640*480, and the effective distance is approximately 0.3 meters to 5 meters. The captured RGB image is a four-channel image with a resolution of 1920*1080. The specific parameters of the RGB-D camera are shown in Table 1:

[0088] Table 1 Kinect v2 parameter table

[0089] S2: Introduce a median filter to remove noise pixels from the depth image. Combine the camera's internal and external parameters to align the color image and the filtered depth image to obtain an aligned color image. Referring to Figure 2, step S2 primarily implements median filtering of the depth image, calibrates the internal and external parameters of the depth camera and color camera, and aligns the processed depth image and color image. At the same time, a color point cloud file can also be generated. This is achieved by: generating a point cloud file based on the filtered depth image, and coloring the point cloud using the aligned color image to obtain a color point cloud file.

[0090] Optionally, as shown in FIG3 , step S2 includes:

[0091] S21 . Remove noise pixels in the depth image by introducing a median filter to obtain a filtered depth image.

[0092] Since the original depth map has edge noise and the depth information may not be collected at the center of the depth map, an effective filtering method is needed to make the depth map smoother and eliminate the noise.

[0093] The basic principle of median filtering is to replace the value of a pixel with the median of the surrounding pixels, so that the image can be smoother. The specific definition of median is as follows:

[0094] If x1x2…x n For a set of sequences, arrange them by size: x i1 ≤x i2 ≤x i3 …≤x in .

[0095] Then the median y of the sequence is:

[0096] When the sequence or pixel is one-dimensional, the median filter is implemented by using a one-dimensional sliding window with an odd size. When the window moves, the values ​​in the window are sorted and the median is obtained. Then the median is used to replace the value of the middle pixel in the sliding window to implement the median filter. Let the input sequence be {x i , i∈I}, I is a set or subset of natural numbers, the window length is n and let The output of the filter is: y i =Med{x i}=Med{x i-u …x i …x i+u} (2)

[0097] Among them, x i The median value of a point is only related to the median value of the pixels before and after the sliding window, y i For the sequence x i The median value of .

[0098] Then, the concept of median filtering is extended from one dimension to two dimensions. The simple sliding window used in one dimension will also become a two-dimensional graphic, such as a circle, square, rectangle, etc. Therefore, the two-dimensional median filter is defined as follows:

[0099] Let {x ij , (i, j)∈I 2} represents the gray value of each point in the digital image, A represents the filter window, y ij Indicates that window A is in x ij The median of the points, then:

[0100] Formula (3) shows that the filter window is A at x ij The median expression at point .

[0101] In practice, this application uses the Python OpenCV library for filtering. Figure 4 shows the results before and after median filtering. Before filtering, there is noise at the edges of the image, and many pixels on the surface of the cargo where depth values ​​were not captured due to environmental and camera factors. After median filtering with a kernel size of 5, isolated noise points are eliminated, and the image becomes smoother.

[0102] S22. Use an RGB-D camera to shoot a pre-set calibration plate at multiple locations to obtain a calibration color image and a calibration infrared image.

[0103] We can obtain some pixel values ​​in the image, but we cannot determine their specific correspondence with the real-world length. Camera calibration helps convert these pixel values ​​into actual lengths to accurately calculate the distance between objects and unify the coordinates of the depth map and RGB map into the same coordinate system.

[0104] In a camera imaging system, the following four coordinate systems are usually involved: the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. Generating point clouds and image alignment operations involve converting the world coordinate system and the pixel coordinate system. The world coordinate system can be converted to the camera coordinate system through a rigid body transformation, and the camera coordinate system can be converted to the image coordinate system through a projective perspective transformation. The image coordinate system can be converted to the pixel coordinate system through an affine transformation. The conversion relationship between the world coordinate system and the pixel coordinate system is as follows:

[0105] Among them, (U, V, W) is the physical coordinate of a point in the world coordinate system, (u, v) is the pixel coordinate corresponding to the point in the pixel coordinate system, and Z is the scale factor.

[0106] S23. Calibrate the internal and external parameters of the calibration color image and the infrared image. After excluding images that do not meet the relative position range of RGB with respect to D and images whose calibration result errors differ by a certain threshold, obtain the internal and external parameter matrix of the RGB-D camera.

[0107] Then, the following matrix is ​​called the intrinsic parameter matrix of the camera, which depends on the internal parameters of the camera:

[0108] Where f is the image distance, dX and dY represent the physical lengths of a pixel on the camera plate in the X and Y directions, u0 and v0 represent the coordinates of the center of the camera plate in the pixel coordinate system, and θ represents the angle between the horizontal and vertical edges of the plate (90° indicates no error).

[0109] The matrix is ​​the camera's extrinsic parameter matrix, which is determined by the relative position between the camera coordinate system and the world coordinate system. R represents the rotation matrix, and T represents the translation vector. The single-point distortion-free camera imaging model is as follows:

[0110] The calibration plate is photographed at different positions at the same time. The calculation of the internal and external parameter matrix is ​​mainly divided into three steps: solving the product of the internal parameter matrix and the external parameter matrix, solving the internal parameter matrix, and solving the external parameter matrix. The Camera Calibrator tool in Matlab R2022b can be used to calibrate the internal and external parameters of the camera for color images and infrared images, eliminating images with errors in axis and axis direction and images with excessive errors in calibration results. The internal and external parameter values ​​of the color camera and depth camera are obtained, and the rotation matrix and translation vector from the depth camera to the color camera are calculated. After the calibration is completed, the internal and external parameters of the camera can be obtained. For the external parameters of the same chessboard, the two cameras have the following rigid body transformation relationship: T ir2rgb =T rgb -R ir2rgb T ir (8)

[0111] S24. According to the Zhang Zhengyou calibration principle, the relative positions of the pixels of the color image and the filtered depth image are aligned according to the obtained internal and external parameter matrices to obtain an aligned color image.

[0112] The Zhang Zhengyou camera calibration method, proposed by Professor Zhang Zhengyou in 1998, is based on a single-plane checkerboard grid. This method requires only a single checkerboard grid and then captures color and infrared images of the grid at different angles to calculate the intrinsic and extrinsic parameters of both the color and depth cameras. This method avoids the drawback of traditional calibration methods, which require a high-precision three-dimensional calibration object, while improving the accuracy of the calibration results and simplifying the calibration process.

[0113] Zhang Zhengyou's calibration method fixes the world coordinate system on a checkerboard. Therefore, for any position on the checkerboard, the physical coordinate height is 0, or W = 0. When printing the calibration plate, you can set the size of the checkerboard and fix the direction of the X and Y axes. Then, the physical coordinates of each corner point in the world coordinate system can be calculated as (UVW = 0). By using this information, the camera's internal and external parameter matrix can be calibrated.

[0114] After obtaining the camera's intrinsic and extrinsic parameter matrices, these parameters can be used to align the pixels of the color image to the depth image, and then generate a point cloud file based on the depth image. The aligned color image is used to color the point cloud. The coordinate conversion formula from the pixel coordinate system to the world coordinate system is as follows:

[0115] According to Zhang Zhengyou's calibration principle, using the internal and external parameters of the color camera and depth camera, the pixel coordinates of the color image can be converted first to the world coordinate system and then to the pixel coordinate system of the depth map. This application uses the NumPy library and OpenCV library in Python to implement image alignment and color point cloud generation.

[0116] The original color image and the aligned color image are shown in (a) and (b) in Figure 5 below, and the aligned color point cloud file is shown in Figure 6.

[0117] S3. Use the pre-trained YOLACT model to perform instance segmentation on the aligned color image based on the color, illumination, and grayscale differences between goods and non-goods, and generate a goods mask.

[0118] Optionally, before step S3, as shown in FIG7 and FIG10 , the following steps are further included:

[0119] F31. Obtain a dataset pre-captured by a camera; the dataset includes a depth image with a resolution of 640×480 and a four-channel RGBA color image with a resolution of 1080×1920.

[0120] F32. Use the Labelme tool to label some images in the dataset with target goods to form a coco format dataset, which will serve as the training set for the pre-set YOLACT model.

[0121] F33. Align the remaining images in the dataset except for the training set and use them as the test set for the pre-set YOLACT model. The number of depth images and color images in the training set and test set is 1:1.

[0122] This application uses an RGB-D depth camera for data collection to obtain a data set. The depth map is collected as a PGM file with a resolution of 640×480, and the color map is collected as a four-channel RGBA file with a resolution of 1080×1920 and saved in PNG format. The shooting environment simulates an industrial application environment, and the goods are placed flat on the ground and photographed from above using a camera. The YOLACT model supports small sample training, so about 1,500 photos were collected. As shown in Figure 8, about 1,200 of them were annotated using Labelme as a data set for training the YOLACT model. The remaining 300 photos were aligned and used for testing.

[0123] Labelme is a commonly used image annotation software written in Python that can convert annotation structures into the COCO format. Before annotation, you need to convert the collected RGBA photos to RGB format. For each photo to be annotated, click the "Create Polygons" button on the left to mark the target goods and add the "goods" label. After annotation, you also need to create a label.txt file containing the background and labels used. Finally, convert the dataset to the COCO format supported by YOLACT. This can be achieved in Labelme using labelme2coco.py.

[0124] Optionally, as shown in FIG9 and FIG10 , step S3 includes:

[0125] S31. Train the YOLACT model based on the training set and generate corresponding weight parameters.

[0126] S32. Use the generated weight parameters to perform cargo identification and instance segmentation on the aligned color image to obtain cargo and cargo masks.

[0127] S33. Use a rectangular frame to mark the detected goods, and mark the corresponding confidence level on the rectangular frame.

[0128] S34. Mark the detected goods using the generated goods mask.

[0129] Because the pipeline requires real-time data analysis, yolact is used for instance segmentation. YOLACT (You Only Look At CoefficienTs) is the first truly real-time (greater than or equal to 30fps) instance segmentation model. The YOLACT model uses two branch networks to ensure speed. One of them is the Prediction Head branch, which is used to generate the confidence of each candidate box category, the position of the anchor, and the coefficient of the prototype mask. The other network branch is Protonet, which takes the bottom layer of FPN as input and generates multiple prototype masks. The results of the two networks are segmented and binarized to obtain the final instance segmentation result.

[0130] 1. Preprocessing: The YOLACT model requires preprocessing of the input image, including adjusting it to a fixed size and normalizing it so that its scale range is within a certain range.

[0131] 2. Feature Extractor: The YOLACT model uses a convolutional neural network called ResNet-101 to extract features from the input image. This network consists of five convolutional layers, each with a different size. This design allows it to detect objects of varying sizes.

[0132] 3. Feature Pyramid: After feature extraction, YOLACT uses a feature pyramid mechanism to process image features. This mechanism enables YOLACT to detect objects at different scales to better adapt to objects of different sizes. The P3, P4, and P5 layers of the FPN network correspond to the C3, C4, and C5 layers of the convolutional layers, respectively. P3 serves as the input data for the Protonet network, P6 is generated by performing a convolution operation on P5, and P7 is convolved from P6. These layers have different sizes and receptive fields, allowing them to detect objects of different sizes. At the same time, P3 to P7 are also fed into the Prediction Head.

[0133] 4. Prototype network: The ProtoNet model is used for predicting masks in YOLACT. Its input is P3 and its output is a mask with a dimension of 138×138×k, that is, k prototype masks of size 138×138.

[0134] 5. Prediction Head: The input of the Prediction Head comes from feature maps of different sizes in the FPN. For each feature map, convolution and upsampling operations are performed to increase its resolution to the size of the original input image and generate mask coefficients and bounding boxes.

[0135] 6. Non-maximum suppression: After the Prediction Head branch network, many anchors are generated. Some of these may overlap or have low confidence. To ensure quality, the YOLACT model uses non-maximum suppression (NMS) to filter candidate boxes.

[0136] 7. Mask merging: After the NMS operation, the surviving candidate boxes will undergo a mask merging operation. By linearly combining the results of the two branch networks, the mask of each target object in the image can be obtained.

[0137] 8. Post-processing: The Crop operation is to clear the part outside the mask boundary to zero. The Threshold operation is to binarize the generated mask, and its threshold is 0.5.

[0138] It's worth noting that the experimental environment for this application is based on the Windows 10 operating system, using Python version 3.7, CUDA version 11.7, and CUDNN version 8.7.0. The hardware environment includes an Intel i7 9750H processor, 8GB DDR4 2666MHz*2 memory, and an NVIDIA GeForce GTX 1660 Ti (6GB) graphics card.

[0139] During the network training phase, ResNet101 was used as the backbone network. The YOLACT training parameter values ​​were as follows: the batch parameter was set to 2, which was explicitly indicated when executing the command; the learning rate parameter was set to (6000, 10000); and the maximum number of iterations parameter was set to 12000.

[0140] After the network training is completed, the corresponding weight parameters will be generated.

[0141] Using the generated weight parameters, the aligned RGB images are inspected. The inspection results are shown in Figure 11. Detected objects are identified by rectangular boxes, with the corresponding confidence levels annotated within them. Furthermore, the generated masks are used to mark the detected objects with a red, transparent mark. The YOLACT model performs well in instance segmentation, with its recognition results generally aligning with the edges of the actual goods. However, some detected edges may be missing or inaccurate.

[0142] S4. Use the aligned color image, filtered depth image and cargo mask to perform edge fitting, surface extraction and background separation of the cargo, thereby determining the volume of the cargo. This application mainly optimizes the cargo surface mask based on the cargo mask identified by YOLACT, and calculates the distance from the camera to the upper surface of the cargo and the ground based on the depth map information. Then, the area occupied by a single pixel in the aligned image is calculated based on the camera's internal parameters, and finally the volume of the cargo is calculated. The specific steps are shown in Figure 12.

[0143] Optionally, as shown in FIG13 , step S4 includes:

[0144] S41. Extracting the boundary line of the goods using the aligned color image and the filtered depth image; separating the goods from the background in the aligned color image based on the fitted edge line, and performing surface extraction using the obtained goods mask.

[0145] S42: Optimize the cargo mask based on the extracted depth value of the mask to obtain a new mask.

[0146] Optionally, step S42 includes:

[0147] S421. Initialize a numpy array with the same resolution as the depth map to store the new mask.

[0148] S422: Extract all depth values ​​of the masked portion in the depth map, and calculate the median of all depth values ​​as the upper surface depth value of the cargo mask.

[0149] S423. Obtain a new mask based on the numpy array and the upper surface depth value of the cargo mask.

[0150] During the instance segmentation process in YOLACT, a mask of the cargo surface is generated. However, due to certain errors between the aligned color image, the generated mask, and the depth map, the depth map itself is noisy, and the depth at the center of the camera cannot be captured. Therefore, the generated mask needs to be optimized. The specific steps are as follows: First, initialize a numpy array with the same resolution as the depth map to store the new mask; then, extract the depth values ​​of the masked part in the depth map, and take the median of these values ​​as the depth value of the cargo surface; finally, determine the new mask based on the depth value of the surface. The comparison of the masks before and after optimization is shown in Figure 14 (a) and Figure 14 (b), where g: 1.00 in Figure 14 has the same meaning as in Figure 11.

[0151] S43. Determine the cargo volume based on the new mask, the filtered depth image, the intrinsic parameter matrix of the RGB-D camera, and the pixel information in the new mask.

[0152] Optionally, step S43 includes:

[0153] S431. Calculate the actual distance between any two pixels in the filtered depth image according to the new mask, the filtered depth image, and the intrinsic parameter matrix of the RGB-D camera.

[0154] S432. Select two coordinate points in the longest continuous row in the new mask, calculate the positions of the two coordinate points in the world coordinate system, and calculate the distance between the two coordinate points.

[0155] S433: Divide the distance between the two coordinate points by the number of non-zero pixel points between them to obtain the length of each pixel.

[0156] S434. Obtain the area of ​​each pixel according to the length of each pixel.

[0157] S435 . Calculate the volume of the cargo based on the filtered depth image, the internal parameters of the depth camera, and the number of non-zero pixels in the new mask, the height of the non-zero pixels, and the area of ​​each pixel.

[0158] Based on the optimized mask, depth map, and the intrinsic parameter matrix of the RGB-D camera, the actual distance between any two pixels in the depth map can be calculated. First, two coordinate points are selected from the mask. These two points should be located in a row in the mask, and the length of this row is the longest continuous row in the mask. Then, the positions of these two points in the world coordinate system are calculated, and the distance between them is calculated using the Pythagorean theorem. Since the two points are on the same plane, dividing the distance by the number of pixels yields the length of each pixel. This can help calculate the area of ​​each pixel. Finally, the volume of the cargo can be calculated based on the number of nonzero pixels in the mask, the height, and the area of ​​each pixel.

[0159] In a specific embodiment, the same cargo is placed on the ground in different ways, photographed, and its volume is calculated and compared with the actual volume. The length, width, and height of the cargo are 0.19m, 0.155m, and 0.325m respectively, and the volume is approximately 0.00957m 3 The average error across the 16 data sets was calculated to be 0.65%. These results demonstrate the high stability of the volume measurement algorithm used, enabling reliable data to be obtained regardless of the location of the goods. Furthermore, this method is easy for workers to operate, eliminating the need to position goods in a specific orientation, which helps improve work efficiency.

[0160] In addition, an embodiment of the present application provides a real-time online cargo volume identification system, including:

[0161] The image acquisition module is used to acquire a depth image and a color image of a set area containing the goods through a depth camera.

[0162] The noise filtering and image alignment module is used to introduce a median filter to remove noise pixels in the depth image. It then aligns the color image and the filtered depth image using the camera's internal and external parameters to obtain an aligned color image and color point cloud file.

[0163] The model calling and mask generation module is used to perform instance segmentation on the aligned color image based on the color, illumination and grayscale differences between goods and non-goods using the pre-trained YOLACT model, and generate a goods mask.

[0164] The cargo volume determination module is used to use the aligned color image, the filtered depth image and the cargo mask to perform cargo edge fitting, surface extraction and background separation, thereby determining the cargo volume.

[0165] Furthermore, an embodiment of the present application provides a real-time online cargo volume identification device, comprising: at least one database; an RGB-D depth camera for capturing depth images and color images of a set area containing cargo and transmitting them to the database; and a memory communicatively connected to the at least one database; wherein the memory stores instructions executable by the at least one database, and the instructions are executed by the at least one database so that the at least one database can execute the real-time online cargo volume identification method as described above.

[0166] At the same time, an embodiment of the present application provides a computer-readable medium having computer-executable instructions stored thereon, which, when executed by a processor, implements the above-mentioned real-time online cargo volume identification method.

[0167] In summary, this application discloses a method, system, device, and medium for real-time online identification of cargo volume. This application uses a self-developed RGBD depth camera as a data acquisition tool to capture depth maps and color images of cargo. Subsequently, filtering technology is used to filter out noise in the depth map, and the filtered depth image is aligned with the color image to generate a color point cloud file. Next, the YOLACT model is used to identify objects in the color image, and the edges and upper surfaces of the objects are fitted based on the depth image and mask information, ultimately achieving rapid measurement of cargo volume on the assembly line.

[0168] Based on the above implementation plan, it mainly achieves the following points:

[0169] 1. Use the Labelme tool to label K-level RGB cargo data to achieve instance-level segmentation of small amounts of data using ResNet.

[0170] 2. Based on the obtained RGB image of the product identification, a median filter is introduced to remove the noise pixels in the depth image. The processed RGB image and the depth image are then aligned with RGB-D according to the internal and external parameters of the camera.

[0171] 3. Use YOLACT analysis to obtain a mask file. This mask file is then optimized based on the depth information to fit the cargo surface. Finally, the cargo volume is calculated by calculating the size of individual pixels, the number of non-zero pixels, and the cargo height.

[0172] 4. The actual measurement speed of this method can reach 60fps, and the detection accuracy is higher than 1%, which meets the needs of conventional logistics cargo volume measurement.

[0173] Since the systems / devices described in the above embodiments of this application are systems / devices used to implement the methods of the above embodiments of this application, those skilled in the art will be able to understand the specific structures and variations of these systems / devices based on the methods described in the above embodiments of this application, and therefore, they will not be described in detail here. All systems / devices used in the methods of the above embodiments of this application fall within the scope of protection to be provided by this application.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0176] It should be noted that the word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present application can be implemented by means of hardware comprising several distinct components and by means of a suitably programmed computer. The use of the words first, second, third, etc., is merely for convenience and does not imply any order. These words should be understood as part of the component name.

[0177] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0178] Although the optional embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments after learning the basic creative concept. Therefore, the claims should be interpreted as including the optional embodiments and all changes and modifications that fall within the scope of the present application.

[0179] Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application shall also include these modifications and variations.

Claims

1. A real-time online cargo volume identification method, characterized in that: include: Capturing a depth image and a color image of a set area containing the goods through a camera; A median filter is introduced to remove noise pixels in the depth image, and the color image and the filtered depth image are aligned by combining the internal and external parameters of the camera to obtain an aligned color image. According to the color and illumination grayscale differences between goods and non-goods, the pre-trained YOLACT model is used to perform instance segmentation on the aligned color images and generate a goods mask; The aligned color image, filtered depth image and cargo mask are used to perform cargo edge fitting, surface extraction and background separation, thereby realizing cargo volume determination.

2. The method for real-time online identification of cargo volume according to claim 1, characterized in that: A median filter is introduced to remove the noise pixels in the depth image. The color image and the filtered depth image are aligned by combining the internal and external parameters of the camera. The aligned color image includes: A median filter is introduced to remove noise pixels in the depth image to obtain a filtered depth image; Use an RGB-D camera to shoot the pre-set calibration plate at multiple locations to obtain calibration color images and calibration infrared images; The internal and external parameters of the calibration color image and infrared image are calibrated. After excluding the images that do not meet the relative position range of RGB to D and the images whose calibration result errors differ by a certain threshold, the internal and external parameter matrix of the RGB-D camera is obtained. According to Zhang Zhengyou's calibration principle, the relative positions of the pixels of the color image and the filtered depth image are aligned according to the obtained internal and external parameter matrices to obtain an aligned color image.

3. The real-time online cargo volume identification method according to claim 1, characterized in that: According to the color and illumination grayscale differences between goods and non-goods, the pre-trained YOLACT model is used to perform instance segmentation on the aligned color image and generate the goods mask, which also includes: Get the data set pre-captured by the camera; Use the Labelme tool to label some images in the dataset with target goods to form a coco format dataset as a training set for the pre-set YOLACT model; The remaining images of the dataset except the training set are aligned and used as the test set of the pre-set YOLACT model; The dataset includes depth images with a resolution of 640×480 and four-channel RGBA color images with a resolution of 1080×1920. The number of depth images and color images in the training set and test set is 1:

1.

4. The method for real-time online identification of cargo volume according to claim 3, characterized in that: According to the color and illumination grayscale differences between goods and non-goods, the pre-trained YOLACT model is used to perform instance segmentation on the aligned color images and generate the goods mask including: Train the YOLACT model based on the training set and generate corresponding weight parameters; Using the generated weight parameters, the aligned color images are subjected to cargo identification and instance segmentation to obtain cargo and cargo masks; Use a rectangular box to mark the detected goods and mark the corresponding confidence level on the rectangular box; Use the generated cargo mask to label the detected cargo.

5. The method for real-time online identification of cargo volume according to claim 2, characterized in that: Using the aligned color image, the filtered depth image and the cargo mask to perform cargo edge fitting, surface extraction and background separation, the cargo volume is determined, including: Extract the boundary line of the goods using the aligned color image and the filtered depth image; Separate the goods from the background in the aligned color images based on the fitted edges, Surface extraction is achieved using the resulting cargo mask; The cargo mask is optimized based on the depth value of the extracted mask to obtain a new mask; The cargo volume is determined based on the new mask, the filtered depth image, the intrinsic parameter matrix of the RGB-D camera, and the pixel information in the new mask.

6. The method for real-time online identification of cargo volume according to claim 5, characterized in that: The cargo mask is optimized based on the depth value of the extracted mask, and the new mask includes: Initialize a numpy array with the same resolution as the depth map to store the new mask; Extract all the depth values ​​of the masked part in the depth map, and calculate the median of all the depth values ​​as the upper surface depth value of the cargo mask; Get the new mask based on the numpy array and the upper depth value of the cargo mask.

7. The method for real-time online identification of cargo volume according to claim 5, characterized in that: Determining the cargo volume based on the new mask, the filtered depth image, the intrinsic parameter matrix of the RGB-D camera, and the pixel information in the new mask includes: According to the new mask, the filtered depth image and the intrinsic parameter matrix of the RGB-D camera, the actual distance between any two pixels in the filtered depth image is calculated; Select two coordinate points in the longest continuous row in the mask from the new mask, calculate the positions of the two coordinate points in the world coordinate system, and calculate the distance between the two coordinate points; Divide the distance between two coordinate points by the number of non-zero pixels between them to get the length of each pixel; The area of ​​each pixel is obtained according to the length of each pixel; Based on the filtered depth image, the internal parameters of the depth camera, and the number of non-zero pixels in the new mask, the height of the non-zero pixels, and the area of ​​each pixel, the volume of the cargo is calculated.

8. A real-time online cargo volume identification system, characterized in that: include: An image acquisition module, used for acquiring a depth image and a color image of a set area including the goods through a depth camera; The noise filtering and image alignment module is used to introduce a median filter to remove noise pixels in the depth image, and align the color image and the filtered depth image in combination with the internal and external parameters of the camera to obtain an aligned color image; The model calling and mask generation module is used to perform instance segmentation on the aligned color images and generate a cargo mask based on the color, illumination and grayscale differences between cargo and non-cargo using the pre-trained YOLACT model; The cargo volume determination module is used to use the aligned color image, the filtered depth image and the cargo mask to perform edge fitting, surface extraction and background separation of the cargo, thereby determining the cargo volume.

9. A real-time online cargo volume identification device, characterized in that: include: at least one database; RGB-D depth camera, used to collect depth and color images of a set area containing goods and transmit them to the database; as well as, a memory in communication with the at least one database; The memory stores instructions that can be executed by the at least one database, and the instructions are executed by the at least one database so that the at least one database can execute the real-time online cargo volume identification method according to any one of claims 1 to 7.

10. A computer-readable medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by the processor, the real-time online cargo volume identification method according to any one of claims 1 to 7 is implemented.

Images