Online real-time identification method, system, apparatus, and medium for cargo volume

Abstract

This application relates to an online real-time identification method, system, apparatus, and medium for cargo volume. The method includes the steps of: acquiring depth images and color images of a set area containing cargo using a camera; introducing a median filter to remove noise pixels in the depth image and aligning the color image and the filtered depth image with reference to the camera's intrinsic and extrinsic parameters to obtain an aligned color image; performing instance segmentation on the aligned color image using a pre-trained YOLACT model based on differences in color, lighting, gradation, etc., between cargo and non-cargo to generate a cargo mask; and performing edge fitting, surface extraction, and background separation on the cargo using the aligned color image, the filtered depth image, and the cargo mask to determine the cargo volume. As a result, this application has high stability and robustness, can acquire reliable data regardless of the cargo's placement position and orientation, and contributes to improved work efficiency.

Description

[Technical Field]

[0001] This application relates to the technical field of cargo identification, and more particularly to online real-time identification methods, systems, apparatus and media for cargo volume.

[0002] Cross-references of related applications

[0003] This application claims priority based on a Chinese application filed with the China Patent Office on 21 August 2023, application number 202311047066.1, titled "Online Real-Time Identification Method, System, Apparatus and Medium for Cargo Volume," the entire contents of which are incorporated into this application by reference. [Background technology]

[0004] Currently, sorting operations at distribution centers are shifting from traditional manual methods to automated sorting, but volume calculation remains a major challenge in the industry. Conventional line-scan structured optical methods are costly and difficult to commercialize in small-scale logistics centers.

[0005] Therefore, cargo sorting is mostly based on the intuitive visual judgment of delivery personnel, resulting in the common occurrence of excessively densely packed packages. However, when humans intuitively judge the size of objects, errors are likely to occur. With the increase in the volume of parcel delivery, there is a need to propose effective methods for measuring cargo volume. By accurately measuring cargo volume, delivery personnel can sort packages more appropriately, improving the efficiency of the sorting process.

[0006] Some conventional technologies make a certain contribution to addressing the above-mentioned challenges.

[0007] For example, in "Deep Learning-Based Stereo Endoscopic Three-Dimensional Measurement Method" by Yu Hao et al., published in *Frontiers of Information Technology & Electronic Engineering*, April 2022, a deep learning-based stereo three-dimensional (3D) measurement method was proposed. This method uses a 3D scanner to acquire 3D data of the target, simulates a stereo camera using 3D rendering software, and creates a simulated stereo image dataset to train a disparity estimation model for 3D measurement, thereby providing accurate and reliable size data.

[0008] In "Method for Measuring Corn Height Based on RGB-D Cameras," Qiu Ruicheng et al., in the Journal of Agricultural Machinery, 2017, S1, described a method for measuring corn height using an RGB-D camera. This method first processes a color image using an EPI-based light field rendering algorithm to identify its center position. Then, point cloud data is clustered to extract individual plants, a skeleton detection algorithm is used to extract the highest point of the central leaf of each plant, and finally, the height of the individual plant is measured by calculating the difference in height between the highest point from the ground and the central leaf. Shi Shuai et al. proposed a three-dimensional surface reconstruction and body size measurement system based on a multi-view RGB-D camera. This system uses a Kinect depth camera to acquire point clouds from three different viewpoints (i.e., top view, left side view, and right side view) as the subject walks freely. Registration parameters are obtained using a rectangular prism, and after reconstructing the three local point clouds, the measurement position is identified using point cloud projection distributions in different directions, and key parameters such as height, body width, and waist circumference are accurately measured. Furthermore, this system developed a polar coordinate transformation to improve the accuracy of waist circumference measurement.

[0009] "Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling," by Elise Lachat et al., published in *Remote Sensing*, examines issues such as the applicability and accuracy of RGB-D cameras in close-range 3D modeling. The study also investigates the recent application of the Kinect v2 sensor to the three-dimensional reconstruction of small objects, and examines the advantages and disadvantages of Kinect v2 compared to the original sensor and imaging / measurement methods.

[0010] As described above, the above method is difficult to satisfy the actual need for rapid, simple, and easy cargo volume measurement in terms of system complexity and measurement speed, and is therefore not suitable for application to cargo sorting lines. [Overview of the project] [Problems that the invention aims to solve]

[0011] In light of the shortcomings and deficiencies of the prior art described above, this application provides a method, system, apparatus, and medium for online real-time identification of cargo volume in order to solve the technical problem of difficulty in rapidly, simply, and accurately identifying cargo volume in the prior art. [Means for solving the problem]

[0012] To achieve the above objectives, the main technological modes adopted in this application are as follows:

[0013] In the first phase, embodiments of the present application provide an online real-time identification method for cargo volume, the method comprising: taking depth images and color images of a set area including cargo using a camera; introducing a median filter to remove noise pixels in the depth image, aligning the color image and the filtered depth image with reference to the intrinsic and extrinsic parameters of the camera to obtain an aligned color image; performing instance segmentation on the aligned color image using a pre-trained YOLACT model based on differences in color, lighting, gradation, etc., between cargo and non-cargo to generate a cargo mask; and performing edge fitting, surface extraction, and background separation on the cargo using the aligned color image, the filtered depth image, and the cargo mask to determine the cargo volume.

[0014] The steps of optionally introducing a median filter to remove noise pixels in the depth image, aligning the color image and the filtered depth image while referring to the intrinsic and extrinsic parameters of the camera to obtain an aligned color image include: the step of introducing a median filter to remove noise pixels in the depth image to obtain a filtered depth image; the step of using an RGB-D camera to photograph a pre-set calibration plate from multiple locations to obtain a calibration color image and a calibration infrared image; the step of performing calibration of the intrinsic and extrinsic parameters on the calibration color image and infrared image, removing images that exceed the range of relative positions of RGB with respect to D, and images whose calibration result error exceeds a certain threshold, and then obtaining a matrix of the intrinsic and extrinsic parameters of the RGB-D camera; and the step of using Zhang's calibration method to align the pixel points of the color image with the relative positions of the filtered depth image based on the obtained matrix of intrinsic and extrinsic parameters to obtain an aligned color image.

[0015] The process further includes the steps of: before performing instance segmentation on aligned color images using a pre-trained YOLACT model based on differences in color, lighting, and gradation between cargo and non-cargo, and generating a cargo mask, acquiring a dataset pre-captured by a camera; labeling some images in the dataset with target cargo using the Labelme tool to form the coco dataset, which serves as the training set for a pre-configured YOLACT model; and aligning the remaining images in the dataset, excluding those for the training set, to create a test set for a pre-configured YOLACT model. The dataset includes depth images with a resolution of 640×480 and 4-channel RGBA color images with a resolution of 1080×1920, and the ratio of depth images to color images in the training set and test set is 1:1 in both cases.

[0016] The step of optionally performing instance segmentation on an aligned color image using a pre-trained YOLACT model based on differences in color, lighting, gradation, etc., between cargo and non-cargo, and generating a cargo mask, includes the steps of training the YOLACT model with a training set and generating corresponding weight parameters, performing cargo identification and instance segmentation on an aligned color image using the generated weight parameters to obtain cargo and cargo masks, marking detected cargo with a rectangular frame and labeling the rectangular frame with the corresponding confidence level, and marking detected cargo using the generated cargo mask.

[0017] The steps for optionally performing edge fitting, surface extraction, and background separation on cargo using an aligned color image, a filtered depth image, and a cargo mask to determine the cargo volume include: extracting the edges of the cargo using the aligned color image and the filtered depth image; separating the cargo from the background in the aligned color image based on the fitted edges and performing surface extraction using the obtained cargo mask; optimizing the cargo mask based on the depth values ​​of the extracted mask to obtain a new mask; and determining the cargo volume from the new mask, the filtered depth image, the intrinsic parameter matrix of the RGB-D camera, and the pixel information in the new mask.

[0018] The step of optionally optimizing the cargo mask based on the extracted mask depth values ​​to obtain a new mask includes the steps of: initializing a numpy array with the same resolution as the depth map to store the new mask; extracting all depth values ​​of the mask portion in the depth map and determining the median of all depth values ​​as the top depth value of the cargo mask; and obtaining a new mask based on the numpy array and the top depth value of the cargo mask.

[0019] The optional step of determining cargo volume from a new mask, a filtered depth image, an RGB-D camera intrinsic parameter matrix, and pixel information in the new mask includes: determining the actual distance between any two pixel points in the filtered depth image based on the new mask, the filtered depth image, and the RGB-D camera intrinsic parameter matrix; selecting two coordinate points in the longest consecutive row in the mask from the new mask, calculating the positions of the two coordinate points in the world coordinate system, and calculating the distance between the two coordinate points; obtaining the length of each pixel by dividing the distance between the two coordinate points by the number of non-zero pixel points within this distance; obtaining the area of ​​each pixel based on the length of each pixel; and calculating the cargo volume based on the filtered depth image, the intrinsic parameters of the depth camera, the number of non-zero pixel points in the new mask, the height of the non-zero pixel points, and the area of ​​each pixel.

[0020] In the second aspect, an embodiment of the present application provides an online real-time identification system for cargo volume, which includes: an image acquisition module that acquires depth images and color images of a set area containing cargo using a depth camera; a noise reduction and image alignment module that introduces a median filter to remove noise pixels in the depth image and aligns the color image and the filtered depth image with reference to the intrinsic and extrinsic parameters of the camera to obtain an aligned color image; a model retrieval and mask generation module that performs instance segmentation on the aligned color image using a pre-trained YOLACT model based on differences in color, lighting, gradation, etc., between cargo and non-cargo to generate a cargo mask; and a cargo volume calculation module that uses the aligned color image, the filtered depth image, and the cargo mask to perform edge fitting, surface extraction, and background separation on the cargo to determine the cargo volume.

[0021] In a third aspect, an embodiment of the present application provides an online real-time cargo volume identification apparatus, comprising: at least one database; an RGB-D depth camera configured to acquire depth images and color images of a set area including cargo and transmit the images to the database; and a memory communicatively connected to the at least one database, wherein the memory stores executable instructions for the at least one database, and when the instructions are executed by the at least one database, the at least one database is caused to perform the foregoing online real-time cargo volume identification method.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable medium, which stores computer-executable instructions that, when executed by a processor, implement the foregoing online real-time cargo volume identification method. Effects of the Invention

[0023] The beneficial effects of the present application are as follows: the present application realizes visualization of depth maps based on an RGB-D camera, alignment from depth data to RGB data, cargo identification and instance segmentation of RGB images based on the Yolact model, and fitting of article edges and top surfaces based on depth images and mask information, and finally can quickly, conveniently and accurately measure cargo volume without making major changes to the production line itself. Accordingly, the present application has high stability and robustness, can obtain highly reliable data regardless of the placement position and orientation of cargo, and contributes to improving work efficiency. Brief Description of the Drawings

[0024] [Figure 1] It is a flowchart of an online real-time cargo volume identification method according to an embodiment of the present application. [Figure 2] It is a flowchart of calibration of depth images and color images in the online real-time cargo volume identification method according to an embodiment of the present application. [Figure 3]This is a specific flowchart of step S2 of the online real-time identification method for cargo volume according to an embodiment of this application. [Figure 4] (a) is a depth image of the online real-time identification method for cargo volume according to an embodiment of this application before median filtering. (b) is a depth image of the embodiment of Figure 4(a) after median filtering. [Figure 5] (a) is an unaligned color image of the online real-time identification method for cargo volume according to an embodiment of the present application. (b) is an aligned color image of the embodiment of Figure 5(a). [Figure 6] This is a color point cloud file generated after alignment of the online real-time identification method for cargo volume according to an embodiment of this application. [Figure 7] This is a specific flowchart of the online real-time identification method for cargo volume according to an embodiment of this application, prior to step S3. [Figure 8] This is a schematic diagram of cargo labeling for an online real-time identification method of cargo volume according to an embodiment of this application. [Figure 9] This is a specific flowchart of step S3 of the online real-time identification method for cargo volume according to an embodiment of this application. [Figure 10] This is a flowchart of a color image cargo identification method based on the YOLACT model for online real-time identification of cargo volume according to an embodiment of this application. [Figure 11] This is a schematic diagram of the cargo identification result of the online real-time cargo volume identification method according to an embodiment of this application. [Figure 12] This is a flowchart of the online real-time identification method for cargo volume according to an embodiment of this application, for calculating cargo volume based on aligned images and masks. [Figure 13] This is a specific flowchart of step S4 of the online real-time identification method for cargo volume according to an embodiment of this application. [Figure 14](a) is a schematic diagram of the mask before optimization of the online real-time identification method for cargo volume according to an embodiment of this application. (b) is a schematic diagram of the mask after optimization according to the embodiment of Figure 14(a). [Modes for carrying out the invention]

[0025] To better describe and facilitate understanding this application, specific embodiments will be described in detail below with reference to the drawings.

[0026] Before that, in order to make it easier to understand the technical mode described in this application, let me first introduce some concepts.

[0027] Edge Fitting: Edge fitting refers to the process of extracting edges of objects or regions from image or point cloud data. This can be achieved using various edge detection algorithms, such as Canny edge detection and the Sobel operator. These algorithms can identify locations with large tonal values ​​in an image or large depth changes within a point cloud, and generate edge lines or sets of edge points.

[0028] Surface Extraction: Surface extraction refers to the process of extracting surface information of an object or scene from point cloud data or voxel grid data. This can be achieved using various algorithms, such as point cloud alignment, point cloud clustering, and triangulation. These algorithms transform discrete point cloud data into a continuous three-dimensional surface, enabling object modeling and analysis.

[0029] Background Separation: Background separation refers to the process of separating foreground objects from the background in images or point cloud data. This can be achieved through methods such as background modeling, image segmentation, and threshold segmentation. These methods separate foreground objects from the background based on pixel color, pattern, depth, and other features, and use this information for further analysis and processing.

[0030] As shown in Figure 1, the online real-time identification method for cargo volume according to the embodiment of this application first acquires depth images and color images of a set area including cargo using a camera, then introduces a median filter to remove noise pixels in the depth image, aligns the color image and the filtered depth image while referring to the intrinsic and extrinsic parameters of the camera to obtain an aligned color image, then performs instance segmentation on the aligned color image using a pre-trained YOLACT model based on differences in color, lighting, gradation, etc., between cargo and non-cargo to generate a cargo mask, and finally performs edge fitting, surface extraction, and background separation on the cargo using the aligned color image, filtered depth image, and cargo mask to determine the cargo volume.

[0031] This application enables the visualization of depth maps based on an RGB-D camera and alignment from depth data to RGB data, cargo identification and instance segmentation of RGB photographs based on a Yolact model, and fitting of item edges and top surfaces based on depth images and mask information. Ultimately, it enables rapid, simple, and accurate measurement of cargo volume without making significant changes to the work line itself. As a result, this application offers high stability and robustness, enabling reliable data acquisition regardless of the cargo's placement and orientation, and contributing to improved work efficiency.

[0032] To facilitate understanding of the above-described technical embodiments, exemplary embodiments of this application will be described in more detail below with reference to the drawings. While the drawings show exemplary embodiments of this application, it can be realized in various forms and is not limited to the embodiments described herein. Rather, these embodiments are intended to provide a clearer and more complete understanding of this application and to fully convey its scope to those skilled in the art.

[0033] Specifically, the online real-time identification method for cargo volume according to this application includes the following steps:

[0034] Step S1: The camera acquires depth images and color images of the set area, including the cargo.

[0035] In this application, images were captured using an RGB-D camera of model number Kinect V2, with a resolution of 640×480 for the obtained depth map and infrared image, and an effective measurement range of approximately 0.3 meters to 5 meters. The captured RGB photograph is a 4-channel photograph with a resolution of 1920×1080. The specific parameters of the RGB-D camera are shown in Table 1.

[0036] [Table 1]

[0037] Step S2: A median filter is introduced to remove noise pixels from the depth image, and the color image and the filtered depth image are aligned while referring to the intrinsic and extrinsic parameters of the camera to obtain an aligned color image. As shown in Figure 2, Step S2 mainly implements median filtering of the depth map, calibrates the intrinsic and extrinsic parameters of the depth camera and color camera, and aligns the processed depth map image and color image. It is also possible to generate a color point cloud file, which is obtained by the following steps: A point cloud file is generated based on the filtered depth image, and the point cloud is colorized using the color image after alignment to obtain a color point cloud file.

[0038] Optionally, as shown in Figure 3, step S2 includes the following steps.

[0039] Step S21: A filtered depth image is obtained by introducing a median filter to remove noisy pixels in the depth image.

[0040] Because the original depth map contains edge noise, and depth information may not be obtainable at the center of the depth map, an effective filtering method is needed to smooth the depth map and remove the noise within it.

[0041] The basic principle of median filtering is to smooth an image by replacing the value of a particular pixel with the median value of the surrounding pixels. The specific definition of the median value is as follows:

[0042] x1x 2··· x n If we consider this as a sequence of numbers and arrange them in order of magnitude, then x i1 ≤x i2 ≤x i3··· ≤x in This is the result.

[0043] The median y of this sequence is

[0044]

number

[0045] This is the result.

[0046] When the sequence or pixels are one-dimensional, median filtering is implemented using a one-dimensional sliding window with an odd window size. During window movement, the values ​​within the window are sorted to find the median value, and this median value is used to replace the pixel value in the center of the sliding window, thereby achieving median filtering.

[0047]

number

[0048] Let I be the set or subset of natural numbers, let n be the length of the window, and let u = (n-1) / 2. Then the output of the filter can be expressed by the following formula.

[0049] [Math.]]

[0050] Point x i has a median value that is only related to the median values of pixels before and after the sliding window, and y i is the median value of the sequence x i .

[0051] Then, the concept of median filter processing is extended from one dimension to two dimensions. The simple sliding window used in one dimension also becomes a two-dimensional shape such as a circle, a square, or a rectangle. Accordingly, the two-dimensional median filter is defined as follows.

[0052] [Math.]]

[0053] be the gradation value at each point of the digital image, let A be the filter window, and let y ij be the median value at point x ij of window A, then

[0054] [Math.]]

[0055] .

[0056] The above formula is an expression representing the median value at point x ij of the filter window A.

[0057] In a specific example, this application performed filtering using the OpenCV library in Python. Figure 4 shows the effect before and after median filtering. Before filtering, noise is present at the edges of the photograph, and many pixels on the cargo surface cannot obtain depth values ​​due to environmental factors or camera issues. Median filtering with a filter kernel size of 5 removes isolated noise points in the surrounding area, resulting in a smoother image.

[0058] Step S22: Using an RGB-D camera, a pre-configured calibration plate is photographed from multiple locations to obtain a color image and an infrared image for calibration.

[0059] While it's possible to obtain some pixel values ​​from a photograph, it's impossible to determine their specific correspondence to real-world lengths. Camera calibration helps convert these pixel values ​​into actual lengths, accurately calculate distances between objects, and unify the coordinates of depth maps and RGB photographs into the same coordinate system.

[0060] A camera imaging system typically involves four coordinate systems: the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. Point cloud generation and image alignment operations involve transformations between the world coordinate system and the pixel coordinate system. The world coordinate system can be transformed into the camera coordinate system by rigid body transformation, and the camera coordinate system can be transformed into the image coordinate system by perspective projection transformation. The image coordinate system can be transformed into the pixel coordinate system by affine transformation. The transformation relationships between the world coordinate system and the pixel coordinate system are as follows:

[0061]

number

[0062] (U,V,W) are the physical coordinates of a point in the world coordinate system, (u,v) are the pixel coordinates in the corresponding pixel coordinate system, and Z is the scale factor.

[0063] Step S23: Calibration of intrinsic and extrinsic parameters is performed on the color image and infrared image for calibration. After removing images that exceed the range of relative positions of RGB with respect to D, and images where the error in the calibration result exceeds a certain threshold, a matrix of intrinsic and extrinsic parameters of the RGB-D camera is obtained.

[0064] The following matrix is ​​called the camera's intrinsic parameter matrix, and the intrinsic parameter matrix depends on the camera's intrinsic parameters.

[0065]

number

[0066] f represents the imaging distance, dX and dY represent the physical length of one pixel on the camera photosensitive plate in the X and Y directions, respectively. u0 and v0 represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system, respectively, and θ represents the angle between the horizontal and vertical sides of the photosensitive plate (90° is error-free).

[0067]

number

[0068] The matrix is ​​the camera's external parameter matrix. The external parameter matrix is ​​determined by the relative position between the camera coordinate system and the world coordinate system. R represents the rotation matrix, and T represents the translation vector. That is, the ideal pinhole camera model is as follows:

[0069]

number

[0070] The step of simultaneously photographing the calibration plate at different positions and calculating the intrinsic and extrinsic parameter matrices mainly involves three steps: calculating the product of the intrinsic parameter matrix and the extrinsic parameter matrix, calculating the intrinsic parameter matrix, and calculating the extrinsic parameter matrix. Using the Camera Calibrator tool in MATLAB R2022b, the intrinsic and extrinsic parameters are calibrated for the camera's color and infrared images, and photographs with axial and axial misalignment and large errors in the calibration results are removed. This allows obtaining the intrinsic and extrinsic parameter values ​​for the color camera and depth camera, and further calculating the rotation matrix and translation vector from the depth camera to the color camera. After calibration is complete, the camera's intrinsic and extrinsic parameters can be obtained. For the extrinsic parameters of the same checkerboard pattern, the two cameras have the following rigid body transformation relationship.

[0071]

number

[0072]

number

[0073] Step S24: Based on the matrix of intrinsic and extrinsic parameters obtained by Zhang's calibration method, the relative positions of the pixel points in the color image and the filtered depth image are aligned to obtain an aligned color image.

[0074] The Zhang Camera Calibration Method is a camera calibration method based on a single-planar checkerboard pattern, proposed by Professor Zhang Zhengyou in 1998. This method allows for the calculation of intrinsic and extrinsic parameters of a color camera and depth camera simply by preparing a single checkerboard pattern and capturing color and infrared images of it from different angles. This method avoids the need for high-precision three-dimensional calibration objects required in conventional calibration methods, improving the accuracy of calibration results while simultaneously simplifying the calibration process.

[0075] In Zhang's calibration method, the world coordinate system is fixed to a checkerboard pattern, so the height of the physical coordinates at any point on the checkerboard pattern becomes 0, i.e., W=0. When printing the calibration plate, the size of the checkerboard pattern can be freely set, and the directions of the X and Y axes can be determined. This makes it possible to calculate the physical coordinates (UVW=0) in the world coordinate system for each corner. Using this information, it becomes possible to calibrate the matrix of the camera's intrinsic and extrinsic parameters.

[0076] After obtaining the intrinsic and extrinsic parameter matrices of the camera, these parameters are used to align the pixel points of the color image to the depth image. A point cloud file is then generated from the depth image, and the point cloud is colorized using the aligned color image. The coordinate transformation formula from the pixel coordinate system to the world coordinate system is as follows:

[0077]

number

[0078] The Zhang calibration method allows for the conversion of pixel coordinates in a color image to the world coordinate system, and then to the depth map's pixel coordinate system, using the intrinsic and extrinsic parameters of a color camera and a depth camera. In this application, photographic alignment and color point cloud generation were achieved using the Python NumPy and OpenCV libraries.

[0079] The original color image and the color image after alignment are shown in Figure 5(a) and (b), respectively, and the color point cloud file after alignment is shown in Figure 6.

[0080] Step S3: Based on differences in color, lighting, and gradation between cargo and non-cargo, instance segmentation is performed on the aligned color image using a pre-trained YOLACT model to generate a cargo mask.

[0081] Optionally, before step S3, the following steps may be further included, as shown in Figures 7 and 10.

[0082] Step F31: Obtain the dataset pre-captured by the camera. The dataset includes a depth image with a resolution of 640 x 480 and a 4-channel RGBA color image with a resolution of 1080 x 1920.

[0083] Step F32: Use the Labelme tool to label some of the images in the dataset with the target cargo, create the coco dataset, and use it as a training set for the pre-configured YOLACT model.

[0084] Step F33: Align the remaining images in the dataset, excluding those used for the training set, to create a test set for the pre-configured YOLACT model. The ratio of depth images to color images in both the training and test sets is 1:1.

[0085] In this application, data is acquired using an RGB-D depth camera to obtain a dataset. A PGM file of a depth map with a resolution of 640×480 and a 4-channel RGBA file of a color image with a resolution of 1080×1920 are acquired and stored in PNG format. The shooting environment simulates an industrial application environment, with cargo placed flat on the ground and photographed from above with the camera. Since the YOLACT model can handle training with a small number of shots, approximately 1500 photographs were acquired, and as shown in Figure 8, approximately 1200 of these were labeled with Labelme and used as the training dataset for the YOLACT model. The remaining 300 were aligned and used for testing.

[0086] Labelme is a general-purpose image labeling software that can convert labeling structures to COCO format, and it is written in Python. Before labeling, the acquired RGBA images need to be converted to RGB format. For each photo to be labeled, you can click the "Create Polygons" button on the left to mark the target cargo and add the "goods" label. After labeling is complete, you need to create a label.txt file containing the background and the labels used. Finally, to convert the dataset to YOLACT-compatible COCO format, this can be done in Labelme using labelme2coco.py.

[0087] Optionally, as shown in Figures 9 and 10, step S3 includes the following steps:

[0088] Step S31: Train the YOLACT model with the training set and generate the corresponding weight parameters.

[0089] Step S32: Cargo identification and instance segmentation are performed on the color image aligned using the generated weight parameters to obtain the cargo and cargo mask.

[0090] Step S33: Mark the detected cargo with a rectangular frame and label the rectangular frame with the corresponding confidence level.

[0091] Step S34: Mark the detected cargo using the generated cargo mask.

[0092] To meet the workflow requirements for real-time data analysis, instance segmentation is performed using YOLACT. YOLACT (You Only Look At CoefficienTs) is the first truly real-time (over 30fps) instance segmentation model. The YOLACT model employs two branched networks to ensure speed. One is the Prediction Head branched network, which generates confidence levels, anchor positions, and prototype mask coefficients for each candidate box class. The other branched network is Protonet, which takes the lowest layer of the FPN as input and generates multiple prototype masks. The final instance segmentation result is obtained by segmenting and binarizing the results of these two networks.

[0093] 1. Preprocessing: The YOLACT model requires preprocessing of the input image. This preprocessing includes adjusting the image to a fixed size and normalizing it, ensuring that the image scale range falls within a certain range.

[0094] 2. Feature Extractor: The YOLACT model employs a convolutional neural network called ResNet-101 for feature extraction from the input image. This network contains five convolutional layers, each with a different size. This structure makes it possible to detect objects of different sizes.

[0095] 3. Feature Pyramid: After feature extraction is complete, YOLACT processes image features using a feature pyramid mechanism. This mechanism allows YOLACT to detect targets at different scales and is applicable to targets of various sizes. Layers P3, P4, and P5 of the FPN network correspond to layers C3, C4, and C5 of the convolutional layers, respectively. P3 functions as input data for the Protonet network, P6 is generated by a convolutional operation from P5, and P7 is obtained by a convolution from P6. Because these layers have different sizes and receptive fields, they can detect objects of different sizes. P3-P7 are also input to the Prediction Head.

[0096] 4. Prototype Network: The prediction mask in YOLACT employs a ProtoNet model, with P3 as its input and a mask as its output. The dimensions are 138 × 138 × k, i.e., k prototype masks of size 138 × 138.

[0097] 5. Prediction Head: The input to the Prediction Head is obtained from feature maps of different sizes within the FPN. Convolution and upsampling are performed on each feature map to increase its resolution to the size of the original input image, and a mask coefficient and bounding box are generated.

[0098] 6. Non-Maximum Suppression: After passing through the Prediction Head branch network, numerous anchors are generated. These may include duplicates or anchors with low confidence levels. To ensure quality, the YOLACT model uses non-maximum suppression (NMS) to select candidate boxes.

[0099] 7. Mask Integration: After NMS processing, the mask integration operation is performed on the remaining candidate boxes. By linearly combining the results of the two branch networks, a mask for each target object in the image is obtained.

[0100] 8. Post-processing: The Crop operation zeros out the parts of the image that are not mask edges. The Threshold operation binarizes the generated mask, with a threshold of 0.5.

[0101] In particular, the experiments described in this application were conducted using the Windows 10 operating system, with Python version 3.7, CUDA version 11.7, and CUDNN version 8.7.0. Regarding hardware, the processor was an Intel i7 9750H, the memory was 8GB DDR4 2666MHz*2, and the graphics card was an NVIDIA GeForce GTX 1660 Ti (6GB).

[0102] During the network training phase, ResNet101 was used as the backbone network. The YOLACT training parameter values ​​were as follows: The batch parameter was set to 2 and explicitly expressed when the command was executed. The learning rate parameter was set to (6000, 10000), and the maximum iteration parameter was set to 12000.

[0103] After network training is complete, the corresponding weight parameters are generated.

[0104] The generated weight parameters are used to detect objects in the aligned RGB photograph. As shown in Figure 11, detected objects are marked with rectangular frames, and the confidence level corresponding to each frame is labeled. Furthermore, the detected objects are marked transparently in red using the generated mask. The YOLACT model demonstrates excellent performance in instance segmentation, and its identification results closely match the edges of the actual cargo. However, some edges in the detected images may be missing or have errors.

[0105] Step S4: Using the aligned color image, filtered depth image, and cargo mask, edge fitting, surface extraction, and background separation are performed on the cargo to determine the cargo volume. This application primarily optimizes the cargo top surface mask based on the cargo mask identified by YOLACT, and calculates the distance from the camera to the cargo top surface and the ground based on the depth map information. Then, the area occupied by a single pixel point in the aligned photograph is calculated based on the camera's internal parameters, and finally the cargo volume is calculated. The specific steps are shown in Figure 12.

[0106] Optionally, as shown in Figure 13, step S4 includes the following steps.

[0107] Step S41: Extract cargo edges using the aligned color image and filtered depth image. Based on the fitted edges, the cargo is separated from the background in the aligned color image, and surface extraction is performed using the resulting cargo mask.

[0108] Step S42: Optimize the cargo mask based on the depth values ​​of the extracted masks to obtain a new mask.

[0109] Optionally, step S42 includes the following steps:

[0110] Step S421: Initialize a numpy array with the same resolution as the depth map to store the new mask.

[0111] Step S422: Extract all depth values ​​from the mask portion of the depth map, and determine the median of all depth values ​​as the top depth value of the cargo mask.

[0112] Step S423: Obtain a new mask based on the numpy array and the top depth value of the cargo mask.

[0113] The YOLACT instance segmentation process generates a mask of the cargo surface. However, there is a certain error between the color image after alignment, the generated mask, and the depth map, and the depth map itself contains noise and cannot capture the depth of the camera's center. Therefore, optimization of the generated mask is necessary. The specific steps are as follows: First, initialize a numpy array with the same resolution as the depth map to store the new mask. Then, extract the depth values ​​of the masked area in the depth map and find the median of these values ​​as the depth value of the cargo's top surface. Finally, determine a new mask based on the top surface depth value. Compare the masks before and after optimization as shown in Figures 14(a) and (b). g:1.00 in Figure 14 has the same meaning as in Figure 11.

[0114] Step S43: Determine the cargo volume from the new mask, filtered depth image, intrinsic parameter matrix of the RGB-D camera, and pixel information in the new mask.

[0115] Optionally, step S43 includes the following steps:

[0116] Step S431: Based on the new mask, the filtered depth image, and the intrinsic parameter matrix of the RGB-D camera, the actual distance between any two pixel points in the filtered depth image is determined.

[0117] Step S432: Select the two coordinate points in the longest consecutive row in the new mask, calculate the positions of the two coordinate points in the world coordinate system, and calculate the distance between the two coordinate points.

[0118] Step S433: The distance between two coordinate points is divided by the number of non-zero pixel points within this distance to obtain the length of each pixel.

[0119] Step S434: Obtain the area of ​​each pixel based on the length of each pixel.

[0120] Step S435: Calculate the cargo volume based on the filtered depth image, the intrinsic parameters of the depth camera, the number of non-zero pixel points in the new mask, the height of the non-zero pixel points, and the area of ​​each pixel.

[0121] Based on the optimized mask, depth map, and intrinsic parameter matrix of the RGB-D camera, the actual distance between any two pixel points in the depth map can be calculated. First, two coordinate points are selected from the mask. These two points are located in the same row in the mask, and their row length is the longest consecutive row in the mask. Then, the positions of these two points in the world coordinate system are calculated, and the distance between them is calculated based on the Pythagorean theorem. Since these two points are on the same plane, the length of each pixel can be obtained by dividing the distance by the number of pixels. This allows the area of ​​each pixel to be calculated. Finally, the cargo volume can be calculated from the number of non-zero pixel points in the mask, their height, and the area of ​​each pixel.

[0122] In one specific example, the same cargo was placed on the ground in different orientations, photographed, and its volume was calculated and then compared to the actual volume. The cargo's length, width, and height were 0.19m, 0.155m, and 0.325m, respectively, and its volume was approximately 0.00957m³. 3 The average error of the 16 sets of data obtained by calculation was 0.65%. The results indicate that the volume measurement algorithm used is highly stable and can obtain reliable data regardless of the cargo's placement. Furthermore, this method is easy for workers to operate, does not require cargo to be positioned in a specific direction, and contributes to improved work efficiency.

[0123] Furthermore, the embodiment of this application provides an online real-time identification system for cargo volume, and this system includes the following modules.

[0124] The image acquisition module uses a depth camera to acquire depth images and color images of a designated area, including the cargo.

[0125] The noise reduction and image alignment module applies a median filter to remove noisy pixels from the depth image, aligns the color image and the filtered depth image while referencing the camera's intrinsic and extrinsic parameters, and obtains aligned color image and color point cloud files.

[0126] The model invocation and mask generation module performs instance segmentation on aligned color images using a pre-trained YOLACT model based on differences in color, lighting, and gradation between cargo and non-cargo, and generates a cargo mask.

[0127] The cargo volume calculation module uses aligned color images, filtered depth images, and cargo masks to perform edge fitting, surface extraction, and background separation on the cargo to determine its volume.

[0128] Furthermore, embodiments of this application provide an online real-time identification device for cargo volume, which includes at least one database, an RGB-D depth camera that acquires depth images and color images of a set area containing cargo and transmits them to the database, and a memory that communicates with the at least one database, wherein the memory stores commands that the at least one database can execute, and when a command is executed on the at least one database, the at least one database is instructed to execute the online real-time identification method for cargo volume described above.

[0129] Furthermore, embodiments of this application provide a computer-readable medium that stores computer-executable commands that, when executed by a processor, realize the online real-time identification method of cargo volume described above.

[0130] As described above, this application discloses an online real-time identification method, system, apparatus, and medium for cargo volume. This application uses a proprietary RGBD depth camera as a data acquisition tool to capture depth maps and color images of cargo. Subsequently, noise is removed from the depth map using filtering techniques, and the filtered depth image is aligned with the color image to generate a color point cloud file. Next, the articles in the color image are identified using a YOLACT model, and the edges and top surfaces of the articles are fitted based on the depth image and mask information, ultimately achieving real-time rapid measurement of cargo volume on the work line.

[0131] Based on the above embodiment, the following points were mainly achieved.

[0132] 1. Label RGB cargo data using the Labelme tool and perform instance segmentation using small amounts of data in ResNet.

[0133] 2. Based on the obtained cargo identification RGB photograph, a median filter is applied to remove noise pixels in the depth map, and the processed RGB photograph and depth map are RGB-D aligned based on the camera's intrinsic and extrinsic parameters.

[0134] 3. A mask file is obtained using YOLACT analysis. Then, the mask file generated based on depth information is optimized and fitted to the cargo surface. Finally, the cargo volume is calculated by calculating information such as the size of single pixels, the number of non-zero pixels, and the height of the cargo.

[0135] 4. This method achieves a measurement speed of 60 fps and a detection accuracy exceeding 1%, meeting the requirements for cargo volume measurement in normal logistics.

[0136] The systems / apparatus described in the above embodiments of this application are systems / apparatus for carrying out the methods of the above embodiments of this application. Therefore, a person skilled in the art can understand the specific structure and variations of such systems / apparatus based on the methods described in the above embodiments of this application. Accordingly, a detailed explanation is omitted here. All systems / apparatus employed in the methods of the above embodiments of this application fall within the scope that this application seeks to protect.

[0137] Those skilled in the art should understand that embodiments of this application may be provided as methods, systems, or computer program products. Accordingly, this application may adopt embodiments of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, this application may adopt embodiments of computer program products implemented on one or more computer-available storage media (including, but not limited to, magnetic disk storage devices, CD-ROMs, optical storage devices, etc.) containing computer-available program code.

[0138] This application will be described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of this application. Each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program commands.

[0139] The term "one" or "one" placed before a component does not preclude the existence of multiple such components. This application may be implemented by hardware and a properly programmed computer comprising multiple different components. The use of terms 1, 2, 3, etc., is for convenience only and does not indicate any order. These terms may be understood as part of a component name.

[0140] In this specification, terms such as “one embodiment,” “several embodiments,” “example,” “specific example,” or “several examples” mean that the specific features, structures, materials, or properties described with reference to the embodiment or example are included in at least one embodiment or example of this application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or properties described may be combined in appropriate ways in any one or more embodiments or examples. In addition, a person skilled in the art may combine or link different embodiments or examples and features of different embodiments or examples described herein, provided that they do not conflict with each other.

[0141] While selectable embodiments of this application have been described, those skilled in the art, knowing the basic creative concepts, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted as encompassing the selectable embodiments and all changes and modifications that fall within the scope of this application.

[0142] A person skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Accordingly, if such changes and modifications to this application fall within the scope of this application and its equivalent art, this application should be considered to include such changes and modifications.

Claims

1. The steps include: acquiring depth images and color images of a set area including cargo using a camera; The process involves applying a median filter to remove noise pixels from the depth image, aligning the color image and the filtered depth image while referencing the camera's intrinsic and extrinsic parameters, and obtaining an aligned color image. Based on differences in color, lighting, and gradation between cargo and non-cargo, the process involves performing instance segmentation on an aligned color image using a pre-trained YOLACT model to generate a cargo mask. The process includes the steps of determining the cargo volume by performing edge fitting, surface extraction, and background separation on the cargo using an aligned color image, a filtered depth image, and a cargo mask. A method for online real-time identification of cargo volume, characterized by the following features.

2. The steps of applying a median filter to remove noise pixels in the depth image, aligning the color image and the filtered depth image while referring to the camera's intrinsic and extrinsic parameters to obtain an aligned color image are as follows: The steps include: obtaining a filtered depth image by introducing a median filter to remove noisy pixels in the depth image; The steps include: using an RGB-D camera to photograph a pre-set calibration plate from multiple locations to obtain a calibration color image and a calibration infrared image; The process involves calibrating the intrinsic and extrinsic parameters of a color image and an infrared image for calibration, removing images that exceed the range of relative positions of RGB with respect to D, and images where the calibration result error exceeds a certain threshold, and then obtaining a matrix of the intrinsic and extrinsic parameters of an RGB-D camera. The method includes the step of aligning the relative positions of the pixel points of the color image and the filtered depth image based on the matrix of intrinsic and extrinsic parameters obtained by Zhang's calibration method to obtain an aligned color image. The online real-time identification method for cargo volume according to feature 1.

3. Before the step of generating a cargo mask, perform instance segmentation on the aligned color image using a pre-trained YOLACT model based on differences in color, lighting, and gradation between cargo and non-cargo, The steps include acquiring a dataset that has been pre-captured by a camera, The process involves using the Labelme tool to label some images in the dataset with the target cargo, forming the coco dataset, and using it as a training set for a pre-configured YOLACT model. The process further includes the step of aligning the remaining images in the dataset, excluding those used for training, to create a test set for a pre-configured YOLACT model. The dataset includes depth images with a resolution of 640 x 480 and 4-channel RGBA color images with a resolution of 1080 x 1920, with a 1:1 ratio of depth images to color images in both the training and test sets. The online real-time identification method for cargo volume according to feature 1.

4. The step of generating a cargo mask by performing instance segmentation on an aligned color image using a pre-trained YOLACT model based on differences in color, lighting, and gradation between cargo and non-cargo is as follows: The steps include training the YOLACT model with the training set and generating the corresponding weight parameters, Using the generated weight parameters, cargo identification and instance segmentation are performed on the aligned color image to obtain the cargo and cargo mask. The steps include marking the detected cargo with a rectangular frame and labeling the rectangular frame with the corresponding confidence level, The steps include marking the detected cargo using the generated cargo mask. The online real-time identification method for cargo volume according to feature 3.

5. The steps to determine the cargo volume by performing edge fitting, surface extraction, and background separation on the cargo using aligned color images, filtered depth images, and cargo masks are as follows: Steps include extracting cargo edges using aligned color images and filtered depth images, The steps include: separating the cargo from the background in an aligned color image based on fitted edges, and achieving surface extraction using the resulting cargo mask; The process involves optimizing the cargo mask based on the depth values ​​of the extracted masks to obtain a new mask, and The process includes the step of determining the cargo volume from a new mask, a filtered depth image, an intrinsic parameter matrix of an RGB-D camera, and pixel information in the new mask. The online real-time identification method for cargo volume according to feature 2.

6. The step of optimizing the cargo mask based on the depth values ​​of the extracted masks and obtaining a new mask is: The steps include initializing a numpy array with the same resolution as the depth map to store the new mask, The steps include: extracting all depth values ​​from the mask portion of the depth map and determining the median of all depth values ​​as the top depth value of the cargo mask; The process includes the step of obtaining a new mask based on the NumPy array and the top depth value of the cargo mask. The online real-time identification method for cargo volume according to feature 5.

7. The step of determining the cargo volume from a new mask, filtered depth image, RGB-D camera intrinsic parameter matrix, and pixel information in the new mask is: The steps include determining the actual distance between any two pixel points in the filtered depth image based on a new mask, the filtered depth image, and the intrinsic parameter matrix of the RGB-D camera, The steps include selecting two coordinate points in the longest consecutive row within a new mask, calculating the positions of the two coordinate points in the world coordinate system, and calculating the distance between the two coordinate points. The steps include: obtaining the length of each pixel by dividing the distance between two coordinate points by the number of non-zero pixel points within that distance; A step of obtaining the area of ​​each pixel based on the length of each pixel, The process includes the step of calculating the cargo volume based on a filtered depth image, intrinsic parameters of the depth camera, the number of non-zero pixel points in a new mask, the height of the non-zero pixel points, and the area of ​​each pixel. The online real-time identification method for cargo volume according to feature 5.

8. An image acquisition module that acquires depth images and color images of a set area including cargo using a depth camera, A noise reduction and image alignment module that uses a median filter to remove noisy pixels in the depth image, aligns the color image and the filtered depth image while referencing the camera's intrinsic and extrinsic parameters, and obtains an aligned color image. A model call and mask generation module that performs instance segmentation on a color image aligned using a pre-trained YOLACT model based on differences in color, lighting, and gradation between cargo and non-cargo, and generates a cargo mask. Includes a cargo volume calculation module that uses aligned color images, filtered depth images, and cargo masks to perform edge fitting, surface extraction, and background separation on the cargo to determine the cargo volume. An online real-time identification system for cargo volume, characterized by the following features.

9. At least one database and An RGB-D depth camera that acquires depth images and color images of a set area including cargo and transmits them to a database, A memory that is connected to the at least one database, The memory stores commands that the at least one database can execute, and when a command is executed by the at least one database, it causes the at least one database to execute the online real-time identification method for cargo volume described in any one of claims 1 to 7. An online real-time identification device for cargo volume, characterized by the following features.

10. The processor stores a computer-executable command that, when executed, realizes the online real-time identification method of cargo volume described in any one of claims 1 to 7. A computer-readable medium characterized by the following:

Images