Vision-based method for simultaneous automatic identification and size measurement of multiple pieces of airline luggage
By constructing RGB-D images through calibration using depth and visible light cameras, and combining preprocessing and morphological algorithms, a 3D point cloud is reconstructed and its dimensions are calculated. This solves the problem of automatic identification and measurement of multiple pieces of luggage, and achieves efficient and accurate measurement of aviation luggage.
Patent Information
- Application Number
- CN202111464456.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-03
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-03
AI Technical Summary
In aviation baggage measurement scenarios, it is difficult to achieve automated and accurate identification and measurement when multiple pieces of baggage appear at the same time. Especially when the transmission channel is under high pressure, manual intervention is required for separation, which affects unmanned and automated transportation.
A depth camera and a visible light camera are used for calibration to construct RGB-D images. Instance segmentation is performed by combining various preprocessing algorithms and morphological algorithms to reconstruct 3D point clouds and calculate luggage dimensions. The RANSAC algorithm is used to fit a plane for coordinate transformation, ultimately achieving automatic identification and size measurement of multiple pieces of luggage.
It enables simultaneous measurement of multiple pieces of luggage, handles complex placement arrangements, ensures measurement accuracy and speed, is suitable for various airport environments, and has high robustness and real-time recognition capabilities.
Smart Images

Figure CN114140514B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer vision, and in particular relates to a vision-based method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage. Background Art
[0002] With the rapid development of the logistics and passenger transport industries, the storage and transportation costs of air cargo and luggage are constantly increasing. Therefore, improving storage space utilization and transportation efficiency has become an important way to reduce aviation costs. In order to improve storage space utilization, a large number of optimization algorithms are applied to cargo placement; in order to reduce heavy manual handling, handling robots are applied to related scenarios. Whether it is optimizing storage space or using robots for handling, accurate three-dimensional perception of air luggage is indispensable.
[0003] In the field of air logistics, different sizes of goods have different charges and transportation methods, so accurate measurement of air cargo is crucial. In actual application scenarios, manual measurement and visual inspection are often used. Among them, manual measurement takes a lot of time, which can easily cause blockage of logistics channels and fatigue of measurement personnel; while visual inspection is heavily dependent on the measurement experience of measurement personnel. Due to the strong subjectivity of measurement personnel, it is easy to cause large measurement errors.
[0004] In the field of air passenger transport, for example, when passengers check in their luggage at an airport, it is necessary to accurately measure the size of the passengers' luggage to determine the luggage transfer channel and avoid oversized or oversized luggage blocking the transfer channel. Similarly, this scenario also requires automated and accurate luggage measurement. In addition, the measurement results can also be used to improve the stacking efficiency of transfer carts and use robots to grab luggage instead of heavy manual handling.
[0005] However, in the above-mentioned air cargo and baggage measurement scenarios, due to the high pressure in the transmission channel, multiple pieces of luggage often appear in the measurement work area at the same time. For example, in the crawler transmission system of air luggage, luggage often contacts each other. When this happens, manual intervention is often required to separate the luggage, which is not conducive to the unmanned and automated process of future air luggage transportation.
[0006] Therefore, it is very necessary to use computer vision to simultaneously perceive and measure multiple pieces of aviation cargo and baggage. Summary of the invention
[0007] In order to solve the above problems, the object of the present invention is to provide an identification and measurement method that can quickly identify and measure multiple pieces of luggage in a working area and has strong robustness to the posture and contact mode of the luggage.
[0008] In order to achieve the above object, the method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage based on vision provided by the present invention comprises the following steps performed in sequence:
[0009] 1) Calibrate the intrinsic parameters of the depth camera and the extrinsic parameters between the depth camera and the visible light camera, then use the calibrated depth camera and visible light camera to obtain the depth image and visible light image of the working plane and the luggage placed on it, and align the depth image to the visible light image coordinate system; then use the aligned depth image and visible light image to construct the RGB-D image;
[0010] 2) Using a variety of preprocessing algorithms to process the aligned depth image to improve imaging quality and obtain a preprocessed depth image;
[0011] 3) performing linear threshold segmentation on the preprocessed depth image obtained in step 2) to obtain a binary image having a foreground area and a background area, and then using a morphological algorithm to perform instance segmentation on multiple pieces of luggage that may exist in the foreground area of the binary image to obtain instance segmentation labels;
[0012] 4) Reconstruct the 3D point cloud of the multiple pieces of luggage using the RGB-D image obtained in step 1) and the intrinsic parameters of the depth camera, and map the instance segmentation labels obtained in step 3) to the reconstructed 3D point cloud of the multiple pieces of luggage;
[0013] 5) Using the three-dimensional point clouds of the multiple pieces of luggage reconstructed in step 4), the sizes of the multiple pieces of luggage are calculated respectively, thereby realizing automatic recognition and size measurement of the multiple pieces of luggage in the foreground area.
[0014] In step 1), the method for calibrating the internal parameters of the depth camera and the external parameters between the depth camera and the visible light camera is: establishing an imaging geometric model of the depth camera and the visible light camera, and solving the geometric model parameters, that is, using a calibration method to calibrate the internal parameters of the depth camera and the external parameters between the depth camera and the visible light camera; wherein the calibration method adopts Zhang's calibration method and active vision camera calibration method;
[0015] The working plane includes a carrier including a conveyor belt or the ground;
[0016] The internal parameters of the depth camera include the depth image width, the depth image height, the depth camera horizontal focal length fx, the depth camera vertical focal length fy, the actual length cx of the depth image pixel, the actual width cy of the depth image pixel, and the actual length depth_scale represented by 1 depth unit in the world coordinate system.
[0017] In step 2), the specific steps of using a plurality of preprocessing algorithms to process the aligned depth image to obtain the preprocessed depth image are as follows:
[0018] 2.1) First, the aligned depth image obtained in step 1) is processed using a spatial filtering algorithm;
[0019] The spatial filtering algorithm uses a spatial edge-preserving filter, which is used to perform two round-trip filtering in the X and Y directions of the aligned depth image, and calculates its one-dimensional exponential moving average at the same time. The expression is:
[0020]
[0021] Among them, α represents the smoothing coefficient, Y represents the instantaneous value of the filter at a certain position, and St represents the one-dimensional exponential moving average of the filter within a certain time period t; when α=1, it means that the filter does not filter; when α=0, it means that all historical values are used for filtering; at the same time, the filter sets a threshold δ, and stops filtering when the current state of the filter differs greatly from the previous state;
[0022] 2.2) Then, a temporal filter is used to temporally filter the depth image after the spatial filtering, and a persistence filter is used to alleviate the holes in the depth image due to occlusion, uneven texture and multiple matching;
[0023] 2.3) Then, a hole filling algorithm is used to reduce the holes in the temporally filtered depth image, and the depth values on the left side of the holes are used to fill the holes with less than 64 pixels in the depth image. Then, a noise region removal algorithm is used to remove isolated noise regions with less than 64 pixels in the depth image. Finally, the preprocessed depth image is obtained.
[0024] In step 3), the preprocessed depth image obtained in step 2) is subjected to linear threshold segmentation to obtain a binary image having a foreground area and a background area, and then a morphological algorithm is used to perform instance segmentation on multiple pieces of luggage that may exist in the foreground area of the binary image. The specific steps of obtaining instance segmentation labels are as follows:
[0025] 3.1) Use the distance from the depth camera to the working plane as a threshold to separate the foreground and background areas of the preprocessed depth image and binarize them to obtain a binary image; in this step, since the imaging plane and the working plane are not necessarily parallel, the average depth value of the four corners of the preprocessed depth image is used as the depth value, and the foreground and background segmentation thresholds of the linear transition are fitted;
[0026] 3.2) Finally, the morphological algorithm is used to separate the multiple pieces of luggage that may exist in the foreground area, where the size of the erosion and dilation kernel is selected to be 3% of the depth image resolution width.
[0027] In step 4), the method of reconstructing the three-dimensional point cloud of the multiple pieces of luggage using the RGB-D image obtained in step 1) and the internal parameters of the depth camera, and mapping the instance segmentation labels obtained in step 3) to the reconstructed three-dimensional point cloud of the multiple pieces of luggage is:
[0028] Using the intrinsic parameters of the depth camera obtained in step 1), the RGB-D image is mapped into the three-dimensional space using the mapping function shown in the following formula to obtain the three-dimensional point cloud of multiple pieces of luggage:
[0029]
[0030] Among them, x, y and z represent the coordinates of any point in the 3D point cloud in the 3D coordinate system, d represents the value of the corresponding point in the depth image, depth_scale represents the actual length represented by 1 depth unit in the world coordinate system, u and v represent the coordinates of the point in the image coordinate system, fx represents the horizontal focal length of the depth camera, fy represents the vertical focal length of the depth camera, cx represents the actual length of the depth image pixel, and cy represents the actual width of the depth image pixel; in addition, the RGB information corresponding to each point in the 3D point cloud and the instance segmentation label obtained by step 3) are also mapped at the same time.
[0031] In step 5), the specific steps of using the three-dimensional point clouds of the multiple pieces of luggage reconstructed in step 4) to respectively calculate the sizes of the multiple pieces of luggage, thereby realizing automatic recognition and size measurement of the multiple pieces of luggage in the foreground area are as follows:
[0032] 5.1) Use the calibrated depth camera to capture the depth image of the working plane, and then process the depth image of the working plane according to the methods of steps 1), 2), and 4) to obtain the three-dimensional point cloud of the working plane;
[0033] 5.2) Use the RANSAC algorithm to fit the maximum plane on the three-dimensional point cloud of the above working plane, and obtain its three-dimensional plane parameters a, b, c and d. The equation of the working plane is expressed as:
[0034] ax+by+cz+d=0
[0035] 5.3) The three-dimensional point clouds of the multiple pieces of luggage obtained in step 4) are rotated and translated to align with the working plane to facilitate the calculation of the size of the luggage. The translation transformation matrix is expressed as:
[0036]
[0037] The rotation angle α along the x-axis is:
[0038]
[0039] The rotation angle β along the y-axis is:
[0040]
[0041] The rotation transformation matrix along the x-axis is expressed as:
[0042]
[0043] The rotation transformation matrix along the y-axis is expressed as:
[0044]
[0045] 5.4) Since the coordinate value z of the three-dimensional point cloud of the multiple pieces of luggage obtained in step 4) is the opposite of the actual value, the coordinate value z needs to be inverted, and its transformation matrix is expressed as:
[0046]
[0047] After the above coordinate transformation, the three-dimensional point clouds of the above-mentioned multiple pieces of luggage are finally aligned with the working plane, thereby obtaining the transformed three-dimensional point clouds of the multiple pieces of luggage;
[0048] 5.5) Using the instance segmentation labels obtained in step 3), extract the 3D point cloud of each piece of luggage from the 3D point clouds of the multiple pieces of luggage after the above transformation, then extract the x and y coordinates of all points from the 3D point cloud point set of each piece of luggage, calculate the feature vector of the 3D point cloud on the xy plane, obtain the maximum and minimum values along the direction of the maximum feature vector and in the perpendicular direction, subtract the corresponding coordinate values to obtain the length and width of a piece of luggage; then, take the distance between the maximum value of the 3D point cloud of a piece of luggage and the working plane as the height of the luggage; thereby, obtain the size of each piece of luggage in the foreground area.
[0049] The advantages of the vision-based method for simultaneous automatic identification and size measurement of multiple pieces of airline baggage provided by the present invention are: 1. It can realize the simultaneous measurement of multiple pieces of airline baggage appearing in the camera's field of view, and can handle a variety of complex placement methods, such as the situation where angular contact and edge contact between luggage are not conducive to instance segmentation. 2. The calculation is simple and the measurement speed is fast. On the basis of achieving real-time identification and measurement, the measurement accuracy is guaranteed. 3. There are no special requirements for the model and installation method of the equipment, and it can be applied to automatic identification and measurement tasks in most airport environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 The present invention provides a flowchart of a method for automatically identifying and measuring the dimensions of multiple pieces of aviation luggage based on vision.
[0051] Figure 2 It is a measurement result of the vision-based method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage provided by the present invention.
[0052] Figure 3 It is another measurement result of the vision-based method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage provided by the present invention. DETAILED DESCRIPTION
[0053] The vision-based method for simultaneous automatic identification and size measurement of multiple pieces of aviation luggage provided by the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0054] like Figure 1 As shown, the vision-based method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage provided by the present invention comprises the following steps performed in sequence:
[0055] 1) Calibrate the intrinsic parameters of the depth camera and the extrinsic parameters between the depth camera and the visible light camera, then use the calibrated depth camera and visible light camera to obtain the depth image and visible light image of the working plane and the luggage placed on it, and align the depth image to the visible light image coordinate system; then use the aligned depth image and visible light image to construct the RGB-D image;
[0056] The method for calibrating the internal parameters of the depth camera and the external parameters between the depth camera and the visible light camera is: establishing an imaging geometric model of the depth camera and the visible light camera, and solving the geometric model parameters, that is, using a calibration method to calibrate the internal parameters of the depth camera and the external parameters between the depth camera and the visible light camera; wherein the calibration method adopts Zhang's calibration method and active vision camera calibration method;
[0057] The working plane includes a carrier including a conveyor belt or the ground;
[0058] The internal parameters of the depth camera include the depth image width, the depth image height, the depth camera horizontal focal length fx, the depth camera vertical focal length fy, the actual length cx of the depth image pixel, the actual width cy of the depth image pixel, and the actual length depth_scale represented by 1 depth unit in the world coordinate system.
[0059] 2) Using a variety of preprocessing algorithms to process the aligned depth image to improve imaging quality and obtain a preprocessed depth image;
[0060] The specific steps are as follows:
[0061] 2.1) First, the aligned depth image obtained in step 1) is processed using a spatial filtering algorithm;
[0062] The spatial filtering algorithm uses a spatial edge-preserving filter, which is used to perform two round-trip filtering in the X and Y directions of the aligned depth image, and calculates its one-dimensional exponential moving average value, which is expressed as:
[0063]
[0064] Among them, α represents the smoothing coefficient, Y represents the instantaneous value of the filter at a certain position, and St represents the one-dimensional exponential moving average of the filter in a certain time period t; when α = 1, it means that the filter does not filter; when α = 0, it means that all historical values are used for filtering; at the same time, the filter sets a threshold δ, and stops filtering when the current state of the filter differs greatly from the previous state, so as to ensure that the edge features of the depth image are not destroyed by the filter. Specifically, the number of filtering times of the spatial edge preservation filter is set to 2, the smoothing coefficient α is set to 0.5, and the threshold δ for enabling smoothing filtering is set to 20.
[0065] 2.2) Then, a temporal filter is used to temporally filter the depth image after the spatial filtering, and a persistence filter is used to alleviate the holes in the depth image due to occlusion, uneven texture and multiple matching. The smoothing coefficient α of the temporal filter is set to 0.4, the threshold δ for enabling the smoothing filter is set to 20, the window size of the persistence filter is set to 4, and the minimum number of valid frames is set to 2.
[0066] 2.3) After completing the above two filterings, holes still appear in the depth image due to the same reasons mentioned above. Therefore, a hole filling algorithm is used to reduce the holes in the depth image after time filtering. The present invention uses the depth value on the left side of the hole to fill the holes in the depth image with a number of pixels less than 64; then the noise area removal algorithm is used to remove the isolated noise areas in the depth image with a number of pixels less than 64; finally, the preprocessed depth image is obtained.
[0067] 3) performing linear threshold segmentation on the preprocessed depth image obtained in step 2) to obtain a binary image having a foreground area and a background area, and then using a morphological algorithm to perform instance segmentation on multiple pieces of luggage that may exist in the foreground area of the binary image to obtain instance segmentation labels;
[0068] The specific steps are as follows:
[0069] 3.1) Use the distance from the depth camera to the working plane as a threshold to separate the foreground area and the background area of the preprocessed depth image and binarize them to obtain a binary image. Since the optical axis of the camera cannot be absolutely perpendicular to the working plane, the depth value of the background area in the preprocessed depth image is not uniform. Therefore, the present invention uses the average depth value of the four corners of the preprocessed depth image as the depth value, and fits the linear transition foreground area and background area segmentation threshold;
[0070] 3.2) Finally, the morphological algorithm is used to open the operation to separate the multiple pieces of luggage that may exist in the foreground area, where the size of the erosion and dilation kernel can be adjusted according to the contact angle and minimum measurement size of the objects in the actual working scene. In this step, the size of the erosion and dilation kernel is selected as 3% of the depth image resolution width.
[0071] 4) Reconstruct the 3D point cloud of the multiple pieces of luggage using the RGB-D image obtained in step 1) and the intrinsic parameters of the depth camera, and map the instance segmentation labels obtained in step 3) to the reconstructed 3D point cloud of the multiple pieces of luggage;
[0072] Using the intrinsic parameters of the depth camera obtained in step 1), the RGB-D image is mapped into the three-dimensional space using the mapping function shown in the following formula to obtain the three-dimensional point cloud of multiple pieces of luggage:
[0073]
[0074] Among them, x, y and z represent the coordinates of any point in the 3D point cloud in the 3D coordinate system, d represents the value of the corresponding point in the depth image, depth_scale represents the actual length represented by 1 depth unit in the world coordinate system, u and v represent the coordinates of the point in the image coordinate system, fx represents the horizontal focal length of the depth camera, fy represents the vertical focal length of the depth camera, cx represents the actual length of the depth image pixel, and cy represents the actual width of the depth image pixel; in addition, the RGB information corresponding to each point in the 3D point cloud and the instance segmentation label obtained by step 3) are also mapped at the same time.
[0075] 5) Using the three-dimensional point clouds of the multiple pieces of luggage reconstructed in step 4), the sizes of the multiple pieces of luggage are calculated respectively, thereby realizing automatic recognition and size measurement of the multiple pieces of luggage in the foreground area.
[0076] The specific steps are as follows:
[0077] 5.1) Use the calibrated depth camera to capture the depth image of the working plane, and then process the depth image of the working plane according to the methods of steps 1), 2), and 4) to obtain the three-dimensional point cloud of the working plane;
[0078] 5.2) Use the RANSAC algorithm to fit the maximum plane on the three-dimensional point cloud of the above working plane, and obtain its mathematical plane parameters a, b, c and d. The working plane equation can be expressed as:
[0079] ax+by+cz+d=0
[0080] 5.3) The three-dimensional point clouds of the multiple pieces of luggage obtained in step 4) are rotated and translated to align with the working plane to facilitate the calculation of the size of the luggage. The translation transformation matrix is expressed as:
[0081]
[0082] The rotation angle α along the x-axis is:
[0083]
[0084] The rotation angle β along the y-axis is:
[0085]
[0086] The rotation transformation matrix along the x-axis is expressed as:
[0087]
[0088] The rotation transformation matrix along the y-axis is expressed as:
[0089]
[0090] 5.4) Since the coordinate value z of the three-dimensional point cloud of the multiple pieces of luggage obtained in step 4) is the opposite of the actual value, the coordinate value z needs to be inverted, and its transformation matrix is expressed as:
[0091]
[0092] After the above coordinate transformation, the three-dimensional point clouds of the above-mentioned multiple pieces of luggage are finally aligned with the working plane, thereby obtaining the transformed three-dimensional point clouds of the multiple pieces of luggage;
[0093] 5.5) Using the instance segmentation labels obtained in step 3), extract the 3D point cloud of each piece of luggage from the 3D point clouds of the multiple pieces of luggage after the above transformation, then extract the x and y coordinates of all points from the 3D point cloud point set of each piece of luggage, calculate the feature vector of the 3D point cloud on the xy plane, obtain the maximum and minimum values along the direction of the maximum feature vector and in the perpendicular direction, subtract the corresponding coordinate values to obtain the length and width of a piece of luggage; then, take the distance between the maximum value of the 3D point cloud of a piece of luggage and the working plane as the height of the luggage; thereby, obtain the size of each piece of luggage in the foreground area.
[0094] The effect of the vision-based method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage provided by the present invention can be further illustrated by the following experimental results.
[0095] The experiment of the present invention is based on RealSense TM D455 depth camera, which can output depth image and visible light image at the same time, adopts the method of the present invention to measure common aviation logistics goods such as cartons and suitcases to verify the effectiveness of the method of the present invention. The experimental results are shown in Table 1. Figure 2 , Figure 3 shown.
[0096] Table 1. Statistical data of the measurement results of the present invention
[0097]
[0098]
[0099] Through the above experimental comparison results, it can be concluded that the method of the present invention can meet the real-time measurement tasks in the production environment and has a high measurement accuracy. For the situation where multiple pieces of luggage are in contact, the method of the present invention can basically achieve instance segmentation of objects and has a high measurement success rate. Therefore, the method of the present invention has high robustness for complex measurement scenarios that are prone to occur in production scenarios, and can achieve real-time segmentation measurement for multiple pieces of luggage to be measured that may exist in the foreground area.
Claims
1. A method for automatically identifying and measuring the dimensions of multiple pieces of airline luggage based on vision, characterized in that: The vision-based method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of airline luggage comprises the following steps performed in sequence: 1) Calibrate the intrinsic parameters of the depth camera and the extrinsic parameters between the depth camera and the visible light camera, then use the calibrated depth camera and visible light camera to obtain the depth image and visible light image of the working plane and the luggage placed on it, and align the depth image to the visible light image coordinate system; then use the aligned depth image and visible light image to construct the RGB-D image; 2) Using a variety of preprocessing algorithms to process the aligned depth image to improve imaging quality and obtain a preprocessed depth image; 3) performing linear threshold segmentation on the preprocessed depth image obtained in step 2) to obtain a binary image having a foreground area and a background area, and then using a morphological algorithm to perform instance segmentation on multiple pieces of luggage that may exist in the foreground area of the binary image to obtain instance segmentation labels; 4) Reconstruct the 3D point cloud of the multiple pieces of luggage using the RGB-D image obtained in step 1) and the intrinsic parameters of the depth camera, and map the instance segmentation labels obtained in step 3) to the reconstructed 3D point cloud of the multiple pieces of luggage; 5) using the three-dimensional point clouds of the multiple pieces of luggage reconstructed in step 4), respectively calculating the sizes of the multiple pieces of luggage, thereby realizing automatic recognition and size measurement of the multiple pieces of luggage in the foreground area; In step 2), the specific steps of using a plurality of preprocessing algorithms to process the aligned depth image to obtain the preprocessed depth image are as follows: 2.1) First, the aligned depth image obtained in step 1) is processed using a spatial filtering algorithm; The spatial filtering algorithm uses a spatial edge-preserving filter, which is used to perform two round-trip filtering in the X and Y directions of the aligned depth image, and calculates its one-dimensional exponential moving average at the same time. The expression is: Among them, α represents the smoothing coefficient, Y represents the instantaneous value of the filter at a certain position, and St represents the one-dimensional exponential moving average of the filter within a certain time period t; when α=1, it means that the filter does not filter; when α=0, it means that all historical values are used for filtering; at the same time, the filter sets a threshold δ, and stops filtering when the current state of the filter differs greatly from the previous state; 2.2) Then, a temporal filter is used to temporally filter the depth image after the spatial filtering, and a persistence filter is used to alleviate the holes in the depth image due to occlusion, uneven texture and multiple matching; 2.3) Then, a hole filling algorithm is used to reduce the holes in the temporally filtered depth image, and the depth values on the left side of the holes are used to fill the holes with less than 64 pixels in the depth image. Then, a noise region removal algorithm is used to remove isolated noise regions with less than 64 pixels in the depth image. Finally, the preprocessed depth image is obtained.
2. The method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage based on vision according to claim 1, characterized in that: In step 1), the method for calibrating the internal parameters of the depth camera and the external parameters between the depth camera and the visible light camera is: establishing an imaging geometric model of the depth camera and the visible light camera, and solving the geometric model parameters, that is, using a calibration method to calibrate the internal parameters of the depth camera and the external parameters between the depth camera and the visible light camera; The calibration method adopts Zhang's calibration method and active vision camera calibration method; The working plane includes a carrier including a conveyor belt or the ground; The internal parameters of the depth camera include the depth image width, the depth image height, the depth camera horizontal focal length fx, the depth camera vertical focal length fy, the actual length cx of the depth image pixel, the actual width cy of the depth image pixel, and the actual length depth_scale represented by 1 depth unit in the world coordinate system.
3. The method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage based on vision according to claim 1, characterized in that: In step 3), the preprocessed depth image obtained in step 2) is subjected to linear threshold segmentation to obtain a binary image having a foreground area and a background area, and then a morphological algorithm is used to perform instance segmentation on multiple pieces of luggage that may exist in the foreground area of the binary image. The specific steps of obtaining instance segmentation labels are as follows: 3.1) Use the distance from the depth camera to the working plane as a threshold to separate the foreground and background areas of the preprocessed depth image and binarize them to obtain a binary image; in this step, since the imaging plane and the working plane are not necessarily parallel, the average depth value of the four corners of the preprocessed depth image is used as the depth value, and the foreground and background segmentation thresholds of the linear transition are fitted; 3.2) Finally, the morphological algorithm is used to separate the multiple pieces of luggage that may exist in the foreground area, where the size of the erosion and dilation kernel is selected to be 3% of the depth image resolution width.
4. The method for simultaneously automatically identifying and measuring the dimensions of multiple pieces of aviation luggage based on vision according to claim 1, characterized in that: In step 4), the method of reconstructing the three-dimensional point cloud of the multiple pieces of luggage using the RGB-D image obtained in step 1) and the internal parameters of the depth camera, and mapping the instance segmentation labels obtained in step 3) to the reconstructed three-dimensional point cloud of the multiple pieces of luggage is: Using the intrinsic parameters of the depth camera obtained in step 1), the RGB-D image is mapped into the three-dimensional space using the mapping function shown in the following formula to obtain the three-dimensional point cloud of multiple pieces of luggage: Among them, x, y and z represent the coordinates of any point in the 3D point cloud in the 3D coordinate system, d represents the value of the corresponding point in the depth image, depth_scale represents the actual length represented by 1 depth unit in the world coordinate system, u and v represent the coordinates of the point in the image coordinate system, fx represents the horizontal focal length of the depth camera, fy represents the vertical focal length of the depth camera, cx represents the actual length of the depth image pixel, and cy represents the actual width of the depth image pixel; in addition, the RGB information corresponding to each point in the 3D point cloud and the instance segmentation label obtained by step 3) are also mapped at the same time.
5. The method for simultaneous automatic identification and size measurement of multiple pieces of aviation luggage based on vision according to any one of claims 1 to 4, characterized in that: In step 5), the specific steps of using the three-dimensional point clouds of the multiple pieces of luggage reconstructed in step 4) to respectively calculate the sizes of the multiple pieces of luggage, thereby realizing automatic recognition and size measurement of the multiple pieces of luggage in the foreground area are as follows: 5.1) Use the calibrated depth camera to capture the depth image of the working plane, and then process the depth image of the working plane according to the methods of steps 1), 2), and 4) to obtain the three-dimensional point cloud of the working plane; 5.2) Use the RANSAC algorithm to fit the maximum plane on the three-dimensional point cloud of the above working plane, and obtain its three-dimensional plane parameters a, b, c and d. The equation of the working plane is expressed as: ax+by+cz+d=0 5.3) The three-dimensional point clouds of the multiple pieces of luggage obtained in step 4) are rotated and translated to align with the working plane to facilitate the calculation of the size of the luggage. The translation transformation matrix is expressed as: The rotation angle α along the x-axis is: The rotation angle β along the y-axis is: The rotation transformation matrix along the x-axis is expressed as: The rotation transformation matrix along the y-axis is expressed as: 5.4) Since the coordinate value z of the three-dimensional point cloud of the multiple pieces of luggage obtained in step 4) is the opposite of the actual value, the coordinate value z needs to be inverted, and its transformation matrix is expressed as: After the above coordinate transformation, the three-dimensional point clouds of the above-mentioned multiple pieces of luggage are finally aligned with the working plane, thereby obtaining the transformed three-dimensional point clouds of the multiple pieces of luggage; 5.5) Using the instance segmentation labels obtained in step 3), extract the 3D point cloud of each piece of luggage from the 3D point clouds of the multiple pieces of luggage after the above transformation, then extract the x and y coordinates of all points from the 3D point cloud point set of each piece of luggage, calculate the feature vector of the 3D point cloud on the xy plane, obtain the maximum and minimum values along the direction of the maximum feature vector and in the perpendicular direction, subtract the corresponding coordinate values to obtain the length and width of a piece of luggage; then, take the distance between the maximum value of the 3D point cloud of a piece of luggage and the working plane as the height of the luggage; thereby, obtain the size of each piece of luggage in the foreground area.
