Luggage sorting and loading subsystem and luggage sorting and loading method

Abstract

The invention discloses a luggage sorting and loading subsystem and a luggage sorting and loading method, the subsystem is connected with a check-in system, a conveying and loading system and a central management system, and the subsystem comprises a hyperspectral imaging module, a spectral space preprocessing pipeline, a feature extraction and fusion module, a decision module and an execution module. The hyperspectral imaging module collects luggage spectral data, the spectral space preprocessing pipeline performs smooth dimension reduction on an original spectrum, the feature extraction and fusion module fuses RGB image features and hyperspectral features, the decision module predicts a destination through a split network, and the execution module controls a mechanical arm to sort. Based on hyperspectral imaging and multi-feature fusion, the problem of a traditional recognition system in a complex environment is solved through spectral information acquisition, multi-feature fusion convolutional neural network recognition classification and a multi-modal verification mechanism, spectral and spatial feature adaptive weighted fusion is achieved, the error recognition rate is reduced, and the recognition efficiency is improved. And the identification accuracy and the processing efficiency of the airport luggage processing system are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of baggage identification, and in particular to an airport baggage sorting and loading subsystem and a baggage sorting and loading method based on hyperspectral imaging and multi-feature fusion. Background Art

[0002] With the continued growth of air passenger traffic, airport baggage handling systems face the dual challenges of improving recognition accuracy and handling efficiency. Traditional baggage identification technology primarily relies on barcode or RFID scanning, but in practice, recognition accuracy can drop significantly due to factors such as labels being stained, folded, or partially obscured. Furthermore, existing computer vision solutions based on RGB images, while performing well under ideal lighting conditions, struggle to distinguish between luggage made of similar materials, such as a black hardshell suitcase and a black plastic box.

[0003] The limitations of existing technologies are primarily reflected in the following aspects: First, the reliance of traditional computer vision systems or barcodes on a single feature makes it difficult to capture the intrinsic properties of materials. While multispectral methods can provide richer information, their insufficient band resolution makes it difficult to address the recognition needs of various complex materials. Second, existing HSI-CNN (Hyperspectral Image Convolutional Neural Network) recognition systems are mostly trained offline and lack support for real-time spectral correction, resulting in insufficient adaptability in dynamic environments. Finally, existing automated sorting systems often lack effective coupling with recognition systems, resulting in a lengthy decision-making process from recognition to execution, making it difficult to further improve efficiency.

[0004] In summary, the existing baggage identification technology has obvious deficiencies in terms of accuracy and real-time performance in complex environments. There is an urgent need to develop more efficient and reliable new technologies to meet the growing demand for airport baggage handling. Summary of the Invention

[0005] The purpose of the present invention is to overcome the limitations of RGB or barcode systems in the existing technology, such as their inability to obtain intrinsic material properties, poor adaptability to dynamic environments, and low system coupling, and to provide an airport baggage sorting and loading subsystem and baggage sorting and loading method based on hyperspectral imaging and multi-feature fusion.

[0006] In order to achieve the above object, the solution of the present invention is: A baggage sorting and loading subsystem, deployed in the baggage sorting area between the check-in and baggage drop-off areas and the conveying and loading system, and connected to the check-in system, the conveying and loading system, and the central management system, includes a hyperspectral imaging module, a spectral spatial preprocessing pipeline, a feature extraction and fusion module, a decision module, and an execution module; The hyperspectral imaging module is used to collect hyperspectral data of luggage and obtain the spectral characteristics of the luggage material; The spectral spatial preprocessing pipeline is used to smooth and reduce the dimensionality of the raw spectral data collected by the hyperspectral imaging module; The feature extraction and fusion module is based on a deep learning algorithm and includes an RGB feature extraction channel, a hyperspectral feature processing channel, and a feature fusion module. The RGB feature extraction channel is used to extract RGB image features captured by the RGB camera deployed in the baggage sorting area. The hyperspectral feature processing channel is used to process the hyperspectral features processed by the spectral spatial preprocessing pipeline. The feature fusion module is used to fuse the features of the RGB feature extraction channel and the hyperspectral feature processing channel. The decision module predicts the destination through a split network based on the results of the feature extraction and fusion modules; The execution module is used to control the robotic arm to complete the sorting operation according to the results of the decision module.

[0007] Furthermore, the hyperspectral imaging module is a hyperspectral sensor array that collects hyperspectral data of luggage.

[0008] Furthermore, the hyperspectral sensor array uses a push-broom hyperspectral camera with a working band covering 400-2500nm and including 240 spectral channels; the hyperspectral camera collects a three-dimensional hyperspectral data cube of each luggage item.

[0009] in, Represents a three-dimensional hyperspectral data cube; W and H represent the width and height of the image respectively. B=240, indicating the number of spectral wave channels; R is a real number, indicating that the spectral intensity or reflectance value of each pixel in the data cube is a real number; Each pixel Represents the reflectivity of a single pixel at the coordinate position (x, y) in space in different bands; The spatial coordinate indicating the current spectral channel number; T stands for transpose, which converts a row vector into a column vector.

[0010] Furthermore, the spectral spatial preprocessing pipeline performs Savitzky-Golay filtering on the spectral curve acquired by the hyperspectral camera, and eliminates spectral noise by using the Savitzky-Golay filter. The noise elimination formula is:

[0011] in, represents the spectrum value after noise removal after Savitzky-Golay filter processing; represents the original spectrum value; N represents the number of original spectral data points; is the filter window size; is the filter coefficient, is the normalization factor; To sum the index variables, the value range is arrive , used to iterate over the different terms in the summation process.

[0012] Furthermore, the RGB feature extraction channel uses an improved ResNet-50 network, a convolutional neural network that extracts spatial features from RGB images; The input RGB image is I, and the features extracted by the improved ResNet-50 network are , =ResNet-50(I), where I is the input RGB image, which is a three-dimensional tensor with a shape of (H, W, 3), where H and W are the height and width of the image respectively, 3 represents the three RGB channels, and ResNet-50 represents the improved ResNet-50 network.

[0013] Furthermore, the feature fusion module uses a weighted summation method to dynamically fuse spectral and other features. The cross-attention mechanism dynamically adjusts the weights of hyperspectral and RGB features by calculating the relationship between them. Based on the characteristics of the input data, it adaptively assigns more weight to more discriminative features and dynamically adjusts the importance of different sensor inputs through learnable weights. The feature fusion module performs feature fusion on the RGB feature extraction channel and the hyperspectral feature processing channel. Represents the hyperspectral features, using Represents RGB features, weighted fusion features as follows:

[0014] Where α is the weight coefficient, which is used to balance the contribution of RGB features and hyperspectral features; The decision module uses a lightweight Transformer structure to process fusion features , predicting the luggage destination based on fused features.

[0015] Another solution of the present invention is: A baggage sorting and loading method using the baggage sorting and loading subsystem comprises the following steps: Step S1: The luggage is placed in the check-in and baggage check-in areas and transported via a conveyor belt. When the luggage enters the sorting area, the hyperspectral imaging module and RGB camera collect hyperspectral data and RGB image information of the luggage respectively. Step S2: Hyperspectral data is pre-processed and fused with RGB image features. Step S3: The features fused in step S2 are used to predict the destination of the baggage through a classification network; Step S4: Cross-validate the prediction result of step S3 with the check-in system information to generate the final sorting instruction; Step S5: The robotic arm sorts the luggage to the corresponding conveyor belt according to the sorting instruction.

[0016] After adopting the above scheme, the airport baggage identification device and method based on hyperspectral imaging and multi-feature fusion of the present invention first obtains the spectral information of the baggage material through the hyperspectral imaging system, then uses the multi-feature fusion convolutional neural network for identification and classification, and finally improves the accuracy of identification through a multimodal verification mechanism to achieve efficient and accurate baggage sorting.

[0017] Compared to existing technologies, this invention effectively addresses the recognition challenges faced by traditional recognition systems in complex environments by introducing hyperspectral imaging technology and a multi-feature fusion convolutional neural network. Hyperspectral imaging technology captures the spectral characteristics of baggage materials, providing intrinsic characteristic information that traditional RGB or barcode systems cannot capture. The multi-feature fusion convolutional neural network uses end-to-end real-time processing to achieve adaptive weighted fusion of spectral and spatial features, ensuring recognition accuracy under various complex conditions (such as label damage, partial occlusion, or poor lighting conditions). The multimodal verification mechanism further reduces the false recognition rate by cross-validating hyperspectral recognition results with check-in system metadata. The collaborative work of dynamic hyperspectral and spatial feature fusion, end-to-end real-time processing, and the multimodal verification mechanism significantly improves the recognition accuracy and processing efficiency of airport baggage handling systems, effectively addressing the challenges posed by label damage, material similarity, and dynamic environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the overall architecture of the airport baggage handling system of the present invention.

[0019] Figure 2 Schematic diagram of the architecture of the present invention. DETAILED DESCRIPTION

[0020] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0021] like Figure 1 and Figure 2 As shown, the present invention discloses a baggage sorting and loading subsystem based on hyperspectral imaging and multi-feature fusion. The subsystem is deployed in the baggage sorting area between the check-in and baggage check-in areas and the conveying and loading system, and is connected to the check-in system, the conveying and loading system, and the central management system. The subsystem includes a hyperspectral imaging module, a spectral spatial preprocessing pipeline, a feature extraction and fusion module, a decision module, and an execution module. The hyperspectral imaging module is used to collect hyperspectral data of luggage and obtain the spectral characteristics of the luggage material; The spectral spatial preprocessing pipeline is used to smooth and reduce the dimensionality of the raw spectral data collected by the hyperspectral imaging module; The feature extraction and fusion module is based on a deep learning algorithm and includes an RGB feature extraction channel, a hyperspectral feature processing channel, and a feature fusion module; The RGB feature extraction channel is used to extract RGB image features captured by the RGB camera deployed in the baggage sorting area; The hyperspectral feature processing channel is used to process the hyperspectral features after being processed by the spectral spatial preprocessing pipeline; The feature fusion module is used to perform feature fusion on the RGB feature extraction channel and the hyperspectral feature processing channel; The decision module predicts the destination through a split network based on the results of the feature extraction and fusion modules; The execution module is used to control the robotic arm to complete the sorting operation according to the results of the decision module.

[0022] Among them, the check-in and baggage check-in area refers to the area where airport passengers check in and check in their luggage; the conveying and loading system is a physical system responsible for luggage transportation and loading, including conveyor belts and robotic arms. The conveyor belts are used to transport luggage, and the robotic arms are used to grab and place luggage according to the sorting results; the central management system is the control center that monitors and coordinates the entire baggage handling process.

[0023] This invention, based on hyperspectral imaging and multi-feature fusion, integrates hyperspectral imaging technology into the airport baggage sorting process for the first time. This replaces traditional barcode / RFID scanning and implements material-property-based baggage identification. This system captures material characteristics that are beyond the reach of traditional RGB cameras, addressing the identification challenges faced by traditional baggage sorting systems in complex environments. By capturing the spectral characteristics of baggage materials and combining them with a deep learning algorithm using a feature extraction and fusion module, this system achieves high-precision baggage identification and classification, maintaining excellent performance even when labels are damaged, partially obscured, or in poor lighting conditions.

[0024] The hyperspectral imaging module is a hyperspectral sensor array that collects hyperspectral data of luggage. In this embodiment, the hyperspectral imaging module uses a push-broom hyperspectral camera with a working band covering 400-2500nm and containing 240 spectral channels. For each luggage item, a three-dimensional hyperspectral data cube is collected. ,in and Represents the width and height of the image respectively, is the number of spectral bands. Each pixel Represents the reflectivity of a single pixel at the coordinate position (x, y) in space in different bands. To eliminate the influence of noise, this paper uses a spectral spatial preprocessing pipeline to perform Savitzky-Golay filtering on the spectral curve collected by the hyperspectral camera. By using the Savitzky-Golay filter to eliminate spectral noise, the noise elimination formula is:

[0025] Among them, is the filter window size, is the filter coefficient, The spectral curve is effectively smoothed by filtering while retaining important spectral features.

[0026] The RGB feature extraction channel uses an improved ResNet-50 network, which is a convolutional neural network that extracts spatial features from RGB images. Assuming the input RGB image is I, the feature extracted by the improved ResNet-50 network is represented as , which can be expressed as =ResNet-50 (I) Where I is the input RGB image, usually a three-dimensional tensor with a shape of (H, W, 3), where H and W are the height and width of the image, respectively, 3 represents the three RGB channels, and ResNet-50 represents the improved ResNet-50 network.

[0027] The feature fusion module dynamically fuses spectral and other features using a weighted summation method. The cross-attention mechanism dynamically adjusts the weights of hyperspectral and RGB features by calculating the relationship between them. Based on the characteristics of the input data, it adaptively assigns more weight to more discriminative features. Dynamically adjusting the importance of different sensor inputs through learnable weights significantly improves recognition robustness in complex scenarios.

[0028] The feature fusion module performs feature fusion on the RGB feature extraction channel and the hyperspectral feature processing channel. Represents the hyperspectral features, using Represents RGB features, weighted fusion features as follows:

[0029] Where α is the weight coefficient, which is used to balance the contribution of RGB features and hyperspectral features.

[0030] The decision module uses a lightweight Transformer structure to process fusion features , predicting the luggage destination based on fused features.

[0031] The present invention also discloses a baggage sorting and loading method based on hyperspectral imaging and multi-feature fusion, which includes the following steps: Step S1: The luggage is placed in the check-in and baggage check-in areas and transported via a conveyor belt. When the luggage enters the sorting area, the hyperspectral imaging module and RGB camera collect hyperspectral data and RGB image information of the luggage respectively. Step S2: Hyperspectral data is pre-processed and fused with RGB image features. Step S3: The features fused in step S2 are used to predict the destination of the baggage through a classification network; Step S4: Cross-validate the prediction result of step S3 with the check-in system information to generate the final sorting instruction; Step S5: The robotic arm sorts the luggage to the corresponding conveyor belt according to the sorting instruction.

[0032] The airport baggage sorting and loading subsystem and baggage sorting and loading method based on hyperspectral imaging and multi-feature fusion in the present invention adopts dynamic fusion of hyperspectral-spatial features to solve the limitation that traditional RGB or barcode systems cannot obtain the intrinsic properties of materials; by deploying a lightweight Transformer architecture on edge devices, starting from the acquisition of hyperspectral data and RGB images of baggage, through preprocessing, feature extraction, fusion, decision-making and execution modules are tightly integrated in a continuous system framework to achieve low-latency online recognition and decision-making; the hyperspectral recognition results are cross-validated with the check-in system metadata to form a multimodal verification mechanism, which significantly reduces the misrecognition rate.

[0033] The technical value of this invention is mainly reflected in three aspects: first, by introducing hyperspectral imaging technology, the system can obtain material feature information that traditional RGB cameras cannot capture; second, the innovative multi-feature fusion mechanism realizes the adaptive weighted fusion of spectral features and spatial features; finally, the end-to-end system design ensures the optimization of the entire process from data acquisition to sorting decision-making.

[0034] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A baggage sorting and loading subsystem, characterized in that: The system is deployed in the baggage sorting area between the check-in and baggage drop-off areas and the conveying and loading systems, and is connected to the check-in system, the conveying and loading system, and the central management system. It includes a hyperspectral imaging module, a spectral spatial preprocessing pipeline, a feature extraction and fusion module, a decision module, and an execution module. The hyperspectral imaging module is used to collect hyperspectral data of luggage and obtain the spectral characteristics of the luggage material; The spectral spatial preprocessing pipeline is used to smooth and reduce the dimensionality of the raw spectral data collected by the hyperspectral imaging module; The feature extraction and fusion module is based on a deep learning algorithm and includes an RGB feature extraction channel, a hyperspectral feature processing channel, and a feature fusion module; The RGB feature extraction channel is used to extract RGB image features captured by the RGB camera deployed in the baggage sorting area; The hyperspectral feature processing channel is used to process the hyperspectral features processed by the spectral spatial preprocessing pipeline; The fusion module is used to perform feature fusion on the RGB feature extraction channel and the hyperspectral feature processing channel; The decision module predicts the destination through a split network based on the results of the feature extraction and fusion modules; The execution module is used to control the robotic arm to complete the sorting operation according to the results of the decision module.

2. The baggage sorting and loading subsystem according to claim 1, characterized in that: The hyperspectral imaging module is a hyperspectral sensor array that collects hyperspectral data of luggage.

3. The baggage sorting and loading subsystem according to claim 2, characterized in that: The hyperspectral sensor array uses a push-broom hyperspectral camera with a working band covering 400-2500nm and containing 240 spectral channels; the hyperspectral camera collects a three-dimensional hyperspectral data cube of each luggage item. in, Represents a three-dimensional hyperspectral data cube; W and H represent the width and height of the image respectively. B=240, indicating the number of spectral wave channels; R is a real number, indicating that the spectral intensity or reflectance value of each pixel in the data cube is a real number; Each pixel Represents the reflectivity of a single pixel at the coordinate position (x, y) in space in different bands; The spatial coordinate indicating the current spectral channel number; T stands for transpose, which converts a row vector into a column vector.

4. The baggage sorting and loading subsystem according to claim 2, characterized in that: The spectral spatial preprocessing pipeline performs Savitzky-Golay filtering on the spectral curve acquired by the hyperspectral camera, and eliminates spectral noise by using the Savitzky-Golay filter. The noise elimination formula is: in, represents the spectrum value after noise removal after Savitzky-Golay filter processing; represents the original spectrum value; N represents the number of original spectral data points; is the filter window size; is the filter coefficient, is the normalization factor; To sum the index variables, the value range is arrive , used to iterate over the different terms in the summation process.

5. The baggage sorting and loading subsystem according to claim 1, characterized in that: The RGB feature extraction channel uses an improved ResNet-50 network, a convolutional neural network that extracts spatial features from RGB images; The input RGB image is I, and the features extracted by the improved ResNet-50 network are , =ResNet-50(I), where I is the input RGB image, which is a three-dimensional tensor with a shape of (H, W, 3), where H and W are the height and width of the image, respectively, 3 represents the three RGB channels, and ResNet-50 represents the improved ResNet-50 network.

6. The baggage sorting and loading subsystem according to claim 5, characterized in that: The feature fusion module uses a weighted summation method to dynamically fuse spectral and other features. The cross-attention mechanism dynamically adjusts the weights of hyperspectral and RGB features by calculating the relationship between them. Based on the characteristics of the input data, it adaptively assigns more weight to more discriminative features and dynamically adjusts the importance of different sensor inputs through learnable weights. The feature fusion module performs feature fusion on the RGB feature extraction channel and the hyperspectral feature processing channel. Represents the hyperspectral features, using Represents RGB features, weighted fusion features as follows: Where α is the weight coefficient, which is used to balance the contribution of RGB features and hyperspectral features; The decision module uses a lightweight Transformer structure to process fusion features , predicting the luggage destination based on fused features.

7. A baggage sorting and loading method using the baggage sorting and loading subsystem according to any one of claims 1 to 6, characterized in that: The following steps are involved: Step S1: The luggage is placed in the check-in and baggage check-in areas and transported via a conveyor belt. When the luggage enters the sorting area, the hyperspectral imaging module and RGB camera collect hyperspectral data and RGB image information of the luggage respectively. Step S2: Hyperspectral data is pre-processed and fused with RGB image features. Step S3: The features fused in step S2 are used to predict the destination of the baggage through a classification network; Step S4: Cross-validate the prediction result of step S3 with the check-in system information to generate the final sorting instruction; Step S5: The robotic arm sorts the luggage to the corresponding conveyor belt according to the sorting instruction.

Images