Cross-modal luggage re-identification method based on RGB-D and attention mechanism

Abstract

The invention provides a cross-modal luggage re-identification method (CMAM-ReID) based on RGB-D and an attention mechanism in order to overcome the defects of an existing luggage re-identification algorithm in a luggage re-identification scene. In order to solve the influence of inherent noise points of the depth image on feature extraction, an optimized median filtering algorithm is introduced, and noise is effectively removed. For the fusion problem of RGB features and depth features, a feature fusion module based on an attention mechanism is designed, so that the algorithm can adaptively endow different features with weights. In order to solve the problem of combination of global features and local features, the invention also provides a ResNet-50-based multi-branch feature extraction network. Furthermore, a dedicated data set is created for the need of airport luggage re-identification. According to the method, the luggage re-identification effect is remarkably improved, and the practical value is high.

Description

Technical Field

[0001] The present invention relates to the field of baggage re-identification, and in particular to a cross-modal baggage re-identification method based on RGB-D and attention mechanism Background Art

[0002] Most of the existing airport baggage sorting systems are based on RFID technology, but RFID has the problem of high cost and difficulty in monitoring the position of baggage on the conveyor belt throughout the entire process. Therefore, realizing the automatic sorting and tracking of baggage in the entire process of civil aviation airports is of great significance to the sustainable development of civil aviation airports. However, it is difficult for traditional solutions to achieve automatic sorting and tracking of the entire process. In order to solve this problem, using re-identification algorithms to identify baggage under different cameras on the conveyor belt has become a feasible implementation solution. However, the existing re-identification algorithms are mostly based on pedestrian re-identification algorithms, which do not take into account the characteristics and problems of baggage re-identification.

[0003] Current person re-identification algorithms are classified based on modality and can be divided into two categories: one is a single-modality person re-identification algorithm and the other is a multi-modality person re-identification algorithm. In the single-modality person re-identification algorithm, Zheng L et al. designed a pose-invariant embedding (PIE) as a pedestrian descriptor, aligned pedestrians to a standard pose, and introduced a PoseBox structure generated by pose estimation and affine transformation. Yuteng Ye et al. proposed a feature pruning and merging (FPC) framework to address the problem of difficulty in re-identifying occluded people due to contamination by occluders. However, most single-modality person re-identification algorithms do not pay attention to the importance of multi-scale features, that is, both local features and global features have a promoting effect on re-identification. In the multi-modal person re-identification algorithm, Zhang Z et al. proposed an RGB-infrared (RGB-IR) person re-identification algorithm, which is based on a generative adversarial network. However, the image quality generated by the generative adversarial network is low, resulting in low model performance. Lu H et al. proposed a Transformer-based RGB-IR pedestrian re-identification algorithm named PMT. The algorithm is based on a progressive strategy for learning, but does not make good use of the features of infrared images. Ye et al. proposed a pedestrian re-identification method based on symmetric ranking metric learning. The algorithm selects two sets of mutually symmetrical samples from two modalities, and then uses ranking loss to optimize the sample pairs, and finally narrows the difference between the two modalities. However, the above multi-modal re-identification research algorithms are all based on pedestrian re-identification. Pedestrian re-identification is mostly carried out in outdoor environments, so it is necessary to consider the difference between day and night. Therefore, infrared images can be introduced to reduce the impact of night on RGB images. The difference between infrared images of luggage is not large, and airport luggage re-identification is in an indoor environment. Its lighting conditions are relatively uniform, but there are fixed differences under different cameras. At the same time, in addition to color, luggage re-identification also needs to pay attention to its surface texture and size, so it is more necessary to consider the introduction of other modalities. Summary of the invention

[0004] The purpose of the present invention is to propose a cross-modal baggage re-identification method based on RGB-D and attention mechanism to solve the problem that the existing re-identification algorithm has low accuracy and cannot meet actual needs.

[0005] The object of the present invention is achieved in this way:

[0006] The depth camera is used to extract the depth image and RGB image of the luggage. First, the optimized median filter algorithm is used to filter and reduce noise on the RGB image. Then, the feature extraction network is used to extract RGB and depth features, and the convolutional attention mechanism is used to fuse the RGB features and depth features. Finally, the network is trained with multiple loss functions to obtain the re-identification results.

[0007] It specifically includes the following steps:

[0008] Step 1: Use the optimized median filtering algorithm to perform noise reduction on the original depth image.

[0009] Step 2: Extract multi-branch depth features and RGB features.

[0010] Step 3: Input the features into the feature fusion module based on the attention mechanism for feature fusion

[0011] Step 4: Obtain network output through training and finally obtain the baggage re-identification result.

[0012] Furthermore, in step 1, the user obtains the RGB image and depth image of the luggage through the depth camera and uses the optimized median filter algorithm to perform noise reduction on the depth image. Assuming the original depth image I, the filter window W size M×N, and the set threshold T. Calculate the difference between the current pixel I (i, j) and each pixel in the window W, and calculate the average value avg of these differences. When avg is greater than the set threshold T, the median filtering algorithm is used to replace the pixel value of the point with the median value of the pixel points in the search box. Otherwise, there is no need to replace the pixel value of the point with the median value of the pixel points in the search box.

[0013] Furthermore, in step 2, after obtaining the result of step 1, the RGB image and the denoised depth image, the depth features and RGB features are extracted in the following manner: the RGB image and the depth image are respectively sent to two ResNet-50 networks that do not share parameters. The first branch of the network obtains the RGB feature map and the depth feature map representing the local features after passing through layer2 (including several 1×1 convolution kernels and 3×3 convolution kernels). The second branch will obtain the RGB feature map and the depth feature map representing the global features after passing through layer4 (including several 1×1 convolution kernels and 3×3 convolution kernels).

[0014] Furthermore, in step 3, after obtaining the result of step 2, i.e., the RGB feature map and the depth feature, the RGB feature and the depth feature are fused, and the features of the RGB image containing color and surface features and the depth image containing depth information are fused. The fusion method is as follows: Assuming that the input RGB feature F RGB and deep features F Depth , whose size is H×W×C. RGB Calculate its channel weight W C =σ(MLP(AvgPool(F RGB ))+MLP(MaxPool(F RGB )))Where σ represents the sigmoid function, MLP represents a shared multi-layer network. The obtained channel weight vector Wc With the deep feature F Depth Perform a dot multiplication operation to obtain a new intermediate feature and then add it to the RGB feature F RGB Concatenate them together to get the fusion feature F based on channel attention RGB-D According to F RGB-D Calculate the spatial weight W s =σ(f 7×7 ([AvgPool(F RGB-D );MaxPool(F RGB-D )]))Where σ represents the sigmoid function, f 7×7 represents a convolutional layer with a kernel size of 7×7. The obtained spatial weight W s With the initial fusion feature F RGB-D The dot multiplication is performed to obtain a feature map of H×W×2C. At the same time, in order to maintain the consistency of the dimension and remove redundant information to achieve the invariance of the feature, the obtained feature map is compressed into a feature map of size H×W×C by performing maximum pooling, that is, the final deep feature F Depth and RGB feature F RGB The fusion feature F′ RGB-D .

[0015] Furthermore, in step 4, after obtaining the result F′ of step 3, RGB-D , combined with the difficult sample triple and cross entropy loss function for training, the final re-identification result is obtained.

[0016] The positive effects of the present invention are:

[0017] (1) Aiming at the baggage re-identification task in real scenarios, the present invention proposes a cross-modal baggage re-identification method CMAM-ReID based on RGB-D and attention mechanism. The core idea of ​​the present invention is to introduce depth images to solve the problem that the color features and texture features of the RGB images of the same baggage under different cameras are obviously different under different lighting conditions. First, the present invention proposes an optimized median filtering algorithm to remove the noise of the depth image and reduce its influence on re-identification. Secondly, the present invention proposes an optimized median filtering algorithm to remove the noise of the depth image and reduce its influence on re-identification. Then, the present invention proposes a multi-branch feature extraction network based on ResNet-50 to extract the global and local features of the image at the same time. Finally, the present invention proposes a feature fusion module based on the attention mechanism to allow the algorithm to decide which feature to give a higher weight and the part that pays more attention to. This method can improve the accuracy of baggage re-identification.

[0018] (2) The present invention proposes to create a baggage re-identification dataset for airport baggage re-identification scenario detection needs. The dataset is a multimodal baggage dataset including RGB and depth images. The dataset was collected on-site on the baggage conveyor belt of the test base, including hard bags, soft bags and baggage of various sizes, and has practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the flow chart of optimized median filtering algorithm.

[0020] Figure 2 Multi-branch feature extraction network structure diagram.

[0021] Figure 3 This is the structural diagram of the feature fusion module based on the attention mechanism. DETAILED DESCRIPTION

[0022] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0023] Step 1 Figure 1 As shown, we use the optimized median filtering algorithm to obtain the denoised depth image from the original depth image through the set filtering window W and the set threshold T.

[0024] Step 1 and step 2 are as follows Figure 1 Phase 1 and Figure 2 As shown, we extract the meta-feature matrix of the dataset from the data sample feature extractor and the dataset meta-feature extractor, and generate a similarity matrix to obtain the initial population input for the next stage.

[0025] Step 2 Figure 2 As shown in the figure, the denoised depth image and RGB image are respectively extracted with a multi-branch feature extraction network to extract their global features and local features as the input of the feature fusion module.

[0026] Step 3, such as Figure 3 As shown in the figure, the RGB features and depth features are calculated by calculating channel attention and spatial attention to obtain the final fusion features.

[0027] Step 4: Train based on the fusion features to obtain the final re-identification result.

Claims

1. A cross-modal baggage re-identification method based on RGB-D and attention mechanism, comprising the following steps: Step 1: Use the optimized median filter algorithm to reduce the noise of the original depth image. Step 2: Extract multi-branch depth features and RGB features. Step 3: Input the features into the feature fusion module based on the attention mechanism for feature fusion. Step 4: Obtain network output through training and finally obtain the baggage re-identification result.

2. In step 1, the user obtains the RGB image and depth image of the luggage through the depth camera and uses the optimized median filtering algorithm to perform noise reduction on the depth image; assuming that the original depth image I, the filter window W size M×N, and the set threshold T; calculate the difference between the current pixel I(i, j) and each pixel in the window W, and calculate the average value avg of these differences; When avg is greater than the set threshold T, the median filtering algorithm needs to be used to replace the pixel value of the point with the median of the pixel points in the search box. Otherwise, the median of the pixel points in the search box does not need to be used to replace the pixel value of the point.

3. In step 2, after obtaining the result of step 1, the RGB image and the denoised depth image, the depth features and RGB features are extracted in the following manner: the RGB image and the depth image are respectively sent to two ResNet-50 networks that do not share parameters; the first branch of the network obtains an RGB feature map and a depth feature map representing local features after passing through layer2 (including several 1×1 convolution kernels and 3×3 convolution kernels); the second branch will obtain an RGB feature map and a depth feature map representing global features after passing through layer4 (including several 1×1 convolution kernels and 3×3 convolution kernels).

4. In step 3, after obtaining the result of step 2, i.e., the RGB feature map and the depth feature, the RGB feature and the depth feature are fused, and the features of the RGB image containing color and surface features and the depth image containing depth information are fused; the fusion method is as follows: Assuming that the input RGB feature F RGB and deep features F Depth , whose size is H×W×C; F RGB Calculate its channel weight: W C =σ(MLP(AvgPool(F RGB ))+MLP(MaxPool(F RGB ))) in, σ represents the sigmoid function, MLP represents a shared multi-layer network; the obtained channel weight vector W c With the deep feature F Depth Perform a dot multiplication operation to obtain a new intermediate feature and then add it to the RGB feature F RGB Concatenate them together to get the fusion feature F based on channel attention RGB-D According to F RGB-D Calculate the spatial weight: W s =σ(f 7×7 ([AvgPool(F RGB-D );MaxPool(F RGB-D )])) Among them, σ represents the sigmoid function, f 7×7 represents a convolutional layer with a kernel size of 7×7; the resulting spatial weight W s With the initial fusion feature F RGB-D The dot multiplication is performed to obtain a feature map of H×W×2C. At the same time, in order to maintain the consistency of the dimension and remove redundant information to achieve the invariance of the feature, the obtained feature map is compressed into a feature map of size H×W×C by performing maximum pooling, that is, the final deep feature F Depth and RGB feature F RGB The fusion feature F′ RGB-D .

5. In step 4, the result F′ of step 3 is obtained. RGB-D , combined with the difficult sample triple and cross entropy loss function for training, the final re-identification result is obtained.

Images