ViT luggage re-identification method based on local data enhancement

Abstract

The invention provides a local data enhancement-based ViT luggage re-identification method, and aims to improve the identification accuracy in automatic luggage sorting. A traditional RFID technology is high in cost and prone to interference, and therefore introduction of a computer vision technology becomes a new trend. The method comprises the following steps that firstly, a data set containing 1500 pieces of luggage with different identities is constructed, each piece of luggage has 10 photos with different angles, and LabelMe and YOLOv5 are used for labeling and detection; secondly, a plurality of cameras are arranged on the luggage check-in and sorting assembly line to collect luggage images; then, based on a ViT model, applying a local data enhancement technology and a position excitation module to improve the attention and generalization ability of the model to luggage structure features; thirdly, luggage features are calculated through a multi-layer visual conversion module, and identity comparison is carried out by adopting a cosine distance; and finally, optimizing the model by using identity loss, triple loss and contrast loss to improve the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to computer vision, and in particular to a ViT luggage re-identification method based on local data enhancement. Background Art

[0002] With the accelerated development of the domestic economy and the continuous improvement of people's living standards, more and more people choose to travel by plane. According to the data in the "Statistics of Major Production Indicators of China's Civil Aviation in November 2023" released by the Ministry of Transport of the People's Republic of China, the total passenger transportation of domestic flights in my country reached 569 million in 2023. During the flight, it is usually the case that a passenger will carry multiple pieces of luggage, which results in the number of luggage that domestic airports need to handle being much larger than the number of passengers. Faced with such a large number of luggage, domestic airports can no longer meet people's travel needs by relying solely on manual sorting of luggage. In this case, for airports, the importance of automated baggage sorting systems in the process of transporting passengers has become increasingly prominent. At present, the mainstream airport baggage automated sorting system uses radio frequency identification technology (RFID) to sort luggage.

[0003] Radio frequency identification technology uses unique electromagnetic tags and electromagnetic readers to identify and transmit data for baggage. As can be seen from the figure, the baggage sorting process is divided into two main stages: check-in and conveyor belt tracking. In the check-in stage, airport staff will attach electromagnetic tags with unique identification codes to the luggage for subsequent tracking and identification. In the subsequent conveyor belt tracking stage, special electromagnetic reading and writing equipment is responsible for the identification and tracking of luggage. Specifically, after the luggage is labeled at the check-in counter and enters the conveyor belt, it will move quickly along the conveyor belt and pass through multiple electromagnetic reading and writing device nodes. In this process, each node can identify the electromagnetic tags attached to the past luggage and integrate the node information that the electromagnetic tags pass through to form the movement trajectory of the luggage, such as passing through node #1, node #3, node #4, etc.

[0004] Although RFID technology has been widely used in baggage tracking and identification, it still has some inherent limitations. For example, the equipment cost is high, the baggage tracking effect is limited by the number of readers, and electromagnetic interference may cause identification failure. These problems limit the further development and promotion of automated baggage sorting systems. At the same time, in recent years, computer vision technology has demonstrated excellent robustness, accuracy and real-time performance in tracking and monitoring pedestrians and vehicles, showing the potential to handle complex scenes. Therefore, in order to meet people's strong demand for efficient travel, it is necessary to upgrade the baggage tracking system based on RFID to a baggage tracking system using computer vision technology.

[0005] The computer vision baggage tracking system realizes the entire cross-camera baggage tracking process by identifying, tracking, and re-identifying the baggage in the captured image. Specifically, this process first involves identifying which objects are baggage from the camera image. Next, the system monitors the movement trajectory of these baggage in real time to complete the single-camera tracking task. The critical last step is that the system needs to confirm the exact identity of these baggage and match the baggage trajectories captured by different cameras to construct the complete path of the baggage. In the process of cross-camera tracking, baggage re-identification technology plays a role similar to that of electromagnetic tags. It identifies the identity of the baggage by analyzing the characteristics of the baggage in the camera image. This is the core link to complete the cross-camera baggage trajectory matching and build a complete baggage path.

[0006] Thanks to the rapid development of neural network technology in recent years, luggage recognition and tracking within a single camera have shown excellent performance. However, in terms of re-identification technology for luggage identity matching across cameras, current solutions still do not fully meet the needs of practical applications. Therefore, this paper focuses on the method of luggage re-identification.

[0007] In the specific scenario of baggage sorting lines, object re-identification technology faces new challenges. These challenges mainly come from the fact that the accuracy of re-identification is highly dependent on the quality of the images captured by the camera and whether the appearance features of the luggage remain consistent in different scenarios. The operating environment of the baggage re-identification task is more complex than the common pedestrian or vehicle re-identification scenarios, and complex environmental conditions often cause the appearance features of the same piece of luggage to change.

[0008] In a baggage sorting flow environment, cameras are installed at multiple key locations on the conveyor belt, where other auxiliary sorting equipment is often located nearby.

[0009] The following is a brief description of how auxiliary sorting equipment causes changes in the appearance and posture of baggage:

[0010] (1) Signal indicator light

[0011] There are multiple timed flashing signal indicators on the baggage sorting line. The rapid alternation of light and dark causes the appearance color of some metallic painted luggage to change rapidly.

[0012] (2) Conveyor belt

[0013] The conveyor belt of the baggage sorting line has ups and downs and many bends. The baggage often reverses during the forward movement, and the posture of the baggage before and after the reversal is often quite different.

[0014] (3) Mechanical swing arm

[0015] The baggage sorting line uses mechanical swing arms for diversion, and the mechanical swing arms will also change the posture of the luggage when changing the luggage conveying channel.

[0016] (4) Fence

[0017] The fences on the baggage sorting line will block the baggage from the camera's field of view.

[0018] In summary, since the same piece of luggage may present significantly different surface colors and moving postures under different camera fields of view, and the camera images may sometimes not fully cover the target luggage, the luggage re-identification task faces the challenge of accurately identifying a large number of target identities in such a complex and changing environment.

[0019] In addition, in airport scenarios with a daily passenger flow of about 100,000 people, the baggage videos taken from different angles on the sorting line every day provide rich data resources for baggage re-identification model training. However, the huge size and complexity of these data make it unfeasible to rely on traditional manual annotation methods. Therefore, fully unsupervised training algorithms have become a better choice for training baggage re-identification models using large-scale datasets. However, most existing fully unsupervised re-identification training algorithms are mainly trained and tested on small-scale datasets. In environments with small data volumes, these algorithms require a large amount of video memory resources for data comparison and clustering. This approach makes model training more difficult when processing large-scale datasets. Therefore, another challenge facing unsupervised training for baggage re-identification is how to handle the training of large-scale datasets with limited video memory resources.

[0020] In addition, this paper is based on the school-level project "Development of Key Technologies for Multi-source Data Fusion and Collaboration in Aviation Logistics" led by the Second Research Institute of the Civil Aviation Administration of China. In summary, in-depth research and development of baggage re-identification models is not only an important scientific research topic, but also has great economic significance. Summary of the invention

[0021] The purpose of the present invention is to propose a ViT baggage re-identification method based on local data enhancement. The model aims to improve the accuracy of baggage re-identification to meet the actual needs of baggage sorting engineering.

[0022] The object of the present invention is achieved in this way:

[0023] Improve the ViT baseline re-ID model with multi-scale feature fusion. Through in-depth research on the baggage sorting process, this chapter noticed that most of the baggage is made of rigid materials, which means that their structure will not change due to rotation or lighting changes. Therefore, the LocEnViT (Local Enhance Vision Transformer) baggage re-ID model is based on the ViT baseline re-ID model with multi-scale feature fusion. It applies the image local area data enhancement technology for ViT, and combines the location excitation module and contrastive learning method in the training process to compare the "new" features generated by data enhancement with the local features of the original image, thereby improving the generalization ability of the re-ID model.

[0024] The specific method is as follows:

[0025] Step 1. Construct a baggage dataset and create a baggage dataset LReID-1500 with identity tags. These videos are recorded by five cameras, covering various locations of the sorting line, with a total duration of 148 hours, and contain more than 100,000 pieces of baggage with different identities. The dataset was produced by combining LabelMe and YOLOv5 tools. LabelMe is used for manual labeling, and YOLOv5 is used for target detection. The detection results are cropped to construct the dataset. LReID-1500 contains 1,500 pieces of baggage with different identities, a total of 15,000 pictures, and an average of 10 pictures of each piece of baggage from different angles. The production steps include: using LabelMe and videos to create a small batch detection dataset, initializing and fine-tuning the YOLOv5 network, generating videos annotated with baggage bounding boxes, extracting a picture every 4 frames, cropping the bounding boxes to generate a spare dataset, and finally manually screening to exclude duplicate pictures, and completing the manual labeling of the unlabeled dataset;

[0026] Step 2: Build a local baggage sorting line video acquisition environment, including multiple non-overlapping cameras and a video stream transmission server for the baggage sorting line;

[0027] Step 3: Build a local baggage check-in video acquisition environment, including a baggage check-in camera and a video stream transmission server;

[0028] Step 4: Obtain baggage check-in images and videos of baggage taken under different viewing areas on the water flow sorting line;

[0029] Step 5: Deploy the local baggage re-identification feature calculation LocEnViT model. The LocEnViT baggage re-identification model improves on the existing multi-scale feature fusion ViT baseline re-identification model, with special emphasis on the structural features of the same identity baggage, which remain relatively stable even when the surface color and movement posture of the baggage change.

[0030] Step 6: Calculate the re-identification features between different luggage using the local luggage re-identification feature calculation model, using the multi-layer visual conversion module and the local feature data enhancement module;

[0031] Step 7: Substitute the re-identification features between different luggage into the distance formula; determine whether the two target luggages have the same identity;

[0032] 1. The ViT baggage re-identification method based on local data enhancement as described in claim 1 is characterized in that in step 5, the LocEnViT model uses a sliding window to divide the baggage image in the video into small tiles with overlapping pixels. The sliding step is represented as S, and the size of the small tile is represented as P. The shape of the overlapping area of ​​two adjacent small tiles can be represented as (PS)×P. Given a target image G∈

[0033] R H×W×C (H, W, C represent the height, width, and number of channels of the target image, respectively), the target image can be split into N small tiles of fixed size, and each small tile after splitting can be represented as x i ∈R H×P×P ;

[0034] 2. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 uses linear projection and additional information to add the small block x i ∈R H×P×P Convert to input token sequence Z o :

[0035]

[0036] Among them, v PE ∈R 1×D Represents a set of learnable position information vectors that are common to all labeled sequences, thereby characterizing the spatial position information of small patches in the input image. Represented as learnable viewpoint information vectors corresponding to different cameras;

[0037] 3. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 performs local data enhancement on the small image block x i ∈R H×P×P Randomly select and copy H rectangles consisting of adjacent small tiles, each rectangular area consists of l small tiles, and use them as the anchor sample input sequence for data enhancement

[0038] 4. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that in the LocEnViT model in step 5, the anchor sample input sequence of data enhancement in the process of local data enhancement is Perform three data enhancements to obtain the corresponding positive sample input sequence And Perform linear projection and add additional information to obtain the corresponding anchor sample label sequence

[0039]

[0040] 5. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 uses a multi-layer visual transformation module The input token sequence Z o , Perform feature extraction;

[0041] 6. The ViT baggage re-identification method based on local data enhancement according to claim 1, characterized in that the LocEnViT model in step 5 uses identity loss L ID , triple loss L triple and local data enhancement contrast loss L data Conduct model training.

[0042] L total =θ 1 L ID +θ 2 L triple +θ 3 L data ,

[0043] subject toθ 1 +θ 2 +θ 3 =1&θ 1 ,θ 2 ,θ3≥0

[0044] Identity loss L ID It can be expressed as:

[0045]

[0046] Triplet loss L triple It can be expressed as:

[0047]

[0048] The contrast loss can be expressed as:

[0049]

[0050] 7. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 measures the similarity of two feature vectors by a distance calculation formula during the inference process. Wherein the cosine distance The calculation formula is as follows:

[0051]

[0052] When determining whether two baggage images belong to the same identity, the LocEnViT baggage re-identification model performs cosine distance calculation:

[0053]

[0054] The positive effects of the present invention are:

[0055] This paper proposes a ViT baggage re-identification model based on local data enhancement and names it LocEnViT, which aims to solve the problem that the existing ViT baseline model has insufficient generalization ability when dealing with the same baggage with changed appearance features and similar different baggage. The core innovation of the LocEnViT baggage re-identification model is that it obtains multiple sets of "new" data by performing data enhancement on local areas of the image, and feeds these data into the feature extraction network in parallel during the training process. The combination of the position excitation module and the contrast loss enables the model to deeply explore the detailed features and structural features of the baggage, thereby improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of luggage data with identity tags.

[0057] Figure 2 LocEnViT re-identification model architecture.

[0058] Figure 3 Schematic diagram of the process of adding additional information.

[0059] Figure 4 Schematic diagram of the self-attention layer structure.

[0060] Figure 5 Schematic diagram of the multi-scale feature extraction module.

[0061] Figure 6 Schematic diagram of the feature processing module.

[0062] Figure 7 Schematic diagram of local data augmentation.

[0063] Figure 8Schematic diagram of image position excitation. DETAILED DESCRIPTION

[0064] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0065] 1. A ViT baggage re-identification method based on local data enhancement, characterized in that the method comprises the following steps:

[0066] Step 1: Build a baggage dataset. Create a baggage dataset LReID-1500 with identity tags. These videos are recorded by five cameras, covering various locations of the sorting line, with a total duration of 148 hours, and contain more than 100,000 pieces of baggage with different identities. The dataset is made by combining LabelMe and YOLOv5 tools. LabelMe is used for manual labeling, and YOLOv5 is used for target detection. The detection results are cropped and the dataset is constructed. LReID-1500 contains 1,500 pieces of baggage with different identities, a total of 15,000 pictures, and an average of 10 pictures of each piece of baggage from different angles.

[0067] Step 2: Build a local baggage sorting line video acquisition environment, including multiple non-overlapping cameras and a video stream transmission server for the baggage sorting line;

[0068] Step 3: Build a local baggage check-in video acquisition environment, including a baggage check-in camera and a video stream transmission server;

[0069] Step 4: Obtain baggage check-in images and videos of baggage taken under different viewing areas on the water flow sorting line;

[0070] Step 5: Deploy the local baggage re-identification feature calculation LocEnViT model. The LocEnViT baggage re-identification model improves on the existing multi-scale feature fusion ViT baseline re-identification model, with special emphasis on the structural features of the same identity baggage, which remain relatively stable even when the surface color and movement posture of the baggage change.

[0071] Step 6: Calculate the re-identification features between different luggage using the local luggage re-identification feature calculation model, using the multi-layer visual conversion module and the local feature data enhancement module;

[0072] Step 7: Substitute the re-identification features between different luggage into the distance formula; determine whether the two target luggages have the same identity;

[0073] 2. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that in step 5, the LocEnViT model uses a sliding window to divide the baggage image in the video into small tiles with overlapping pixels. The sliding step is represented as S, and the size of the small tile is represented as P. The shape of the overlapping area of ​​two adjacent small tiles can be represented as (PS)×P. Given a target image G∈R H×W×C (H, W, C represent the height, width, and number of channels of the target image, respectively), the target image can be split into N small tiles of fixed size, and each small tile after splitting can be represented as x i ∈R H×P×P ;

[0074] 3. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 uses linear projection and additional information to add the small block x i ∈R H×P×P Convert to input token sequence Z o :

[0075]

[0076] Among them, v PE ∈R 1×D Represents a set of learnable position information vectors that are common to all labeled sequences, thereby characterizing the spatial position information of small patches in the input image. Represented as learnable viewpoint information vectors corresponding to different cameras;

[0077] 4. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 performs local data enhancement on the small image block x i ∈R H×P×P Randomly select and copy H rectangles consisting of adjacent small tiles, each rectangular area consists of l small tiles, and use them as the anchor sample input sequence for data enhancement

[0078] 5. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5, in the process of local data enhancement, the anchor sample input sequence of data enhancement Perform three data enhancements to obtain the corresponding positive sample input sequence And Perform linear projection and add additional information to obtain the corresponding anchor sample label sequence

[0079]

[0080] 6. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 uses a multi-layer visual conversion module The input token sequence Zo 、 Perform feature extraction;

[0081] 7. The ViT baggage re-identification method based on local data enhancement according to claim 1, characterized in that the LocEnViT model in step 5 uses identity loss L ID , triple loss L triple and local data enhancement contrast loss L data Conduct model training.

[0082] L total =θ 1 L ID +θ 2 L triple +θ 3 L data ,

[0083] subject toθ 1 +θ 2 +θ 3 =1&θ 1 ,θ 2 ,θ3 ≥ 0

[0084] Identity loss L ID It can be expressed as:

[0085]

[0086] Triplet loss L triple It can be expressed as:

[0087]

[0088] The contrast loss can be expressed as:

[0089]

[0090] 8. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that the LocEnViT model in step 5 measures the similarity of two feature vectors by a distance calculation formula during the inference process. Wherein the cosine distance The calculation formula is as follows:

[0091]

[0092] When determining whether two baggage images belong to the same identity, the LocEnViT baggage re-identification model performs cosine distance calculation:

[0093]

Claims

1. A ViT baggage re-identification method based on local data enhancement, characterized in that: The method comprises the following steps: Step 1: Construct a baggage dataset. Create a baggage dataset LReID-1500 with identity tags. These videos are recorded by five cameras, covering various locations of the sorting line, with a total duration of 148 hours, and contain more than 100,000 pieces of baggage with different identities. The dataset is made by combining LabelMe and YOLOv5 tools. LabelMe is used for manual labeling, and YOLOv5 is used for target detection. The detection results are cropped to construct the dataset. LReID-1500 contains 1,500 pieces of baggage with different identities, a total of 15,000 pictures, and an average of 10 pictures of each piece of baggage from different angles. The production steps include: using LabelMe and videos to create a small batch detection dataset, initializing and fine-tuning the YOLOv5 network, generating videos with baggage bounding boxes annotated, extracting a picture every 4 frames, cropping the bounding boxes to generate a spare dataset, and finally manually screening to exclude duplicate pictures, and completing the manual labeling of the unlabeled dataset. Step 2: Build a local baggage sorting line video acquisition environment, including multiple non-overlapping cameras and a video stream transmission server for the baggage sorting line; Step 3: Build a local baggage check-in video acquisition environment, including a baggage check-in camera and a video stream transmission server; Step 4: Obtain baggage check-in images and videos of baggage taken under different viewing areas on the water flow sorting line; Step 5: Deploy the local baggage re-identification feature calculation LocEnViT model; the LocEnViT baggage re-identification model is an improvement on the existing multi-scale feature fusion ViT baseline re-identification model, with special emphasis on the structural features of the same identity baggage, which remain relatively stable even when the surface color and movement posture of the baggage change; Step 6: Calculate the re-identification features between different luggage using the local luggage re-identification feature calculation model, using the multi-layer visual conversion module and the local feature data enhancement module; Step 7: Substitute the re-identification features between different luggage into the distance formula; determine whether the two target luggages have the same identity.

2. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that: In step 5, the LocEnViT model uses a sliding window to divide the baggage image in the video into small tiles with overlapping pixels; the sliding step is represented as S, the size of the small tile is represented as P, and the shape of the overlapping area of ​​two adjacent small tiles can be represented as (PS)×P; given the target image G∈R H×W×C (H, W, C represent the height, width, and number of channels of the target image, respectively), the target image can be split into N small tiles of fixed size, and each small tile after splitting can be represented as x i ∈R H×P×P .

3. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that: In the LocEnViT model in step 5, the small image patch x is added using linear projection and additional information i ∈R H×P×P Convert to input token sequence Z o : Among them, v PE ∈R 1×D Represents a set of learnable position information vectors that are common to all labeled sequences, thereby representing the spatial position information of small patches in the input image; Represented as learnable viewpoint information vectors corresponding to different cameras.

4. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that: The LocEnViT model in step 5 performs local data enhancement on the small image patch x i ∈R H×P×P Randomly select and copy H rectangles consisting of adjacent small tiles, each rectangular area consists of l small tiles, and use them as the anchor sample input sequence for data enhancement 5. The ViT baggage re-identification method based on local data enhancement according to claim 1 is characterized in that: In the LocEnViT model in step 5, the anchor sample input sequence of data enhancement in the process of local data enhancement Perform three data enhancements to obtain the corresponding positive sample input sequence And Perform linear projection and add additional information, So as to obtain the corresponding anchor sample label sequence 6. The ViT baggage re-identification method based on local data enhancement according to claim 1, characterized in that: The LocEnViT model in step 5 uses a multi-layer visual conversion module The input token sequence Z o , Perform feature extraction.

7. The ViT baggage re-identification method based on local data enhancement according to claim 1, characterized in that: The LocEnViT model in step 5 uses the identity loss L ID , triple loss L triple and local data enhancement contrast loss L data Train the model; L total =θ1L ID +θ2L triple +θ3L data , subject to θ1+θ2+θ3=1&θ1,θ2,θ3≥0 Identity loss L ID It can be expressed as: Triplet loss L triple It can be expressed as: The contrast loss can be expressed as: and 8. The ViT baggage re-identification method based on local data enhancement according to claim 1, characterized in that: In the LocEnViT model in step 5, the similarity of two feature vectors is measured by the distance calculation formula during the inference process; wherein the cosine distance The calculation formula is as follows: When judging whether two luggage images belong to the same identity, the LocEnViT luggage re-identification model uses Perform cosine distance calculation.

Images