A vit luggage re-identification method based on local data enhancement

Abstract

The application provides a ViT luggage re-identification method based on local data enhancement, aiming to improve the identification accuracy in automatic luggage sorting. Traditional RFID technology is high in cost and susceptible to interference, so the introduction of computer vision technology has become a new trend. The method comprises the following steps: first, a dataset containing 1500 different identity luggage is constructed, each luggage has 10 photos of different angles, and LabelMe and YOLOv5 are used for labeling and detection; second, multiple cameras are arranged on the luggage check-in and sorting pipeline to collect luggage images; then, based on the ViT model, the local data enhancement technology and the position excitation module are applied to improve the attention of the model to the structural features of the luggage and the generalization ability; then, the luggage features are calculated through the multi-layer visual conversion module, and the cosine distance is used for identity comparison; finally, the identity loss, the triplet loss and the contrast loss are used to optimize the model to improve the identification accuracy.

Description

TECHNICAL FIELD

[0001] The application relates to computer vision, in particular to a ViT luggage re-identification method based on local data enhancement. BACKGROUND

[0002] With the accelerated development of the domestic economy and the continuous improvement of people's living standards, more and more people choose to take a plane as a means of travel. According to the data in the "2023 November China Civil Aviation Main Production Index Statistics" released by the Ministry of Transport of the People's Republic of China, the total passenger transport volume of domestic flights in China in 2023 reached 569 million person-times. In the process of taking a plane, usually a passenger will carry multiple pieces of luggage, which results in the number of luggage handled by domestic airports being much larger than the number of passengers. In the face of such a large number of luggage, domestic airports have been unable to meet people's travel needs by relying solely on manual sorting of luggage. In this case, for airports, the importance of an automated luggage sorting system in transporting passengers is increasingly prominent. At present, the mainstream airport automated luggage sorting system adopts the form of radio frequency identification technology (RFID) to sort luggage.

[0003] Radio frequency identification technology uses unique electromagnetic tags and electromagnetic readers for luggage identification and data transmission. As can be seen from the figure, the luggage sorting process is divided into two main stages: the check-in link and the conveyor belt tracking link. In the check-in link, airport staff will attach an electromagnetic tag containing a unique identification code to the luggage to facilitate subsequent tracking and identification work. In the subsequent conveyor belt tracking link, a special electromagnetic reading and writing device is responsible for identifying and tracking the luggage. Specifically, when the luggage passes through the tag after checking in at the check-in desk and enters the conveyor belt, it will move quickly along the conveyor belt, passing through multiple electromagnetic reading and writing device nodes. In this process, each node can identify the electromagnetic tag attached to the passing luggage and integrate the node information traversed by the electromagnetic tag to form the movement trajectory of the luggage, such as passing through node #1, node #3, node #4, etc.

[0004] Although radio frequency identification technology has been widely used in luggage tracking and identification, it still has some inherent limitations. For example, high equipment cost, luggage tracking effect limited by the number of readers, electromagnetic interference may cause identification failure, and other problems. These problems limit the further development and promotion of automated luggage sorting systems. At the same time, in recent years, computer vision technology has shown excellent robustness, accuracy and real-time performance in tracking and monitoring pedestrians and vehicles, showing the potential to handle complex scenes. Therefore, in order to meet people's strong demand for efficient travel, it is necessary to upgrade the luggage tracking system based on radio frequency identification to a luggage tracking system using computer vision technology.

[0005] The baggage tracking system of computer vision realizes the whole cross-camera baggage tracking process by identifying, tracking and re-identifying the baggage in the captured picture. Specifically, this process first involves identifying which objects in the picture captured by the camera are baggage. Next, the system will monitor the moving track of these baggage in real time, completing the single-camera tracking task. The key last step, the system needs to confirm the exact identity of these baggage, realize the matching between the baggage tracks captured by different cameras, and construct the complete path of the baggage. In the process of cross-camera tracking, the baggage re-identification technology plays a similar role to the electromagnetic tag, which identifies the identity of the baggage by analyzing the features of the baggage in the camera picture, which is the core link to complete the cross-camera baggage track matching and construct the complete baggage path.

[0006] Thanks to the rapid development of neural network technology in recent years, baggage identification and tracking within a single camera have shown excellent performance. However, in the aspect of re-identification technology for cross-camera baggage identity matching, the current solution still does not fully meet the actual application requirements, so this paper focuses on the method of baggage re-identification.

[0007] In the specific scenario of baggage sorting pipeline, object re-identification technology faces new challenges, which mainly come from the fact that the accuracy of re-identification is highly dependent on the image quality captured by the camera and whether the appearance features of the baggage remain consistent in different scenarios. The operating environment of the baggage re-identification task is more complex compared to the common pedestrian or vehicle re-identification scenarios, and the complex environmental conditions often cause changes in the appearance features of the same baggage.

[0008] In the baggage sorting pipeline environment, cameras are installed at multiple key positions of the conveyor belt, and there are often other auxiliary sorting equipment near these positions.

[0009] The following briefly explains how auxiliary sorting equipment causes changes in the appearance and posture of baggage:

[0010] (1) Signal indicator light

[0011] There are multiple signal indicator lights on the baggage sorting pipeline that flash at regular intervals. The rapid alternation of light and dark causes the appearance color of some metal-painted baggage to change rapidly.

[0012] (2) Conveyor belt

[0013] The conveyor belt of the baggage sorting pipeline has ups and downs and many curves, and the baggage often reverses during its forward movement, resulting in a large difference in posture before and after the reversal.

[0014] (3) Mechanical swing arm

[0015] The luggage sorting pipeline uses mechanical swing arms for sorting, which change the posture of the luggage when changing the luggage conveying channel.

[0016] (4) Fences

[0017] The fences on the luggage sorting pipeline will block the luggage in the camera's field of view.

[0018] In summary, due to the same piece of luggage may present significantly different surface color and running posture under different camera views, and the camera shot picture may sometimes not completely cover the target luggage. Therefore, in this complex and variable environment, the luggage re-identification task faces the challenge of accurately identifying a large number of target identities.

[0019] In addition, in the airport scene with a daily passenger flow of about 100,000 people, the luggage videos taken from different angles by the sorting pipeline every day provide rich data resources for the training of the luggage re-identification model. However, the large scale and complexity of these data make it infeasible to rely on traditional manual annotation methods. Therefore, fully unsupervised training algorithms become a better choice for training luggage re-identification models using large-scale data sets. However, most existing fully unsupervised re-identification training algorithms are mainly trained and tested on small-scale data sets. In the environment with smaller data volume, these algorithms need to consume a large amount of video memory resources for data comparison and clustering. This approach makes model training more difficult when dealing with large-scale data sets. Therefore, another challenge of unsupervised training of luggage re-identification is how to handle the training of large-scale data sets with limited video memory resources.

[0020] In addition, this paper relies on the "Key Technology Development of Aviation Logistics Multi-source Data Fusion and Coordination" school-level project led by the Second Institute of Civil Aviation Administration of China. In summary, in-depth research and development of luggage re-identification models not only an important scientific research topic, but also has great economic significance. SUMMARY

[0021] The purpose of the present application is to propose a ViT luggage re-identification method based on local data enhancement, which aims to improve the accuracy of luggage re-identification to meet the actual luggage sorting engineering needs.

[0022] The purpose of the present application is achieved as follows:

[0023] The ViT-based baseline re-identification model for multi-scale feature fusion is improved. Through in-depth study of the baggage sorting process, this chapter notes that most of the baggage is made of rigid materials, which means that their structure will not change due to rotation or illumination changes. Therefore, the LocEnViT (Local Enhance Vision Transformer) baggage re-identification model, based on the multi-scale feature fusion ViT baseline re-identification model, applies a ViT-specific image local region data augmentation technique and combines a position excitation module and a contrast learning method during training to compare the "new" features generated by data augmentation with the local features of the original image, thereby improving the generalization ability of the re-identification model.

[0024] The specific way is as follows:

[0025] Step 1, construct a baggage dataset, create a baggage dataset LReID-1500 with identity labels. These videos are recorded by five cameras covering various locations on the sorting pipeline, with a total duration of 148 hours, containing more than 100,000 different identities of baggage. The dataset is made in combination with LabelMe and YOLOv5 tools. LabelMe is used for manual annotation, and YOLOv5 is used for target detection to construct the dataset after cutting the detection results. LReID-1500 contains 1500 different identities of baggage, a total of 15000 pictures, with an average of 10 different angle photos for each piece of baggage. The production steps include: using LabelMe and video to make a small batch of detection dataset, initializing and fine-tuning the YOLOv5 network, generating videos labeled with baggage bounding boxes, extracting one picture every 4 frames, cutting the bounding box to generate a backup dataset, and finally manually screening out duplicate pictures and completing the manual annotation of the unlabeled dataset;

[0026] Step 2, build a local baggage sorting pipeline video acquisition environment, including multiple non-intersecting cameras on the baggage sorting pipeline, a video stream transmission server;

[0027] Step 3, build a local baggage check-in video acquisition environment, including a baggage check-in table camera and a video stream transmission server;

[0028] Step 4, get baggage check-in images and baggage videos taken from different perspectives on the sorting line;

[0029] Step 5, deploy the local baggage re-identification feature calculation LocEnViT model. The LocEnViT baggage re-identification model is improved based on the existing multi-scale feature fusion ViT baseline re-identification model, with special emphasis on the attention to structural features of the same identity baggage, which remain relatively stable even when the surface color and posture of the baggage change.

[0030] Step 6, calculate the re-identification features between different pieces of luggage by using the local luggage re-identification feature calculation model, using the multi-layer visual conversion module and the local feature data enhancement module;

[0031] Step 7, substitute the re-identification features between different pieces of luggage into the distance formula; determine whether the two target pieces of luggage belong to the same identity;

[0032] In step 5, the LocEnViT model uses a sliding window to divide the luggage images in the video into small patches with overlapping pixels. The sliding step is denoted as S, and the size of the small patch is denoted as P. The shape of the overlapping area between two adjacent small patches can be represented as (P-S)×P. Given a target image G∈R H×W×C (H, W, C represent the height, width, and channel number of the target image, respectively), the target image can be divided into N small patches with fixed size, and each small patch can be represented as x i ∈R H×P×P ;

[0033] In step 5, the LocEnViT model converts the small patch x i ∈R H×P×P into an input label sequence Z o :

[0034]

[0035] where v PE ∈R 1×D represents a set of learnable position information vectors that are common to all label sequences, which represents the spatial position information of the small patch in the input image. represents the learnable viewpoint information vector corresponding to different cameras;

[0036] In step 5, the LocEnViT model randomly selects and copies H rectangular regions composed of adjacent small patches during local data enhancement, and each rectangular region is composed of l small patches and used as an anchor sample input sequence for data enhancement i ∈R H×P×P

[0037] In step 5, the LocEnViT model performs three kinds of data enhancement to obtain the corresponding positive sample input sequences

[0038] ​​​​

[0039] The LocEnViT model in the step 5 uses a multi-layer visual conversion module The input mark sequence Z o 、 Feature extraction is performed;

[0040] The LocEnViT model in the step 5 respectively uses identity loss L ID , triplet loss L triple and local data enhancement contrast loss L data to train the model.

[0041] L total = θ1L ID + θ2L triple + θ3L data ,

[0042] subject to θ1+θ2+θ3=1&θ1,θ2,θ3≥0

[0043] The identity loss L ID can be expressed as:

[0044]

[0045] The triplet loss L triple can be expressed as:

[0046]

[0047] The contrast loss can be expressed as:

[0048]

[0049] The LocEnViT model in the step 5 measures the similarity of two feature vectors by a distance calculation formula in the inference process. Wherein the cosine distance The calculation formula is as follows:

[0050]

[0051] When judging whether two luggage images belong to the same identity, the LocEnViT luggage re-identification model performs cosine distance calculation:

[0052]

[0053] The positive effects of the present application are:

[0054] The present application solves the problem of insufficient generalization ability of existing ViT baseline model in processing appearance feature changes of the same luggage and similar different luggage, and proposes a ViT luggage re-identification model based on local data enhancement, which is named as LocEnViT. The core innovation of the LocEnViT luggage re-identification model lies in that a plurality of groups of "new" data are obtained by data enhancement on local regions of images, and the data are sent into a feature extraction network in parallel during the training process, so as to combine a position excitation module and a contrast loss to promote the model to deeply mine the detail features and structural features of the luggage and improve the generalization ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a schematic diagram of luggage data with identity tags.

[0056] Figure 2 LocEnViT re-identification model architecture.

[0057] Figure 3 Schematic diagram of additional information adding process.

[0058] Figure 4 Schematic diagram of self-attention layer structure.

[0059] Figure 5 Schematic diagram of multi-scale feature extraction module.

[0060] Figure 6 Schematic diagram of feature processing module.

[0061] Figure 7 Schematic diagram of local data enhancement.

[0062] Figure 8 Schematic diagram of image position excitation. DETAILED DESCRIPTION

[0063] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0064] A ViT luggage re-identification method based on local data enhancement, the method comprising the following steps:

[0065] Step 1, constructing a luggage dataset, a luggage dataset LReID-1500 with identity tags is created. These videos are recorded by five cameras covering various positions of the sorting pipeline, with a total length of 148 hours, containing more than 100,000 different identities of luggage. The dataset is made by combining LabelMe and YOLOv5 tools. LabelMe is used for manual annotation, and YOLOv5 is used for target detection, and the detection results are cropped to construct the dataset. LReID-1500 contains 1500 different identities of luggage, a total of 15000 pictures, and an average of 10 different angle photos for each luggage.

[0066] Step 2, build a local baggage sorting pipeline video acquisition environment, including multiple non-intersecting camera views of the baggage sorting pipeline, a video stream transmission server;

[0067] Step 3, build a local baggage check-in video acquisition environment, including a baggage check-in table camera and a video stream transmission server;

[0068] Step 4, obtain baggage check-in images and baggage videos taken at different views on the sorting line;

[0069] Step 5, deploy a local baggage re-identification feature calculation LocEnViT model. The LocEnViT baggage re-identification model is improved on the basis of the existing multi-scale feature fusion ViT baseline re-identification model, and special emphasis is placed on the attention to the structural features of the same identity baggage, which remain relatively stable even when the surface color and travel posture of the baggage change.

[0070] Step 6, calculate the re-identification features between different bags using the multi-layer visual transformation module and the local feature data enhancement module through the local baggage re-identification feature calculation model;

[0071] Step 7, substitute the re-identification features between different bags into the distance formula; determine whether the two target bags belong to the same identity;

[0072] The LocEnViT model in step 5 divides the baggage images in the video into small patches with overlapping pixels using a sliding window. The sliding step is denoted as S, and the size of the small patch is denoted as P. The shape of the overlapping area of two adjacent small patches can be represented as (P-S)×P. Given a target image G∈R H×W×C (H, W, C represent the height, width, and channel number of the target image), the target image can be split into N small patches of fixed size, and each small patch can be represented as x i ∈R H×P×P ;

[0073] The LocEnViT model in step 5 converts the small patch x i ∈R H×P×P into an input token sequence Z o :

[0074]

[0075] where v PE ∈R 1×D is a set of learnable position information vectors that are common to all token sequences, representing the spatial position information of the small patch in the input image. representing different camera corresponding learnable view information vectors;

[0076] The LocEnViT model in step 5, in the process of local data augmentation, to small patches x i ∈R H×P×P Randomly select and copy H rectangles composed of adjacent small patches, each rectangle region composed of l small patches, used as anchor sample input sequence for data augmentation

[0077] The LocEnViT model in step 5, in the process of local data augmentation, data augmentation anchor sample input sequence Three kinds of data augmentation to get the corresponding positive sample input sequence And Linear projection and additional information addition, get the corresponding anchor sample label sequence

[0078]

[0079] The LocEnViT model in step 5, using multi-layer visual conversion module The input label sequence Z o , Feature extraction is carried out;

[0080] The LocEnViT model in step 5, respectively using identity loss L ID , triplet loss L triple and local data augmentation contrast loss L data Model training.

[0081] L total = θ1L ID + θ2L triple + θ3L data ,

[0082] subject to θ1+θ2+θ3=1&θ1,θ2,θ3≥0

[0083] Identity loss L ID can be expressed as:

[0084]

[0085]

[0086] Triplet loss L triple can be expressed as:

[0087]

[0088] The contrast loss can be expressed as:

[0089]

[0090] The LocEnViT model in step 5 measures the similarity of two feature vectors through a distance calculation formula during the inference process. The cosine distance The calculation formula is as follows:

[0091]

[0092] When judging whether two luggage images belong to the same identity, the LocEnViT luggage re-identification model performs cosine distance calculation:

[0093]

Claims

1. A ViT luggage re-identification method based on local data enhancement, characterized in that, The method comprises the following steps: Step 1, constructing a luggage dataset, creating a labeled luggage dataset LReID-1500; the video is recorded by five cameras covering various positions of the sorting pipeline, with a total length of 148 hours, containing more than 100,000 different identities of luggage; the dataset is made in combination with LabelMe and YOLOv5 tools; LabelMe is used for manual labeling, and YOLOv5 is used for target detection, and the detection results are cropped to construct the dataset; LReID-1500 contains 1500 different identities of luggage, a total of 15000 pictures, and an average of 10 different angle photos for each luggage; the production steps include: using LabelMe and video to make a small batch of detection dataset, initializing and fine-tuning the YOLOv5 network, generating a video labeled with luggage bounding box, extracting one picture every 4 frames, cropping the bounding box to generate a backup dataset, finally manually screening out duplicate pictures, and completing manual labeling of the unlabeled dataset; Step 2, constructing a local luggage sorting pipeline video acquisition environment, including multiple non-intersecting cameras of the luggage sorting pipeline, a video stream transmission server; Step 3, constructing a local luggage check-in video acquisition environment, including a luggage check-in table shooting camera and a video stream transmission server; Step 4, obtaining luggage check-in images and luggage shooting videos under different views on the sorting line; Step 5, deploy the local luggage weight recognition feature calculation LocEnViT model; the LocEnViT luggage weight recognition model is improved on the basis of the multi-scale feature fusion ViT baseline weight recognition model, the LocEnViT model uses a sliding window to divide the luggage image in the video into small blocks with overlapping pixels, uses linear projection and additional information addition to convert the small blocks x i ∈R H ×P×P into input token sequences Z o Among them, v PE ∈R 1×D Represents a set of learnable location information vectors common to all labeled sequences, thereby characterizing the spatial location information of small patches in the input image; Represented as learnable viewpoint information vectors corresponding to different cameras; the LocEnViT model, during local data augmentation, adjusts small image patches x... i ∈R H×P×P H rectangles, each consisting of l small tiles, are randomly selected and copied. These rectangles are used as the anchor point input sequence for data augmentation. Using a multi-layer visual transformation module Input label sequence Z o , Feature extraction is performed; the input sequence is used as an anchor point for data augmentation during local data augmentation. Data augmentation is performed to obtain the corresponding positive sample input sequence. And on Perform linear projection and add additional information. Thus, the corresponding anchor sample label sequence is obtained Step 6, calculating the re-identification features between different pieces of luggage by using a local luggage re-identification feature calculation model, using a multi-layer visual conversion module and a local feature data enhancement module; Step 7, substituting the re-identification features between different pieces of luggage into a distance formula; judging whether two target pieces of luggage belong to the same identity.

2. The local data augmentation based ViT luggage re-identification method according to claim 1, wherein, The LocEnViT model in step 5 divides the luggage image in the video into small patches with overlapping pixels using a sliding window, specifically: the sliding step is represented as S, the size of the small patch is represented as P, and the shape of the overlapping area of two adjacent small patches is represented as (P-S) x P; under the condition that the target image G is given H×W×- , the target image is split into N small patches with fixed size, and each small patch after splitting is represented as x i ∈R H×P×P , where H, W and C represent the height, width and channel number of the target image respectively.

3. The local data augmentation based ViT luggage re-identification method according to claim 1, wherein, The LocEnViT model in step 5 uses identity loss L ID , triple loss L triple and local data augmentation contrastive loss L data respectively to train the model; L total = θ1L ID + θ2L triple + θ3L data , subject to θ1+θ2+θ3=1&θ1,θ2,θ3≥0 Identity loss L ID is represented as: Triplet loss L triple is represented as: The contrast loss is expressed as: and 4. The local data augmentation based ViT luggage re-identification method according to claim 1, characterized in that, The LocEnViT model in step 5 measures the similarity of two feature vectors through a distance calculation formula in the inference process; wherein the cosine distance The calculation formula is as follows: In judging whether two pieces of luggage images belong to the same identity, the LocEnViT luggage re-identification model calculates the cosine distance according to cosine distance calculation.

Images