Training methods for baggage recognition models, target baggage recognition and tracking methods and systems

Abstract

This invention provides a training method for a baggage recognition model, a target baggage recognition and tracking method, and a system. The training method includes the following steps: constructing an initial baggage recognition model, which includes a feature extraction network, a re-identification feature extraction model, a multimodal attribute feature extraction model, a local feature extraction model, a detection head, an attribute recognition head, a key point localization head, and a feature fusion enhancement module; training the initial baggage recognition model to convergence using a pre-constructed dataset to obtain the baggage recognition model; constructing a total loss using target detection prediction results, attribute classification prediction results, key point regression prediction results, and baggage image labels, and updating the parameters of the baggage recognition model based on the total loss. This invention solves the problem in existing technologies that rely solely on single appearance information for baggage recognition, which easily leads to recognition confusion and low recognition accuracy when baggage appearances are highly similar.

Description

Technical Field

[0001] This invention relates to the field of logistics and transportation technology, and in particular to a training method for a baggage recognition model, a target baggage recognition and tracking method and system. Background Technology

[0002] Currently, in civil aviation transportation, a distributed edge computing architecture is adopted, in which multiple edge computing devices are deployed around the transmission line, and each edge computing device is connected to several baggage cameras. The workflow of this distributed edge computing architecture can be divided into three main stages: (1) Video acquisition stage: Each camera captures a baggage video stream within a set area on the transmission line; (2) Edge processing stage: The edge computing device processes the received video stream in real time, identifies the baggage in the video stream, and then initializes the identified baggage with numbers and extracts features to generate initialization data; (3) Data transmission stage: The processed video stream and feature data are transmitted to the backend system to provide a data foundation for baggage tracking.

[0003] During the collection of baggage videos, the baggage needs to be confirmed when it enters a new designated area. In the existing technology, the confirmation of baggage is usually achieved by recognizing appearance information. When the appearance of baggage is highly similar, using only a single appearance information to identify the baggage can easily lead to confusion and result in low recognition accuracy. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a training method for a baggage recognition model, a target baggage recognition and tracking method, and a system. It solves the problem that existing technologies rely solely on single appearance information for baggage recognition, which can easily lead to confusion when baggage appearances are highly similar, resulting in low recognition accuracy.

[0005] According to an embodiment of the present invention, a method for training a baggage recognition model is characterized by comprising the following steps:

[0006] An initial baggage recognition model is constructed, which includes a feature extraction network, a re-identification feature extraction model, a multimodal attribute feature extraction model, a local feature extraction model, a detection head, an attribute recognition head, a key point localization head, and a feature fusion enhancement module; wherein, the feature extraction network is used to perform convolution processing on the input baggage image to obtain a preliminary feature map;

[0007] The re-identification feature extraction model is used to extract global visual features from the output of the feature extraction network, outputting global visual features. The multimodal attribute feature extraction model is used to extract attribute features from the output of the feature extraction network, outputting attribute features. The local feature extraction model is used to extract local features from the output of the feature extraction network, outputting local discriminative features. The detection head is used to predict the global visual features, outputting target detection prediction results. The attribute recognition head is used to predict the attribute features, outputting attribute classification prediction results. The keypoint localization head is used to predict the local discriminative features, outputting keypoint regression prediction results. The feature fusion enhancement module performs feature fusion enhancement based on the global visual features, the attribute features, and the local discriminative features to obtain the final enhanced feature vector.

[0008] Using a pre-constructed dataset, the initial baggage recognition model is trained until convergence to obtain the baggage recognition model; wherein, the dataset includes several baggage images and corresponding labels for the baggage images, the labels including baggage type labels, location labels, and attribute labels; and a total loss is constructed using the object detection prediction results, the attribute classification prediction results, the key point regression prediction results, and the labels corresponding to the baggage images, and the parameters of the baggage recognition model are updated based on the total loss.

[0009] On the other hand, according to another embodiment of the present invention, a target baggage identification and tracking method includes the following steps:

[0010] Construct a baggage fingerprint database on the baggage conveyor line, the baggage fingerprint database including the final enhanced feature vector of the baggage image and the baggage information associated with the final enhanced feature vector;

[0011] The target luggage image is acquired, and the luggage recognition model obtained by any of the training methods described above is used to recognize the target luggage image to obtain the final enhanced feature vector corresponding to the target luggage image. Based on the final enhanced feature vector corresponding to the target luggage image and the luggage fingerprint database, the luggage information corresponding to the target luggage image is determined. The luggage information includes luggage ID, camera tag, time of entering the camera and time of leaving the camera.

[0012] Based on the luggage information corresponding to the target luggage image, cross-camera luggage trajectory association is performed to achieve cross-camera tracking of the target luggage. The luggage trajectory association adopts the following steps:

[0013] A source camera node set and a target camera node set are constructed using cameras on the baggage conveyor line. The set of edges connecting the source camera node set and the target camera node set is used as the edge set. The source camera node set contains baggage corresponding to the target baggage image, and the target camera node set is the set of baggage that newly appears in the field of view of the target camera.

[0014] The optimal matching scheme between the source camera node set and the target camera node set is calculated based on the weights of the edge sets, and the weights of the corresponding edge sets of each optimal matching scheme are compared with the preset matching threshold.

[0015] If the weight is greater than or equal to the preset matching threshold, the optimal matching scheme is received, and the baggage ID of the source camera node set is assigned to the corresponding matching baggage in the target camera to complete the association of baggage trajectory and realize the tracking of the target baggage.

[0016] Furthermore, according to yet another embodiment of the present invention, a target baggage identification and tracking system is also provided, the target baggage identification and tracking system comprising:

[0017] Luggage camera, used to capture images of luggage;

[0018] An edge computing device for deploying the baggage recognition model obtained by any of the above training methods;

[0019] The server is used to acquire images of target luggage and utilize luggage information corresponding to the images obtained from edge computing devices; and to perform cross-camera luggage trajectory association based on the luggage information corresponding to the images of target luggage, so as to realize cross-camera tracking of target luggage.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] By simultaneously extracting the global visual feature vector, attribute feature vector, and local discriminative feature vector of luggage, parallel extraction of multimodal features of luggage images or videos is achieved. Compared with traditional single appearance features, this effectively solves the feature confusion problem caused by overlapping appearances of highly similar luggage.

[0022] By normalizing and dynamically weighting the three types of feature vectors and adaptively adjusting the feature fusion weights according to the scene state during the fusion process, compared with the traditional static fusion mode, the feature combination can be optimized through scene perception, ensuring the robustness of features in different environments and significantly improving the discriminative power of features in complex scenes.

[0023] By using the final enhanced feature vector obtained by weighted fusion to identify target luggage, a feature representation that combines global discriminative power with local detail preservation is constructed. This solves the problem in the existing technology that relies on only a single appearance information for luggage identification, which can easily lead to confusion when luggage appearances are highly similar, resulting in low recognition accuracy. This improves the accuracy of luggage identification. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the training steps of a baggage recognition model according to an embodiment of the present invention.

[0025] Figure 2 This is a flowchart illustrating the steps of a target baggage identification and tracking method according to another embodiment of the present invention.

[0026] Figure 3 This is an overall structural diagram of a target baggage identification and tracking system according to another embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the term "luggage" as used in this invention refers to various items transported on conveyor systems in scenarios including but not limited to airports and train stations, and its specific forms may include hard-shell suitcases, soft cloth suitcases, backpacks, handbags, cardboard boxes, etc.

[0029] like Figure 1 As shown in the figure, this embodiment of the invention proposes a training method for a baggage recognition model, including the following steps:

[0030] S1. Construct an initial baggage recognition model, which includes a feature extraction network, a re-identification feature extraction model, a multimodal attribute feature extraction model, a local feature extraction model, a detection head, an attribute recognition head, a key point localization head, and a feature fusion enhancement module; wherein, the feature extraction network is used to perform convolution processing on the input baggage image to obtain a preliminary feature map;

[0031] The re-identification feature extraction model is used to extract global visual features from the output of the feature extraction network, outputting global visual features. The multimodal attribute feature extraction model is used to extract attribute features from the output of the feature extraction network, outputting attribute features. The local feature extraction model is used to extract local features from the output of the feature extraction network, outputting local discriminative features. The detection head is used to predict the global visual features, outputting target detection prediction results. The attribute recognition head is used to predict the attribute features, outputting attribute classification prediction results. The keypoint localization head is used to predict the local discriminative features, outputting keypoint regression prediction results. The feature fusion enhancement module performs feature fusion enhancement based on the global visual features, the attribute features, and the local discriminative features to obtain the final enhanced feature vector.

[0032] Specifically, the feature extraction network extracts multi-level visual features based on a pre-trained CNN model and finally outputs a preliminary feature map. The pre-trained CNN model is ResNet-18 (ResNet-18 is an existing classic convolutional neural network architecture, first proposed by Kaiming He et al. in the 2015 paper "Deep Residual Learning for Image Recognition"). The re-identification feature extraction model uses the FastReID network (often written as "fast-ReID" or "FastReID", an existing open-source Person Re-Identification framework released by the JD AI Research Institute (JDAI) team around 2020. It is not a single "network" structure, but a modular, configurable deep learning library that incorporates multiple mainstream Re-ID methods and training / inference pipelines). The FastReID network highly supports classic re-identification structures such as PCB, MGN, and BOT, and uses them as built-in strong baseline models. The framework integrates baselines, allowing developers to directly call them or perform secondary development based on these structures. By inputting the preliminary feature map of the luggage image into the pre-trained re-identification feature extraction model, the global visual feature vector of the luggage is output.

[0033] Multimodal attribute feature extraction models include, but are not limited to, CLIP or visual language models, which contain visual encoders and text encoders. The visual encoder typically employs Vision... The Transformer is responsible for encoding the input image into a high-dimensional visual vector. The text encoder, based on the Transformer, encodes the pre-stored text description (including the luggage's orientation, color, and material, such as "black hard-shell suitcase") into a linguistic vector in the same vector space as the visual vector. Then, it is pre-trained on a massive number of image-text pairs. By comparing the loss of the visual features extracted from the image and the linguistic features transformed from the text description, the model minimizes the distance between the correctly paired image and text in the vector space and maximizes the distance between negative samples. During pre-training, the multimodal attribute feature extraction model takes the luggage image (or its ROI region) as input and the attribute text "front", "black", "hard", etc. as output. The output is an attribute feature vector mapped to 512 dimensions, which naturally carries semantic information and supplements the high-level attribute differences that are difficult to capture by the visual branch alone. Through this pre-training, the large model can learn rich semantic attributes (color, material, orientation, etc.) and has good generalization ability. Moreover, this model not only focuses on the overall appearance but also understands abstract attributes such as material, texture, and direction, thus more effectively distinguishing similar luggage.

[0034] The local feature extraction model employs a MobileNetV2-based heatmap regression network. This network uses the inverted residuals structure of MobileNetV2 as the feature extractor, retains linear bottleneck layers to avoid distortion of low-dimensional features, removes the original classification head, and retains the features.18 layer to output a high-dimensional feature map of 14×14×320. By inputting the preliminary feature map of the luggage image into the pre-trained local feature extraction model, the model can define a small fixed-size window around each localized visible keypoint (xi,yi), extract features within this window using RoIAlign, and then concatenate or add these features to obtain local discriminative features (512 dimensions).

[0035] After obtaining the global visual feature vector, attribute feature vector, and local discriminative feature vector of the luggage, the feature fusion enhancement module is used to perform feature fusion enhancement on the global visual feature vector, attribute feature vector, and local discriminative feature vector to obtain the final enhanced feature vector. In the entire feature fusion enhancement process, in order to make the features from different sources on the same scale, the three types of feature vectors are first normalized, and the specific calculation formula is as follows:

[0036]

[0037] Among them, F global F attr and F local These are the global visual feature vector, attribute feature vector, and local discriminative feature vector, respectively; d is the feature dimension (the global visual feature vector is 1×512-dimensional, the attribute feature vector is 1×64-dimensional, and the local discriminative feature vector is 1×512-dimensional).

[0038] The weights for the three types of feature vectors are then generated. The specific process is as follows: First, the three types of feature vectors are concatenated to form a temporary "context vector" F. context Specifically:

[0039] F context =Concat(F′ global F′ attr F′ local )

[0040] Then F context Given a 1088-dimensional vector, this context vector is then fed into a small feedforward neural network (also known as a "gated" or "attention" network), which outputs three weight values: W = Softmax(FC2(ReLU(FC1(F...)) context In this structure, FC1 and FC2 are two fully connected layers used to learn nonlinear relationships from context. For example, FC1 maps 1088 dimensions to 128 dimensions, and FC2 maps 128 dimensions to 3 dimensions. ReLU is the activation function, and Softmax ensures that the three weights W of the output are equal. global W attr W local The sum is 1 (W) global +W attr +W local =1), and all are positive numbers; at the same time, W global W attr W local The value of W can be dynamically adjusted according to the scene state. For example, when the reliability of the global visual feature vector decreases in certain scenes (such as when visual features are weak in low light environments), W can be automatically reduced. global Increase W attr or W local This improves overall robustness.

[0041] Finally, based on the generated dynamic weights, the normalized feature vectors are summed using a weighted average. The specific calculation formula is as follows:

[0042] F fused =W global ·F′ global +W attr ·F′ attr +Wlocal ·F′ local

[0043] Note that during weighted summation, it is necessary to ensure that the three feature vectors have the same dimension. A common practice is to map them to the same dimension (e.g., 512 dimensions) through independent fully connected layers before fusion, thus obtaining a final 512-dimensional feature vector F. fused The feature vector F can be used to... fused Directly used as the final enhanced feature vector F final Alternatively, to further integrate the information, F can be... fused Features F of the original splicing context (After dimension alignment) they are combined again, passed through a final fully connected layer to obtain the final representation vector, and this final representation vector F is then used. final As the final enhanced feature vector F final (The final enhanced feature vector is a high-dimensional vector that integrates global, local, and attribute information. This vector is one of the core outputs of this invention and is mainly used for subsequent high-precision similarity calculation and retrieval tasks.)

[0044] When weighting and fusing the three types of feature vectors, the advantage of using a dynamic weighting mechanism lies in its adaptability: for a piece of luggage with very distinctive features (such as a large and unique sticker), the model may learn to assign a very high W. local Weights allow local features to dominate the final representation; for a black hard-shell box with a very ordinary appearance and no significant local features, the model might assign a higher weight to W. global and W attr The weights rely more on their overall shape, color, and material properties for differentiation. Thus, the final enhanced feature vector F obtained through fusion... final It not only contains multi-dimensional information, but the proportion of this information is dynamically adjusted according to the specific situation of the current luggage, which greatly improves the robustness and discriminative power of the features, thereby effectively solving the problem of recognition confusion caused by the overall similarity of appearance.

[0045] S2. Using a pre-constructed dataset, train the initial baggage recognition model until convergence to obtain the baggage recognition model; wherein, the dataset includes several baggage images and corresponding labels for the baggage images, the labels including baggage type label, location label, and attribute label.

[0046] Specifically, the pre-built dataset includes several luggage images that have been extracted through object detection and matting. The luggage images can come from a library of previously captured static images or be captured in real time by a camera on site. In the pre-training stage, static data is used to build an initial luggage recognition model, and the luggage recognition model is obtained by training the initial luggage recognition model.

[0047] S3. Construct a total loss using the target detection prediction result, the attribute classification prediction result, the key point regression prediction result, and the label corresponding to the baggage image, and update the parameters of the baggage recognition model based on the total loss.

[0048] Specifically, during training, a loss function is used for supervision, where the total loss function L... total It is the weighted sum of the losses from each task. The specific formula for calculating the total loss is:

[0049] L total =λ det ·L det +λ attr ·L attr +λ key ·L key

[0050] Where, λ det , λ attr and λ key λ is a hyperparameter used to balance the three sub-losses (detection loss, attribute recognition loss, and keypoint localization loss). Its value significantly affects the contribution of each branch to the final goal. det The value of λ is 0.7. attr The value of λ is 0.2. key The value is 0.1.

[0051] L det (Detection Loss): This is the standard object detection loss, which measures the difference between the model's predicted location and range of luggage and the actual location and range. It is constructed based on the difference between the predicted output of the detection head and the actual detection labels, and consists of three parts. The specific calculation formula is as follows:

[0052] L det =W box ·L box +W obj ·L obj +W cls ·L cls

[0053] Among them, L boxThe bounding box regression loss is calculated using the CIoULoss loss function, which simultaneously considers the overlapping area, center point distance, and aspect ratio, resulting in faster and more stable convergence; L obj The object confidence loss, calculated using the binary cross-entropy loss (BCEWithLogitsLoss) function, is used to determine whether a predicted bounding box contains the target; L cls The category loss, also calculated using binary cross-entropy loss, is used to distinguish luggage categories, such as suitcases / backpacks; W box W obj and W cls L respectively box L obj and L cls The internal weights are assigned values ​​based on the actual situation, but must satisfy W. box +W obj +W cls =1.

[0054] L attr (Attribute Recognition Loss): This loss function independently measures the model's prediction accuracy for each attribute, allowing the model to predict multiple attributes simultaneously. It is constructed based on the difference between the predicted output of the attribute recognition head and the true attribute labels. Since this is a multi-label classification task (a piece of luggage can have multiple attributes simultaneously), the most suitable loss function is the binary cross-entropy loss, i.e.: L attr =BCE(σ(p) i ),t i ), where p i It is the model's predicted output for the i-th attribute (logits, i.e., the value without Sigmoid activation), t i σ is the truth label (0 or 1) of the i-th attribute, and σ is the Sigmoid activation function.

[0055] L key (Keypoint localization loss): This is constructed based on the difference between the predicted output of the keypoint localization head and the actual keypoint localization labels. It employs the Smooth L1 Loss function, which measures the Euclidean distance between the model-predicted keypoint locations and the actual locations. The specific calculation formula for the Smooth L1 Loss function is as follows:

[0056]

[0057] Where, N key N represents the total number of predefined keypoints. vis Indicates the number of visible keypoints in the image; and v represents the predicted coordinates and the actual coordinates of the i-th keypoint, respectively; i The visibility label for the i-th keypoint, when v i When x > 0, it participates in the calculation; the SmoothL1(x) function is defined as:

[0058]

[0059] Furthermore, the output process of the target detection prediction result specifically includes:

[0060] A re-identification feature extraction model is used to extract global visual features from the preliminary feature map, and the global visual features are output.

[0061] The global visual features are input to the detection head, which predicts the bounding box coordinates and confidence scores of the luggage in the global visual features, and outputs several grid cells of the luggage. Each grid cell generates several candidate boxes, and the output of each candidate box includes coordinates and confidence scores.

[0062] Candidate boxes with confidence scores below a preset threshold and overlapping candidate boxes are filtered out, and the ROI region of the luggage is generated and output. The ROI region is the effective pixel region in the image that is related to the luggage.

[0063] Global average pooling is performed on the feature map within the ROI region to output the target detection prediction result.

[0064] Specifically, the detection head is similar to a standard object detector (such as YOLOv5 / v8) and is used to extract features that can effectively distinguish the overall visual appearance of luggage. The detection head adopts mainstream Re-ID networks (such as PCB, MGN, BOT and other classic structures). The initial feature map of the luggage is input into the re-identification feature extraction model for global visual feature extraction, which outputs global visual features. Then, the global visual features are input into the detection head, which predicts the bounding box coordinates and confidence of the luggage based on the input global visual features, and outputs several grid cells. Each grid cell generates multiple candidate boxes. The output format of the candidate boxes includes coordinates, confidence and class probability. Then, overlapping candidate boxes are filtered out, and the ROI region of the luggage (i.e., the effective pixel region in the luggage image related to the target luggage) is generated and output. Then, the feature map in the ROI region is convolved and global average pooled to obtain the target detection prediction result of the luggage.

[0065] Furthermore, the output process of the attribute classification prediction result specifically includes:

[0066] A multimodal attribute feature extraction model is used to extract attribute features from the preliminary feature map, and the attribute features are output.

[0067] The attribute features are then encoded into high-dimensional visual vectors, and the pre-stored attribute text descriptions are encoded into language vectors in the same vector space as the visual vectors. The attribute text descriptions include the luggage's orientation, color, and material.

[0068] The distance between the visual vector and the language vector is calculated one by one. The visual vector and the language vector are matched based on the rule that the distance between the correctly matched image and the text is minimized in the vector space. The specific score of the attribute feature is output, the specific score of the attribute feature is input into the attribute recognition head, and finally the attribute classification prediction result is output.

[0069] Specifically, the attribute recognition head can capture higher semantic attribute features using a large model to supplement the deficiencies of overall visual features. The attribute recognition head is a multi-label classification branch. After the initial feature map of the luggage is input into the multimodal attribute feature extraction model for attribute feature extraction, the attribute features will be output. Then, the multimodal attribute feature extraction model will encode the attribute features (image features within the ROI region) into visual vectors and encode the pre-stored attribute text descriptions (including the luggage's orientation, color, and material) into language vectors in the same vector space as the visual vectors. The distance between the visual vectors and the language vectors will be calculated one by one. Based on the rule of minimizing the distance between the correctly matched image and text in the vector space, the visual vectors and the language vectors will be matched, and the specific score of the attribute feature will be output. The specific score of the attribute feature will be input into the attribute recognition head, and finally, the attribute classification prediction result will be output.

[0070] Furthermore, the output process of the keypoint regression prediction results specifically includes:

[0071] A local feature extraction model is used to extract local features from the preliminary feature map and output local discriminative features. Specifically, a small fixed-size window is defined around each located visible key point, and features within the window are extracted from the preliminary feature map. The extracted features are then concatenated or added together to obtain the local discriminative features.

[0072] The local discriminative features are input into the key point positioning head, and the key point regression prediction results are output, so that the key point positioning head can locate the coordinates and visibility of multiple predefined key points on the luggage;

[0073] Specifically, the keypoint localization head is a regression branch, similar to human pose estimation (keypoint detection uses a MobileNetV2-based heatmap regression network). After the local discriminative features of the luggage are input into the keypoint localization head, the keypoint localization head can predict the coordinates (xi, yi) and visibility vi of N predefined keypoints on the luggage (such as the top of the handle, the two wheels, and the center of the brand logo), and output the keypoint regression prediction results.

[0074] Furthermore, before inputting the local discriminative features into the keypoint localization head, the spatial features are enhanced by an attention module, specifically including:

[0075] Input local discriminative features into the attention generation module to generate a spatial attention map of the same size as the initial feature map;

[0076] The attention map is multiplied by the preliminary feature map one by one, so that the key features of the preliminary feature map are enhanced and the irrelevant features are weakened.

[0077] Global average pooling is performed on the enhanced preliminary feature map to obtain an attention-based enhanced local discriminative feature vector.

[0078] Specifically, the attention generation module is a lightweight convolutional module (using CBAM or ECA attention mechanisms, for example, a 7x7 convolution followed by a Sigmoid activation function). After the local discriminative features of the luggage are input into the attention generation module, the attention generation module generates a spatial attention map of the same size as the initial feature map (the value of each pixel in the map (between 0 and 1) represents the model's judgment of the "importance" of that spatial location; ideally, areas such as luggage stickers, logos, and special wear will have higher values). Then, the attention map is multiplied element-wise (dot product) with the initial feature map, so that the features of the most discriminative regions are "enhanced" while the features of the background or ordinary regions are "suppressed". The enhanced feature map is then pooled within the ROI region to obtain an enhanced local discriminative feature vector (512 dimensions) based on attention.

[0079] like Figure 2 As shown, another embodiment of the present invention also proposes a target baggage identification and tracking method, including the following steps:

[0080] S4. Construct a baggage fingerprint database on the baggage conveyor line. The baggage fingerprint database includes the final enhanced feature vector of the baggage image and the baggage information associated with the final enhanced feature vector.

[0081] Specifically, as each piece of luggage on the baggage conveyor passes a camera, the camera captures images or videos of the luggage in real time. Luggage with the same tag and within the same time period are ranked by their location scores. The bag with the highest score is taken as the clearest image corresponding to that tag. The final enhanced feature vector of this clearest image is extracted and calculated through the above process. This final enhanced feature vector, along with its corresponding time, luggage ID, and other information, is stored to form a structured "luggage fingerprint database" (e.g., {Luggage ID_1:[F_final_1,Cam_ID,Time,...],Luggage ID_2:[F_final_2,Cam_ID,Time,...],...}).

[0082] S5. Acquire the target luggage image and use the luggage recognition model obtained by any of the training methods described above to recognize the target luggage image, so as to obtain the final enhanced feature vector corresponding to the target luggage image, and determine the luggage information corresponding to the target luggage image based on the final enhanced feature vector corresponding to the target luggage image and the luggage fingerprint database. The luggage information includes luggage ID, camera tag, entry time and exit time of the camera.

[0083] Specifically, after acquiring the target luggage image, the pre-trained model is called to process the target luggage image or video, extracting the final enhanced feature vector of the target luggage image or video. Then, cosine similarity or Euclidean distance is used to match the final enhanced feature vector of the target luggage with the final enhanced feature vector of each luggage in the "luggage fingerprint database". The luggage with the highest similarity or the smallest distance is selected as the matching luggage corresponding to the target luggage image, and the luggage information of the matching luggage is output. The luggage information includes the following parts:

[0084] Luggage Category: A general classification of luggage, such as "suitcase", "backpack", "handbag", etc.

[0085] Luggage Attribute Set: A series of labels that describe the fine-grained characteristics of luggage, such as {color: black, material: hard shell, stickers: yes, ...}.

[0086] The bounding box of luggage: its precise location in the image, in the format [x,y,w,h].

[0087] Keypoints: Coordinates of predefined local areas on the luggage, such as [coordinates of the top of the handle, coordinates of the left wheel, coordinates of the right wheel, ...].

[0088] Final Enhanced Feature Vector: This is a high-dimensional vector that represents the final result by fusing global, local, and attribute information. This vector is one of the core outputs of this invention and is primarily used for subsequent high-precision similarity calculation and retrieval tasks.

[0089] It also includes baggage ID, camera tag, time of entering the camera, and time of leaving the camera.

[0090] S6. Based on the luggage information corresponding to the target luggage image, cross-camera luggage trajectory association is performed to achieve cross-camera tracking of the target luggage.

[0091] The baggage tracking is accomplished using the following steps:

[0092] A source camera node set and a target camera node set are constructed using cameras on the baggage conveyor line. The set of edges connecting the source camera node set and the target camera node set is used as the edge set. The source camera node set contains baggage corresponding to the target baggage image, and the target camera node set is the set of baggage that newly appears in the field of view of the target camera.

[0093] The optimal matching scheme between the source camera node set and the target camera node set is calculated based on the weights of the edge sets, and the weights of the corresponding edge sets of each optimal matching scheme are compared with the preset matching threshold.

[0094] If the weight is greater than or equal to the preset matching threshold, the optimal matching scheme is received, and the baggage ID of the source camera node set is assigned to the corresponding matching baggage in the target camera to complete the association of baggage trajectory and realize the tracking of the target baggage.

[0095] Specifically, within the field of view of a single camera, once a piece of luggage (i.e., the luggage corresponding to the target luggage image) is detected, its possible position and time in the next few frames can be predicted using a Kalman filter or a simple motion model (such as a uniform velocity model). When the luggage enters the field of view of the next camera, its appearance time and position should roughly match the prediction of the previous camera. Therefore, luggage detected by different cameras within a similar time period is regarded as nodes in the image. The edge weights between nodes can be calculated based on feature similarity, temporal continuity (the time of entering a new camera should be later than the time of leaving the previous camera, and the interval should be within a reasonable range), and spatial continuity (if the camera position is known, the approximate conveyor belt path can be estimated). Then, the optimal cross-camera luggage association is found through algorithms such as the Hungarian algorithm or more complex graph matching algorithms.

[0096] Specifically:

[0097] Define a source camera node set U and a target camera node set V. U represents the set of luggage that has formed a complete trajectory and left within the field of view of the source camera. Each node U... i ∈U, each u i Each piece of luggage corresponds to a unique luggage ID and its complete trajectory information (including departure time, departure location, and the extracted final enhanced feature vector). V represents the newly appearing set of luggage in the field of view of the target camera, and each node v j ∈V, each v j For a newly detected piece of luggage (not yet assigned an ID), this includes its appearance time, location, and the extracted final enhanced feature vector. Additionally, the edges connecting U and V are defined as edge set E, where each edge e(i,j) represents the luggage u from the source camera. i Luggage v with target camera j Assuming the match is for the same object, the weight W(u) of each edge e(i,j) is... i ,v j ) is a comprehensive similarity score that quantifies u i and v j The probability that they are the same piece of luggage is weighted by the feature similarity S. feat Time continuity S time and spatial continuity S spat The weighted product of the three components is obtained, with the weight range being [0,1]. A higher value indicates a greater probability of matching. The specific calculation formula is as follows:

[0098] W(u i ,v j ) = S feat ·α+S time ·β+S spat ·γ

[0099] Where α, β, and γ are the feature similarity S, respectively. feat Time continuity S time and spatial continuity S spat The hyperparameters are used to adjust the relative importance of each component (for example, in scenarios where the physical distance between cameras is very close and the time interval is stable, the weights of β and γ can be increased; in scenarios with extremely high feature recognition, the weight of α can be increased), and must satisfy α+β+γ=1;

[0100] Feature similarity S feat : Used to measure the similarity in appearance between two pieces of luggage, directly using the final enhanced feature vector F obtained by extraction and calculation as described above. final The calculation can be performed using cosine similarity, specifically as follows:

[0101] S feat(u i ,v j )=(F final_i ·F final_j ) / (||F final_i ||2*||F final_j ||2)

[0102] Among them, F final_i Represents node u i The corresponding final enhanced feature vector of the luggage, F final_j Represents node v j The corresponding luggage's final enhanced feature vector, where · represents the vector dot product, ‖·‖2 represents the L2 norm (Euclidean length) of the vector, and the feature similarity S feat The range is [-1, 1], and it can be obtained through (1+S feat Normalized to [0,1], 1 / 2 measures the consistency of two eigenvectors in direction and is insensitive to changes in illumination.

[0103] Time continuity S time This is used to measure whether the time interval between luggage leaving the source camera and appearing at the target camera is reasonable. A Gaussian function can be used to model this reasonableness (this Gaussian function represents the time difference between the actual transit time Δt and the expected time T). exp When closest, S time The maximum value of 1 is achieved as Δt deviates from T. exp S time (It decreases exponentially), specifically:

[0104]

[0105] Where Δt represents v j Subtract u from the time of occurrence i If the time difference between departure times is Δt≤0, then it is an impossible event, and the similarity is directly 0; T exp This represents the empirically estimated transit time, which can be estimated based on the average speed and distance of the conveyor belt between the two cameras, or obtained through historical data statistics; σ t σ represents the standard deviation of time tolerance. t The larger the value, the higher the system's tolerance for changes in conveyor belt speed or luggage stopping.

[0106] Spatial continuity S spat This function measures whether the predicted departure position of luggage at the source camera and its actual appearance position at the target camera conform to the motion pattern. This requires using a Kalman filter for state prediction (this function indicates that the closer the actual appearance position is to the predicted position, the better the motion pattern). spat The higher the score, the better.

[0107]

[0108] Where d represents spatial distance, d = ||P pred_i -P enter_j ||2,P pred_i Indicates when luggage u i When tracking in the source camera, its associated Kalman filter continuously predicts its state, P pred_i It predicts its departure from the source camera and its passage through T based on its last observed state (position, velocity). exp After a certain time, it should appear at the predicted position in the entrance area of ​​the target camera. This prediction needs to be combined with the coordinate system transformation of the two cameras; P enter_j luggage j The location where the target camera first detects the target within its field of view; σ s σ represents the standard deviation of spatial tolerance. s The larger the value, the higher the tolerance for error in the predicted location.

[0109] After calculating all possible matching pairs (u i ,v j The overall weight W(u) i ,v j After that, the goal is to find a globally optimal matching scheme that maximizes the sum of the weights of all matched baggage pairs and satisfies the constraint that each baggage is matched at most once. This is a typical assignment problem, which can be solved using the Hungarian Algorithm or its more efficient implementations (such as the Jonker-Volgenant algorithm).

[0110] Detailed matching steps:

[0111] First, construct the cost matrix: The Hungarian algorithm solves the problem of minimizing the total cost; therefore, the problem of maximizing similarity needs to be transformed into a problem of minimizing cost. The elements C of the cost matrix C... ij Defined as:

[0112] C ij =1-W(u) i ,v j )

[0113] Thus, the higher the weight, the lower the cost, and the dimension of the cost matrix is ​​|U|x|V|.

[0114] Next, the Hungarian algorithm is applied: the cost matrix C is taken as input, and the Hungarian algorithm is called. The implementation of this algorithm is usually a standard library function (e.g., scipy.optimize.linear_sum_assignment in Python's SciPy library).

[0115] Then, the matching results are parsed: the algorithm outputs a set of paired indices (i,j), representing the optimal allocation scheme. For example, the output [(0,2), (1,0)] indicates that u0 matches v2 and u1 matches v0.

[0116] Finally, set a matching threshold: To prevent two completely unrelated bags from being forcibly matched, a matching threshold τ needs to be set. match For each matching pair (u) given by the algorithm i ,v j ), check its original comprehensive weight W(u) i ,v j ), if W(u i ,v j )≥τ match If so, then accept the match and set u i The ID is assigned to v j If the trajectories are associated, the matching is completed; otherwise, the match is rejected, even if they are in the optimal solution, because their similarity is too low to be certain that they are the same luggage.

[0117] After rejecting the optimal matching scheme, there are unmatched source camera nodes and target camera nodes. The processing steps are as follows:

[0118] For a source camera node that does not match, indicating that the luggage has left the source camera but has not successfully matched with any new luggage at any target camera, the luggage is marked as having left;

[0119] For unmatched target camera nodes, indicating that the luggage appeared in the target camera but did not successfully match any luggage from the source camera, a brand new, unique ID is assigned to the luggage to indicate that the luggage is a newly entered luggage.

[0120] In addition, during continuous tracking within a single camera or short-term cross-camera switching, if a piece of luggage briefly disappears from a camera (e.g., due to occlusion by other objects or missed detection by the model), but then reappears within a short time in the same or a nearby camera with consistent features and motion trajectory, the system should be able to associate it back to its original number. In this case, a lifecycle-based tracking strategy is used, specifically:

[0121] First, define the track status: define three statuses for each baggage track in the system: Active means that the track has successfully matched a detection box in the most recent frame; Lost means that the track has not matched any detection box in the current frame, but its life cycle has not ended; Terminated means that the track has been continuously lost for more than the preset maximum time limit and has been confirmed as having left the field of view or been permanently lost.

[0122] Next, tracking and state transitions are performed: When a detection box cannot be associated with any existing Active or Lost trajectory, a new Active trajectory is initialized, and matching is then performed. In each frame, a Kalman filter is first used to predict the positions of all Active trajectories in the current frame. Then, the new detection box is matched with these predicted positions (e.g., via IOU or Mahalanobis distance). If a successful match is achieved, the state of the Active trajectory (including position, feature vector, etc.) is updated, and its Active state is maintained. If an Active trajectory does not match any detection box in the current frame, its state becomes Lost (the state is the first loss), and a "loss timer" (initialized to 1) is started. The final augmented feature vector F of this trajectory is... final The Kalman filter state is preserved; if the trajectory of a Lost state is still not matched in subsequent frames (the state is continuously lost), the count of its "loss timer" gradually increases.

[0123] Then, re-association is performed. In each frame, for new bounding boxes that fail to be associated with the active trajectory, the system attempts to perform a secondary match between them and all lost-state trajectories. The secondary matching criterion is: since the object may have moved a considerable distance, the IOU is no longer reliable. At this point, the feature similarity S... feat This becomes the primary criterion for calculating the final enhanced feature vector F of the new detection box. finak_new The final enhanced feature vector F saved with all Lost trajectories final_lost The cosine similarity between them, if S feat (F final_new F final_lost If the value exceeds a preset high re-identification threshold, the lost luggage is considered to have been found. At this point, the state of the Lost trajectory is restored to Active, its ID is assigned to a new detection box, the "Lost Timer" is reset to 0, and the Kalman filter of the trajectory is corrected and updated using the position information of the new detection box.

[0124] Finally, the track is terminated (Lost->Terminated): The system periodically checks all tracks in the Lost state. If the loss time of a track exceeds the preset maximum allowable loss time (e.g., 30 frames or 2 seconds), it is considered that the baggage has indeed left, its state is set to Terminated, and it is removed from the current tracking pool.

[0125] Through the aforementioned lifecycle management and secondary re-identification association mechanism, the system can effectively cope with trajectory interruptions caused by short-term occlusion or model miss detection, reconnecting fragmented trajectory segments to ensure the continuity and stability of baggage IDs.

[0126] On the other hand, such as Figure 3 As shown, another embodiment of the present invention also provides a target baggage identification and tracking system, the target baggage identification and tracking system comprising:

[0127] Luggage camera, used to capture images of luggage;

[0128] An edge computing device for deploying the baggage recognition model obtained by any of the above training methods;

[0129] The server is used to acquire images of target luggage and utilize luggage information corresponding to the images obtained from edge computing devices; and to perform cross-camera luggage trajectory association based on the luggage information corresponding to the images of target luggage, so as to realize cross-camera tracking of target luggage.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A training method for a baggage recognition model, characterized in that: Includes the following steps: An initial baggage recognition model is constructed, which includes a feature extraction network, a re-identification feature extraction model, a multimodal attribute feature extraction model, a local feature extraction model, a detection head, an attribute recognition head, a key point localization head, and a feature fusion enhancement module; wherein, the feature extraction network is used to perform convolution processing on the input baggage image to obtain a preliminary feature map; The re-identification feature extraction model is used to extract global visual features from the output of the feature extraction network, outputting global visual features. The multimodal attribute feature extraction model is used to extract attribute features from the output of the feature extraction network, outputting attribute features. The local feature extraction model is used to extract local features from the output of the feature extraction network, outputting local discriminative features. The detection head is used to predict the global visual features, outputting target detection prediction results. The attribute recognition head is used to predict the attribute features, outputting attribute classification prediction results. The keypoint localization head is used to predict the local discriminative features, outputting keypoint regression prediction results. The feature fusion enhancement module performs feature fusion enhancement based on the global visual features, the attribute features, and the local discriminative features to obtain the final enhanced feature vector. Using a pre-constructed dataset, the initial baggage recognition model is trained until convergence to obtain the baggage recognition model; wherein, the dataset includes several baggage images and corresponding labels for the baggage images, the labels including baggage type labels, location labels, and attribute labels; and the total loss is constructed using the object detection prediction results, the attribute classification prediction results, the key point regression prediction results, and the labels corresponding to the baggage images, and the parameters of the baggage recognition model are updated based on the total loss; The process of outputting the target detection prediction results specifically includes: A re-identification feature extraction model is used to extract global visual features from the preliminary feature map, and the global visual features are output. The global visual features are input to the detection head, which predicts the bounding box coordinates and confidence scores of the luggage in the global visual features, and outputs several grid cells of the luggage. Each grid cell generates several candidate boxes, and the output of each candidate box includes coordinates and confidence scores. Candidate boxes with confidence scores below a preset threshold and overlapping candidate boxes are filtered out, and the ROI region of the luggage is generated and output. The ROI region is the effective pixel region in the image that is related to the luggage. Global average pooling is performed on the feature map within the ROI region to output the target detection prediction result; The output process of the attribute classification prediction results specifically includes: A multimodal attribute feature extraction model is used to extract attribute features from the preliminary feature map, and the attribute features are output. The attribute features are then encoded into visual vectors, and the pre-stored attribute text descriptions are encoded into language vectors in the same vector space as the visual vectors. The attribute text descriptions include the luggage's orientation, color, and material. The distance between the visual vector and the language vector is calculated one by one. The visual vector and the language vector are matched based on the rule that the distance between the correctly matched image and the text is minimized in the vector space. The specific score of the attribute feature is output. The specific score of the attribute feature is input into the attribute recognition head, and the attribute classification prediction result is finally output. The output process of the keypoint regression prediction results specifically includes: A local feature extraction model is used to extract local features from the preliminary feature map and output local discriminative features. Specifically, a small fixed-size window is defined around each located visible key point, and features within the window are extracted from the preliminary feature map. The extracted features are then concatenated or added together to obtain the local discriminative features. The local discriminative features are input into the key point positioning head, and the key point regression prediction results are output, so that the key point positioning head can locate the coordinates and visibility of multiple predefined key points on the luggage.

2. The training method for a baggage recognition model as described in claim 1, characterized in that: The specific formula for calculating the total loss is as follows: in, The detection loss is constructed based on the difference between the predicted output of the detection head and the actual detection label. The detection loss is a weighted fusion of bounding box regression loss, object confidence loss and category loss. The bounding box regression loss is calculated using the CIoULoss loss function, and the object confidence loss and category loss are calculated using the binary cross-entropy loss function. The attribute recognition loss is constructed based on the difference between the predicted output of the attribute recognition head and the true attribute label, and is calculated using the binary cross-entropy loss function. The key point localization loss is constructed based on the difference between the predicted output of the key point localization head and the actual key point localization label, and is calculated using the smooth L1 loss function. , and These are the hyperparameter weights for detection loss, attribute recognition loss, and keypoint localization loss, respectively. The value is 0.

7. The value is 0.

2. The value is 0.

1.

3. The training method for a baggage recognition model as described in claim 1, characterized in that: Before inputting the local discriminative features into the keypoint localization head, the spatial features are further enhanced by an attention module, specifically including: Input local discriminative features into the attention generation module to generate a spatial attention map of the same size as the initial feature map; The attention map is multiplied by the preliminary feature map one by one, so that the key features of the preliminary feature map are enhanced and the irrelevant features are weakened. Global average pooling is performed on the enhanced preliminary feature map to obtain attention-based enhanced local discriminative features.

4. The training method for a baggage recognition model as described in claim 1, characterized in that: The specific formula for calculating the final enhanced feature vector is as follows: in, , and These are the global visual feature vector, attribute feature vector, and local discriminative feature vector extracted by the re-identification feature extraction model, the multimodal attribute feature extraction model, and the local feature extraction model, respectively. For feature dimensions; , , They are respectively , and The weights, and .

5. A method for identifying and tracking target baggage, characterized in that: Includes the following steps: Construct a baggage fingerprint database on the baggage conveyor line, the baggage fingerprint database including the final enhanced feature vector of the baggage image and the baggage information associated with the final enhanced feature vector; The target luggage image is acquired, and the luggage recognition model obtained by the training method described in any one of claims 1-4 is used to recognize the target luggage image to obtain the final enhanced feature vector corresponding to the target luggage image. The luggage information corresponding to the target luggage image is determined based on the final enhanced feature vector corresponding to the target luggage image and the luggage fingerprint database. The luggage information includes luggage ID, camera tag, time of entering the camera and time of leaving the camera. Based on the luggage information corresponding to the target luggage image, cross-camera luggage trajectory association is performed to achieve cross-camera tracking of the target luggage. The luggage trajectory association adopts the following steps: A source camera node set and a target camera node set are constructed using cameras on the baggage conveyor line. The set of edges connecting the source camera node set and the target camera node set is used as the edge set. The source camera node set contains baggage corresponding to the target baggage image, and the target camera node set is the set of baggage that newly appears in the field of view of the target camera. The optimal matching scheme between the source camera node set and the target camera node set is calculated based on the weights of the edge sets, and the weights of the corresponding edge sets of each optimal matching scheme are compared with the preset matching threshold. If the weight is greater than or equal to the preset matching threshold, the optimal matching scheme is received, and the baggage ID of the source camera node set is assigned to the corresponding matching baggage in the target camera to complete the association of baggage trajectory and realize the tracking of the target baggage.

6. The target baggage identification and tracking method as described in claim 5, characterized in that: The specific formula for calculating the weight of the edge set is as follows: in, The feature similarity is represented by cosine similarity, which is used to calculate the final enhanced feature vector of the two luggage items. The rationale for using a Gaussian function to model time continuity is evident. To represent spatial continuity, a Kalman filter is used for state prediction; , and They are respectively , and The hyperparameters are used to adjust the relative importance of each component, and .

7. A target baggage identification and tracking system, characterized in that: The target baggage identification and tracking system includes: Luggage camera, used to capture images of luggage; An edge computing device for deploying a baggage recognition model obtained by the training method as described in any one of claims 1-4; The server is used to acquire images of target luggage and utilize luggage information corresponding to the images obtained from edge computing devices; and to perform cross-camera luggage trajectory association based on the luggage information corresponding to the images of target luggage, so as to realize cross-camera tracking of target luggage.

Images