A vehicle re-identification method and system for use in complex environments

Abstract

The present application provides a vehicle re-identification method and system for use in complex environments, the method comprising: step one, building and training a multi-branch ReID model; step two, extracting and storing vehicle features; step three, re-identifying across lenses; step four, performing feature matching: after obtaining the ReID features of each vehicle under the second camera, query and compare the ReID features of each vehicle in the feature database, select cosine similarity to represent the distance between features, and select the records whose cosine similarity between vehicle ReID features meets the set threshold condition as the result of successful matching. The present invention can avoid single feature matching by using database technology to store features, can process larger-scale data, and ensure the real-time performance of the algorithm.

Description

Technical Field

[0001] The present application relates to the field of target recognition technology, and in particular to a vehicle re-identification method and system for use in complex environments. Background Art

[0002] Target re-identification (ReID) technology refers to matching the same target captured by different cameras in video surveillance. Traditional re-identification methods mainly rely on the global encoding of the target object as the matching basis. This method requires extracting the feature information of the target object, such as color, shape, texture, etc. to construct a unique identifier for the object. These features are then used to match in different camera images to identify the same target, thereby achieving target re-identification under different lens perspectives. However, traditional re-identification methods mainly focus on the overall characteristics of the target and lack the ability to recognize detailed features.

[0003] The object re-identification method based on machine learning mainly extracts features from images by training models. This type of method requires manually designing feature extraction operators for feature extraction, training and classification for different attribute information in the image, and then using the trained model to extract target features and match them in different camera images to achieve the effect of re-identification.

[0004] Although the above methods can achieve the effect of vehicle re-identification, there are still some problems. For example, the robustness and generalization performance of vehicle re-identification based on traditional methods are poor. The method based on machine learning requires specific manual feature operator design for the target, which is heavily dependent on the experience and professional knowledge of researchers. In addition, this type of re-identification algorithm takes a long time and has high requirements for computing resources, and cannot guarantee the accuracy and real-time performance of the algorithm. The existing technology also has certain research applications, but they all have their own shortcomings. For example, Patent 1: "CN 116682077 A A vehicle re-identification method and device based on vehicle re-identification network" has only one main line network in the network, and the category loss and ternary loss function are used on the original network, and the feature extraction effect is relatively limited. Patent 2: "CN 117392617 A A vehicle re-identification model training method, vehicle re-identification method and device" uses a multi-branch network, but it is divided according to local features and global features. The category loss and ternary loss function are also used on a single network, which is not easy to converge during training.

[0005] Based on the above analysis, the high similarity between different types of vehicles and the differences between the same type of vehicles, the differences in acquisition cameras, the changes in camera perspectives, and the occlusion problem between vehicles make the vehicle re-identification task still face great challenges. Summary of the invention

[0006] The present application provides a vehicle re-identification method and system for use in complex environments, which can be used to solve the technical problem of low vehicle re-identification accuracy.

[0007] The present application provides a vehicle re-identification method applied in a complex environment, the method comprising:

[0008] Step 1: Build and train a multi-branch ReID model;

[0009] Step 2: Extract and store vehicle features:

[0010] The detection result of the first camera is used as the input of the ReID feature extraction network to extract the ReID features corresponding to each vehicle. After the ReID features of the vehicle are obtained, the ReID features of the vehicle are stored in the ReID feature database;

[0011] Step 3: cross-shot re-identification;

[0012] Step 4: perform feature matching: After obtaining the ReID features of each vehicle under the second camera, query and compare the ReID features of each vehicle in the feature database, select cosine similarity to represent the distance between features, and select the records whose cosine similarity between vehicle ReID features meets the set threshold condition as the result of successful matching.

[0013] Furthermore, in step 1, a multi-branch ReID model is constructed and trained, including:

[0014] Step 11, construct a data set for vehicle ReID feature extraction;

[0015] Step 12, establishing a multi-branch ReID model;

[0016] Step 13, constructing a structural enhancement module SEM;

[0017] Step 14, setting the loss function;

[0018] Step 15: Perform training and regularization of the multi-branch ReID model.

[0019] Furthermore, step 11, constructing a data set for vehicle ReID feature extraction; including:

[0020] The vehicle data is added as a dataset; the dataset contains multiple vehicle IDs, each ID contains photos of the vehicle at multiple angles and different positions, and contains different lighting and weather conditions, and any photo includes more than 50% of the vehicle body at the current viewing angle.

[0021] Furthermore, in step 12, a multi-branch ReID model is established; including:

[0022] The multi-branch ReID model contains a total of 4 feature extraction branches after the backbone network performs features;

[0023] The backbone network of the model is composed of the first three stages of ResNet50 and the structure enhancement module SEM through interval splicing;

[0024] The four branches are summarized into two groups, one is the global feature extraction module, and the other is the attention extraction module; the first branch is the global classification feature extraction branch, the second branch is the global metric feature extraction branch, the third branch is the attention classification feature extraction branch, and the fourth branch is the attention metric feature extraction branch; each branch performs a specific type of feature extraction, and the fourth stage of ResNet is used for feature extraction in the global feature extraction module, and the Transformer structure of BoTNet is used for feature extraction in the attention feature extraction module; after global average pooling and batch normalization operations, the extracted features of each branch are finally obtained, as shown below:

[0025] (1);

[0026] in represents the features finally extracted by the Nth branch, , BN is the batch normalization operation, GAP is the global pooling operation, represents the fourth stage feature of the Nth branch, corresponding to It represents the characteristics of the network after the three-stage backbone network. are the parameters of the backbone network, is the parameter of the Nth branch; the 4 features correspond to , , , , after L2 normalization operation, concatenated into the final ReID model extraction feature , which is also the final re-identification feature.

[0027] Further, step 13, building a structural enhancement module SEM; including:

[0028] The SEM module extracts structural information from appearance features by calculating pixel gradients. The three SEM modules are respectively set after the first three stages of ResNet50.

[0029] The SEM module first performs an operation on the intermediate feature map Processing, for each pixel position x and the surrounding size The pixel-level non-negative local gradient is calculated in the area, which is expressed as:

[0030] (2);

[0031] in, represents the characteristics of the SEM module input, , Representation characteristics The number of channels, is the side length of the specified region, Represents a contiguous region;

[0032] Then, the feature weighting operation is used to integrate the local gradient features into the structure descriptor, thereby learning the original geometric structure enhancement feature S:

[0033] (3);

[0034] Finally, the feature obtained in equation (3) is multiplied by the factor And combined with the input features of formula (2) After adding, the obtained features The following is the optimization result of the structure enhancement module:

[0035] (4);

[0036] in is a hyperparameter used to balance the original features And the enhanced features obtained by the corresponding module .

[0037] Furthermore, step 14, setting a loss function, includes:

[0038] Two independent loss functions are used for different branches, namely, the two branches of the global feature extraction module and the attention extraction module, corresponding to the classification loss and metric loss respectively;

[0039] The classification branch is dedicated to extracting features to distinguish the vehicle ID information, using cross entropy loss:

[0040] (5);

[0041] (6);

[0042] in is the model prediction value, is the true value of the category, using smooth labels, is the label smoothing hyperparameter, is the total number of categories;

[0043] The metric branch will be used to measure the extracted vehicle's finer detail feature information; the metric loss uses the batch-hard example ternary loss function:

[0044] (7);

[0045] in, Indicates the number of IDs. Indicates the number of samples belonging to the specified ID. Indicates setting the threshold. and Respectively represent the positive and negative sample sets of a given anchor point in each batch; , , Represent the extracted feature values ​​of anchor images, positive samples, and negative samples respectively;

[0046] Finally, the total loss is:

[0047] (8).

[0048] Furthermore, step 15, training and regularization of the multi-branch ReID model are performed; including:

[0049] Furthermore, step three, cross-shot re-identification, includes:

[0050] For the cross-lens vehicle re-identification process, the second camera B is used as another cross-lens perspective; in the field of view of the second camera, the vehicle real-time detection function is also turned on to achieve real-time detection of all vehicles in the field of view of the second camera; then ReID features are extracted for all vehicles detected by the second camera.

[0051] Furthermore, the cosine similarity in step 4 is as follows:

[0052] (9);

[0053] in and Respectively represent the two feature vectors to be compared; cosine similarity The value range is , This means that the two vectors point in opposite directions. It means that their pointing directions are the same. The larger the cosine similarity, the more similar they are. Therefore, it is expected that the larger the cosine similarity value, the better.

[0054] The present application also provides a vehicle re-identification system for use in a complex environment. The system provided by the present application is used to implement the method provided by the present application. The technical details of the two are the same and will not be repeated here. For specific technical details, please refer to the method provided by the present application. The system includes:

[0055] Model building module, used to build and train multi-branch ReID models;

[0056] Storage module, used for vehicle feature extraction and storage:

[0057] The detection result of the first camera is used as the input of the ReID feature extraction network to extract the ReID features corresponding to each vehicle. After the ReID features of the vehicle are obtained, the ReID features of the vehicle are stored in the ReID feature database;

[0058] Re-identification module, cross-lens re-identification;

[0059] The matching module is used for feature matching: after obtaining the ReID features of each vehicle under the second camera, the ReID features of each vehicle are queried and compared in the feature database, and the cosine similarity is selected to represent the distance between the features. The records whose cosine similarity between the vehicle ReID features meets the set threshold conditions are selected as the results of successful matching.

[0060] The present invention obtains the real-time detection frame of the vehicle through the vehicle real-time detection model YOLOv8 based on deep learning, and then sends the detection frame of the vehicle to the ReID feature extraction network to obtain the ReID feature of the vehicle and the corresponding ReID value, and stores the ReID feature in the feature database. After that, the vehicle target and the ReID feature of the vehicle are detected in real time in the field of view of another camera, and the similarity is measured between the vehicle ReID feature in the cross-lens and the existing feature records in the ReID feature library to obtain the record with the greatest characteristic similarity. When the record with the greatest feature similarity is obtained, the set prior threshold condition must be met at the same time. If not, it is considered that no re-identification has occurred; if satisfied, it is determined that the same vehicle under the cross-lens has been successfully re-identified, and the ReID value of the vehicle in the previous lens is assigned, that is, the vehicle re-identification function under the cross-lens is completed. The present invention provides a complete process from the input of the original video stream to the final successful matching, including the application of database technology, which improves the robustness and generalization of the vehicle re-identification algorithm in complex scenes, and takes the entire implementation chain of re-identification into consideration, further integrates the re-identification algorithm, and improves the model accuracy and real-time performance.

[0061] The present invention uses database technology to store features, thereby avoiding single feature matching, being able to process larger-scale data, and ensuring the real-time performance of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A flow chart of the method provided in the embodiment of the present application;

[0063] Figure 2A diagram of the architecture of a multi-branch ReID model provided in an embodiment of the present application;

[0064] Figure 3 A structural diagram of a structural enhancement module provided in an embodiment of the present application;

[0065] Figure 4 This is a general framework diagram of the vehicle re-identification system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0067] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0068] Step 1: Build and train a multi-branch ReID model;

[0069] Step 11, construct a data set for vehicle ReID feature extraction;

[0070] Use public vehicle datasets, such as VeRi-776, VERI-Wild, DNWild, and any other datasets, and add sorted vehicle data as the dataset; the dataset contains multiple vehicle IDs, each ID contains photos of the vehicle at multiple angles and different positions, and contains different lighting and weather conditions, so as to improve the diversity of the data, where any photo includes more than 50% of the vehicle body at the current perspective, and has sufficient resolution and quality to fully display the vehicle characteristics. Considering privacy protection, personal information such as license plate numbers are blocked when collecting and publishing data.

[0071] Step 12, establishing a multi-branch ReID model;

[0072] The multi-branch ReID model contains a total of 4 feature extraction branches after the backbone network is characterized. The backbone network of the model is composed of the first three stages of ResNet50 and the Structure Enhancement Module (SEM) through interval splicing; the 4 branches are summarized into two groups, one for the global feature extraction module and the other for the attention extraction module; each group contains a category information feature extraction branch and a metric information feature extraction branch. The first branch is the global classification feature extraction branch, the second branch is the global metric feature extraction branch, the third branch is the attention classification feature extraction branch, and the fourth branch is the attention metric feature extraction branch. The structure diagram is as follows Figure 2As shown. Each branch corresponds to a specific task and performs a specific type of feature extraction, thereby achieving the extraction of diversified features, making the overall features more robust and more effective in information representation, and further improving the re-identification capability; the fourth stage of ResNet is used for feature extraction in the global feature extraction module, and the Transformer structure of BoTNet is used for feature extraction in the attention feature extraction module; after global average pooling and batch normalization operations, the extracted features of each branch are finally obtained, as shown below:

[0073] (1);

[0074] in represents the features finally extracted by the Nth branch, , BN is the batch normalization operation (BatchNormalization), GAP is the global average pooling operation (Global Average Pooling), represents the fourth stage feature of the Nth branch, corresponding to It represents the characteristics of the network after the three-stage backbone network. are the parameters of the backbone network, is the parameter of the Nth branch; the 4 features correspond to Figure 2 of , , , , after L2 normalization operation, concatenated into the final ReID model extraction feature , which is also the final re-identification feature.

[0075] Step 13: Build the Structure Enhancement Module (SEM)

[0076] In the multi-branch ReID model architecture, except for the SEM module, the other modules are already common modules and will not be described in detail. Here we will introduce the SEM module proposed in this paper in detail. The SEM module extracts structural information from appearance features by calculating pixel gradients; the three SEM modules are respectively set after the first three stages of ResNet50;

[0077] The SEM module first performs an operation on the intermediate feature map Processing, for each pixel position x and the surrounding size The pixel-level non-negative local gradient is calculated in the area, which is expressed as:

[0078] (2);

[0079] in, represents the characteristics of the SEM module input, , Representation characteristics The number of channels, is the side length of the specified region, Represents a contiguous region;

[0080] Then, the feature weighting operation is used to integrate the local gradient features into the structure descriptor, thereby learning the original geometric structure enhancement feature S:

[0081] (3);

[0082] Finally, the feature obtained in equation (3) is multiplied by the factor And combined with the input features of formula (2) After adding, the obtained features The following is the optimization result of the structure enhancement module:

[0083] (4);

[0084] in is a hyperparameter used to balance the original features And the enhanced features obtained by the corresponding module .

[0085] The above (2), (3), and (4) form the SEM module. The SEM module structure diagram is as follows: Figure 3 shown.

[0086] Step 14, setting the loss function;

[0087] Two independent loss functions are used for different branches, namely the two branches of the global feature extraction module and the attention extraction module, corresponding to the classification loss and metric loss respectively; the classification loss mainly enables the model to extract semantic information, and the metric loss enables the model to acquire the ability to discriminate features.

[0088] The classification branch is dedicated to extracting features to distinguish the vehicle ID information, using cross entropy loss:

[0089] (5);

[0090] (6);

[0091] in is the model prediction value, is the true value of the category, using smooth labels, is the label smoothing hyperparameter, is the total number of categories;

[0092] The metric branch will be used to measure the extracted vehicle's finer detail feature information; the metric loss uses the batch-hard example ternary loss function:

[0093] (7);

[0094] in, Indicates the number of IDs. Indicates the number of samples belonging to the specified ID. Indicates setting the threshold. and Respectively represent the positive and negative sample sets of a given anchor point in each batch; , , Represent the extracted feature values ​​of anchor images, positive samples, and negative samples respectively;

[0095] Finally, the total loss is:

[0096] (8).

[0097] Step 15, training and regularizing the multi-branch ReID model;

[0098] When the multi-branch ReID model is being trained, the features extracted by the global classification features of the first branch are And the third branch of attention classification feature extraction features Fully connected and After the classification loss is calculated, the features extracted by the global metric feature of the second branch And the features extracted by the attention metric feature of the fourth branch It is a feature that has undergone batch normalization (Batch Normalization, BN). Its corresponding feature before BN is used to calculate the metric loss, and its feature before batch normalization is used to calculate the metric loss; refer to Figure 2 shown.

[0099] During the inference process, the features of the four branches are regularized and then concatenated to form the final ReID features.

[0100] At this point, the establishment and training of the multi-branch ReID model has been completed.

[0101] For a vehicle re-identification system used in a complex environment mentioned in this patent, the flow chart is as follows Figure 4 As shown:

[0102] Step 2: Extract and store vehicle features

[0103] The detection result of the first camera is used as the input of the ReID feature extraction network to extract the ReID features corresponding to each vehicle. After the ReID features of the vehicle are obtained, the ReID features of the vehicle are stored in the ReID feature database.

[0104] Step 3: cross-shot re-identification;

[0105] For the cross-lens vehicle re-identification process, the second camera B is used as another cross-lens perspective; in the field of view of the second camera, the vehicle real-time detection function is also turned on to achieve real-time detection of all vehicles in the field of view of the second camera; then ReID features are extracted for all vehicles detected by the second camera.

[0106] Step 4: feature matching: after obtaining the ReID features of each vehicle captured by the second camera, the ReID features of each vehicle are searched and compared in the feature database, and cosine similarity is selected to represent the distance between features. The records whose cosine similarity between the vehicle ReID features meets the set threshold condition are selected as the result of successful matching;

[0107] The cosine similarity function is as follows:

[0108] (9);

[0109] in and Respectively represent the two feature vectors to be compared; cosine similarity The value range is , This means that the two vectors point in opposite directions. It means that their pointing directions are the same. The larger the cosine similarity, the more similar they are. Therefore, it is expected that the larger the cosine similarity value, the better.

[0110] The present application also provides a vehicle re-identification system for use in a complex environment. The system provided by the present application is used to implement the method provided by the present application. The technical details of the two are the same and will not be repeated here. For specific technical details, please refer to the method provided by the present application. The system includes:

[0111] Model building module, used to build and train multi-branch ReID models;

[0112] Storage module, used for vehicle feature extraction and storage:

[0113] The detection result of the first camera is used as the input of the ReID feature extraction network to extract the ReID features corresponding to each vehicle. After the ReID features of the vehicle are obtained, the ReID features of the vehicle are stored in the ReID feature database;

[0114] Re-identification module, cross-lens re-identification;

[0115] The matching module is used for feature matching: after obtaining the ReID features of each vehicle under the second camera, the ReID features of each vehicle are queried and compared in the feature database, and the cosine similarity is selected to represent the distance between the features. The records whose cosine similarity between the vehicle ReID features meets the set threshold conditions are selected as the results of successful matching.

[0116] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A vehicle re-identification method applied in a complex environment, characterized in that: The method comprises: Step 1: Build and train a multi-branch ReID model; Step 2: Extract and store vehicle features: The detection result of the first camera is used as the input of the ReID feature extraction network to extract the ReID features corresponding to each vehicle. After the ReID features of the vehicle are obtained, the ReID features of the vehicle are stored in the ReID feature database; Step 3: cross-shot re-identification; Step 4: perform feature matching: After obtaining the ReID features of each vehicle under the second camera, query and compare the ReID features of each vehicle in the feature database, select cosine similarity to represent the distance between features, and select the records whose cosine similarity between vehicle ReID features meets the set threshold condition as the result of successful matching.

2. The method according to claim 1, characterized in that Step 1: Build and train a multi-branch ReID model; including: Step 11, construct a data set for vehicle ReID feature extraction; Step 12, establishing a multi-branch ReID model; Step 13, constructing a structural enhancement module SEM; Step 14, setting the loss function; Step 15: Perform training and regularization of the multi-branch ReID model.

3. The method according to claim 2, characterized in that Step 11, constructing a data set for vehicle ReID feature extraction; including: The vehicle data is added as a dataset; the dataset contains multiple vehicle IDs, each ID contains photos of the vehicle at multiple angles and different positions, and contains different lighting and weather conditions, and any photo includes more than 50% of the vehicle body at the current viewing angle.

4. The method according to claim 2, characterized in that: Step 12, establishing a multi-branch ReID model; include: The multi-branch ReID model contains a total of 4 feature extraction branches after the backbone network performs features; The backbone network of the model is composed of the first three stages of ResNet50 and the structure enhancement module SEM through interval splicing; The four branches are summarized into two groups, one is the global feature extraction module, and the other is the attention extraction module; the first branch is the global classification feature extraction branch, the second branch is the global metric feature extraction branch, the third branch is the attention classification feature extraction branch, and the fourth branch is the attention metric feature extraction branch; each branch performs a specific type of feature extraction, and the fourth stage of ResNet is used for feature extraction in the global feature extraction module, and the Transformer structure of BoTNet is used for feature extraction in the attention feature extraction module; after global average pooling and batch normalization operations, the extracted features of each branch are finally obtained, as shown below: （1）； in represents the features finally extracted by the Nth branch, , BN is the batch normalization operation, GAP is the global pooling operation, represents the fourth stage feature of the Nth branch, corresponding to It represents the characteristics of the network after the three-stage backbone network. are the parameters of the backbone network, is the parameter of the Nth branch; the 4 features correspond to , , , , after L2 normalization operation, concatenated into the final ReID model extraction feature , which is also the final re-identification feature.

5. The method according to claim 2, characterized in that: Step 13, building a structural enhancement module SEM; including: The SEM module extracts structural information from appearance features by calculating pixel gradients. The three SEM modules are respectively set after the first three stages of ResNet50. The SEM module first performs an operation on the intermediate feature map Processing, for each pixel position x and the surrounding size The pixel-level non-negative local gradient is calculated in the area, which is expressed as: （2）； in, represents the characteristics of the SEM module input, , Representation characteristics The number of channels, is the side length of the specified region, Represents a contiguous region; Then, the feature weighting operation is used to integrate the local gradient features into the structural descriptor, thereby learning the original geometric structure enhancement feature S: （3）； Finally, the feature obtained in equation (3) is multiplied by the factor And combined with the input features of formula (2) After adding, the obtained features The following is the optimization result of the structure enhancement module: (4)； in is a hyperparameter used to balance the original features And the enhanced features obtained by the corresponding module .

6. The method according to claim 2, characterized in that Step 14, setting the loss function; including: Two independent loss functions are used for different branches, namely, the two branches of the global feature extraction module and the attention extraction module, corresponding to the classification loss and metric loss respectively; The classification branch is dedicated to extracting features to distinguish the vehicle ID information, using cross entropy loss: (5)； （6）； in is the model prediction value, is the true value of the category, using smooth labels, is the label smoothing hyperparameter, is the total number of categories; The metric branch will be used to measure the extracted vehicle's finer detail feature information; the metric loss uses the batch-hard example ternary loss function: (7)； in, Indicates the number of IDs. Indicates the number of samples belonging to the specified ID. Indicates setting the threshold. and Respectively represent the positive and negative sample sets of a given anchor point in each batch; , , Represent the extracted feature values ​​of anchor images, positive samples, and negative samples respectively; Finally, the total loss is: （8）。 7. The method according to claim 1, characterized in that Step 3: cross-lens re-identification; including: For the cross-lens vehicle re-identification process, the second camera B is used as another cross-lens perspective; in the field of view of the second camera, the vehicle real-time detection function is also turned on to achieve real-time detection of all vehicles in the field of view of the second camera; then ReID features are extracted for all vehicles detected by the second camera.

8. The method according to claim 1, characterized in that The cosine similarity in step 4 is as follows: （9）； in and Respectively represent the two feature vectors to be compared; cosine similarity The value range is , This means that the two vectors point in opposite directions. It means that their pointing directions are the same. The larger the cosine similarity, the more similar they are. Therefore, it is expected that the larger the cosine similarity value, the better.

9. A vehicle re-identification system for use in complex environments, characterized in that: The system comprises: Model building module, used to build and train multi-branch ReID models; Storage module, used for vehicle feature extraction and storage: The detection result of the first camera is used as the input of the ReID feature extraction network to extract the ReID features corresponding to each vehicle. After the ReID features of the vehicle are obtained, the ReID features of the vehicle are stored in the ReID feature database; Re-identification module, cross-lens re-identification; The matching module is used for feature matching: after obtaining the ReID features of each vehicle under the second camera, the ReID features of each vehicle are queried and compared in the feature database, and the cosine similarity is selected to represent the distance between the features. The records whose cosine similarity between the vehicle ReID features meets the set threshold conditions are selected as the results of successful matching.

Images