A viewpoint-based vehicle re-identification method and system based on multi-task learning

Abstract

This invention discloses a view perception-based vehicle re-identification method and system based on multi-task learning, relating to the field of intelligent transportation technology. The method includes the following steps: receiving a 3D object detection dataset; inputting LiDAR annotations from the 3D object detection dataset into a pre-established view predictor to obtain a trained view predictor; generating view labels for a ReID dataset using the trained view predictor; inputting the view labels from the ReID dataset into a pre-established multi-task feature extractor to output view feature vectors and identity feature vectors; and calculating a comprehensive loss function using the view feature vectors and identity feature vectors to obtain a total loss function. This improves the performance of vehicle re-identification and decouples identity and view features using a network, thereby enhancing the performance of the vehicle re-identification task.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically a method and system for re-identifying vehicles based on multi-task learning perspective perception. Background Technology

[0002] Vehicle re-identification (V-ReID) aims to match vehicle images across different cameras and is an important research topic in intelligent transportation systems. However, due to the rapid changes in the shooting angles of different cameras, the appearance of vehicles can change significantly, posing a huge challenge to re-identification. Summary of the Invention

[0003] To address the shortcomings mentioned in the background section, the present invention aims to provide a viewpoint-based vehicle re-identification method and system based on multi-task learning, which can utilize networks to decouple identity and viewpoint features, thereby improving the performance of vehicle re-identification tasks.

[0004] Firstly, the objective of this invention can be achieved through the following technical solution: a view-perception vehicle re-identification method based on multi-task learning, the method comprising the following steps:

[0005] Receive a 3D object detection dataset, input the LiDAR annotations in the 3D object detection dataset into a pre-built view predictor, and obtain the trained view predictor.

[0006] The trained viewpoint predictor generates viewpoint labels for the ReID dataset. The viewpoint labels of the ReID dataset are then input into a pre-built multi-task feature extractor, which outputs viewpoint feature vectors and corresponding predicted viewpoints, identity feature vectors, and corresponding identity IDs.

[0007] By using the viewpoint feature vector and the identity feature vector to calculate the comprehensive loss function, the total loss function can be obtained, thereby improving the performance of vehicle re-identification.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: cropping the target image based on the 2D bounding box of the 3D object detection dataset, filtering out images smaller than a preset pixel, and then performing super-resolution processing using the ESRGAN method.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the pre-established view predictor uses ResNet-101 as the backbone network, and the original classification layer is replaced by the view prediction layer.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the loss function L of the pre-established view predictor.predictor as follows:

[0011]

[0012] Where N is the number of training samples, It is the true viewpoint value of the i-th sample, α i It is the predicted perspective value.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the pre-established multi-task feature extractor uses ResNet50-IBN-a as the backbone network and combines it with an SE channel attention module to enhance feature representation capabilities.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the comprehensive loss function includes cross-entropy loss for identity classification, cross-entropy loss for viewpoint prediction, triplet loss, center loss, and orthogonal loss.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the calculation formula for the comprehensive loss function L:

[0016] L = L id +λ1L viewpoint +λ2L triplet +λ3L center +λ4L orth ,

[0017] Where λ1, λ2, λ3, and λ4 are hyperparameters controlling the relative importance of each loss term, L id For cross-entropy loss in identity classification, L viewpoint For the cross-entropy loss of view prediction, L triplet For triplet loss, L center For the central loss, L orth This is an orthogonal loss.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the orthogonal loss:

[0019]

[0020] Where r id (x i ) represents the i-th sample x i Identity characteristics representation, r viewpoint (x i ) represents the i-th sample x i The perspective feature is represented by |B|, where |B| is the number of samples in the batch.

[0021] Secondly, in order to achieve the above objectives, this invention discloses a view-perception vehicle re-identification system based on multi-task learning, comprising:

[0022] The model training module receives the 3D object detection dataset, inputs the LiDAR annotations in the 3D object detection dataset into the pre-built view predictor, and obtains the trained view predictor.

[0023] The feature extraction module is used to generate view labels for the ReID dataset using the trained view predictor. The view labels of the ReID dataset are input into the pre-built multi-task feature extractor, and the output is the view feature vector and the corresponding predicted view and identity feature vector and the corresponding identity ID.

[0024] The comprehensive loss module is used to calculate the comprehensive loss function using the view feature vector and the identity feature vector, thereby improving the performance of vehicle re-identification.

[0025] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: cropping the target image from the 3D object detection dataset of the model training module according to the 2D bounding box, filtering out images smaller than a preset pixel, and then using the ESRGAN method for super-resolution processing;

[0026] Alternatively, the pre-built view predictor in the model training module uses ResNet-101 as the backbone network, and the original classification layer is replaced by the view prediction layer.

[0027] Or the loss function L of the pre-built view predictor in the model training module. predictor as follows:

[0028]

[0029] Where N is the number of training samples, It is the true viewpoint value of the i-th sample, α i It is the predicted viewpoint value;

[0030] The pre-built multi-task feature extractor of the feature extraction module uses ResNet50-IBN-a as the backbone network and combines it with the SE channel attention module to enhance the feature representation capability.

[0031] The comprehensive loss function of the comprehensive loss module includes cross-entropy loss for identity classification, cross-entropy loss for view prediction, triplet loss, center loss, and orthogonality loss.

[0032] The formula for calculating the comprehensive loss function L in the comprehensive loss module is as follows:

[0033] L = L id +λ1Lviewpoint +λ2L triplet +λ3L center +λ4L orth ,

[0034] Where λ1, λ2, λ3, and λ4 are hyperparameters controlling the relative importance of each loss term, L id For cross-entropy loss in identity classification, L viewpoint For the cross-entropy loss of view prediction, L triplet For triplet loss, L center For the central loss, L orth Orthogonal loss;

[0035] Orthogonal loss of the integrated loss module:

[0036]

[0037] Where r id (x i ) represents the i-th sample x i Identity characteristics representation, r viewpoint (x i ) represents the i-th sample x i The perspective feature is represented by |B|, where |B| is the number of samples in the batch.

[0038] The beneficial effects of this invention are:

[0039] This invention utilizes LiDAR annotations from a 3D object detection dataset to train a viewpoint predictor, automatically generating high-quality viewpoint labels to provide rich supervision signals for training. Multi-task learning decouples viewpoint features from identity features, ensuring that the learned features have robust viewpoint invariance. The loss function ensures that identity features and viewpoint features do not interfere with each other. The network can decouple identity and viewpoint features, thereby improving the performance of vehicle re-identification tasks. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0042] Figure 2 This is a schematic diagram of the overall framework of the multi-task learning perspective modeling VMM of this invention;

[0043] Figure 3This is a schematic diagram of the angle between the vehicle's driving direction and the camera's optical axis.

[0044] Figure 4 This is a schematic diagram of the V-ReID network structure for perspective perception in this embodiment;

[0045] Figure 5 This is a schematic diagram of the system structure of the present invention;

[0046] Figure 6 This is a diagram showing the experimental results of an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Example 1:

[0049] The following is a description of the relevant terms used in the embodiments of this application:

[0050] Feature extraction: In machine learning, pattern recognition, and image processing, feature extraction begins with an initial set of measurements and builds derived values ​​(features) designed to provide information and avoid redundancy, thereby facilitating subsequent learning and generalization steps and, in some cases, leading to better interpretability. Feature extraction is related to dimensionality reduction. The quality of the features has a crucial impact on generalization ability.

[0051] Loss function: A loss function, or cost function, is a function that maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. In applications, the loss function is often used as a learning criterion in relation to optimization problems; that is, the model is solved and evaluated by minimizing the loss function.

[0052] like Figure 1 As shown, a view-aware vehicle re-identification method based on multi-task learning includes the following steps:

[0053] S101: Receive the 3D object detection dataset, input the LiDAR annotations in the 3D object detection dataset into the pre-built view predictor, and obtain the trained view predictor.

[0054] The 3D object detection dataset cropped the object images based on the 2D bounding boxes and filtered out images smaller than 50x50 pixels. Then, the ESRGAN method was used for super-resolution processing to make the resolution closer to that of the ReID dataset.

[0055] The pre-built viewpoint predictor is a key innovation of VMM, which automatically acquires viewpoint supervision signals using LiDAR annotations from 3D object detection datasets. This approach effectively addresses the challenge of estimating vehicle viewpoints from a single image, which is crucial for learning viewpoint-invariant features in V-ReID.

[0056] In the KITTI dataset, the observation angle α (α∈[0,2π)) is defined as the angle between the vehicle's direction of travel and the camera's optical axis. Figure 3 The left sub-image shows a top view of the car and object's coordinate system, while the right sub-image shows a perspective view of the object's coordinate system. The viewing angle α is calculated using the formula α = -[π + r]. y +π+β]. This angle reflects the angle at which the object is detected from the camera's perspective, making it an accurate estimate of the object's orientation.

[0057] Utilizing the KITTI dataset, which contains a large number of traffic scene frames and corresponding LiDAR point clouds, we used 2D bounding boxes and the viewing angles of the targets. Target images were cropped based on the 2D bounding boxes, and the viewing angles from the 3D annotations were used as viewpoint prediction labels. This created a dataset of target image-viewpoint pairs.

[0058] The view prediction layer uses ResNet-101 as its backbone, replacing the original classification layer. Traditional methods typically predict the sine and cosine values ​​of the angle to handle its periodicity. However, these methods can suffer from discontinuities at boundaries (e.g., transitioning from 359° to 0°), leading to inaccurate predictions and increased training complexity. To address these issues, we directly predict the angle itself and use a novel loss function to supervise training:

[0059]

[0060] Where N is the number of training samples. It is the true viewpoint value of the i-th sample, α i It is the predicted perspective value.

[0061] This loss function simultaneously constrains the accuracy of the predicted sine and cosine values, avoiding the discontinuity problem caused by the periodicity of the angle. It allows the network to directly output the angle in radians without additional post-processing, making the prediction more direct and accurate. By directly optimizing the angle prediction, our method simplifies the training process and improves the robustness of the viewpoint estimation.

[0062] By leveraging these LiDAR annotations, the view predictor can automatically generate view labels for a large number of unlabeled images, providing rich view supervision information for V-ReID training. This innovative approach significantly enhances the frame's ability to model view changes and learn view-invariant features.

[0063] S102: Use the trained viewpoint predictor to generate viewpoint labels for the ReID dataset. Input the viewpoint labels of the ReID dataset into the pre-established multi-task feature extractor and output viewpoint feature vectors and corresponding predicted viewpoints and identity feature vectors and corresponding identity IDs.

[0064] The multi-task feature extractor (view-aware V-ReID feature extractor) learns robust view-invariant features by utilizing explicit view information provided by the view predictor through multi-task learning and advanced hard sample mining.

[0065] The proposed view-aware V-ReID network structure is as follows: Figure 4 As shown. It adopts the ResNet50-IBN-a

[22] backbone network and uses the SE

[10] channel attention module to enhance feature representation. The network connects the view prediction and identity recognition branches, which share the same backbone network to achieve parameter sharing, information interaction and collaborative learning. This multi-task learning method ensures that the identity recognition features are both view invariant and discriminative.

[0066] S103: The overall loss function is calculated by using the view feature vector and the identity feature vector to improve the performance of vehicle re-identification.

[0067] To effectively learn discriminative and robust vehicle re-identification features, we designed a comprehensive loss function that combines multiple loss components. The total loss function is a weighted sum of the cross-entropy loss for identity classification, the cross-entropy loss for viewpoint prediction, the triplet loss, the center loss, and the orthogonality loss:

[0068] L = L id +λ1L viewpoint +λ2L triplet +λ3L center +λ4L orth (2)

[0069] Where λ1, λ2, λ3, and λ4 are hyperparameters that control the relative importance of each loss term. This design ensures that the features learned by the network are both discriminative of vehicle identity and invariant to changes in viewpoint.

[0070] Next, we will introduce the different loss components in Formula 2. a)L id: Cross-entropy loss for identity classification. It ensures the network correctly classifies vehicle identities. This is crucial for the primary task of vehicle re-identification because it directly optimizes the network to distinguish between different vehicles; b)L viewpoint Cross-entropy loss for viewpoint prediction. To minimize the error in viewpoint labeling, we frame the viewpoint prediction problem as a 12-class classification problem, dividing the 360-degree range into 30-degree intervals. This discretization reduces noise and inaccuracy, lowers the learning difficulty, and facilitates the mining of hard samples by simplifying the definitions of similar and different angles; c)L triplet Triplet loss is used to minimize intra-class distance while maximizing inter-class distance. This loss function encourages the network to learn more discriminative features by ensuring that images of the same vehicle are closer in the feature space, while images of different vehicles are farther apart; d)L center Center loss enhances intra-class compactness by pulling features toward their class centers. This loss function complements the triplet loss by further reducing intra-class variance, resulting in more robust feature representations.

[0071] Specifically, the present invention will be further illustrated below through embodiments:

[0072] To further facilitate the decoupling of identity recognition features and viewpoint prediction features, we introduce an orthogonal loss:

[0073]

[0074] Where r id (x i ) represents the i-th sample x i Identity characteristics representation, r viewpoint (x i ) represents the i-th sample x i The viewpoint feature is represented by |B|, where |B| is the number of samples in the batch. By minimizing L... orth We ensure that identity and perspective features are orthogonal to prevent them from interfering with each other and to enhance the overall performance of the network.

[0075] A comprehensive loss function design combines cross-entropy loss for identity classification, cross-entropy loss for viewpoint prediction, triplet loss, center loss, and orthogonality loss to ensure that the features learned by the network are both discriminative of vehicle identity and invariant to viewpoint changes. By leveraging LiDAR annotations and multi-task learning with orthogonal constraints, the VMM framework effectively addresses the viewpoint change challenge in V-ReID. This innovative approach allows the network to decouple identity and viewpoint features, thereby improving the performance of vehicle re-identification tasks.

[0076] like Figure 6As shown, this is a sample ReID retrieval result on the VeRI dataset using VMM. The leftmost image represents the query image, and the rightmost images represent the top ten matches sorted by confidence. Images within the red boxes indicate no matches.

[0077] Example 2: Second aspect, such as Figure 5 As shown, in order to achieve the above objectives, this invention discloses a view-perception vehicle re-identification system based on multi-task learning, comprising:

[0078] The model training module 11 is used to receive the 3D object detection dataset, input the LiDAR annotations in the 3D object detection dataset into the pre-established view predictor, and obtain the trained view predictor.

[0079] The feature extraction module 12 is used to generate view labels for the ReID dataset using the trained view predictor. The view labels of the ReID dataset are input into the pre-established multi-task feature extractor, and the output is the view feature vector and the corresponding predicted view and identity feature vector and the corresponding identity ID.

[0080] The comprehensive loss module 13 is used to calculate the comprehensive loss function using the view feature vector and the identity feature vector to obtain the total loss function, thereby improving the performance of vehicle re-identification.

[0081] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: cropping the target image from the 3D object detection dataset of the model training module 11 according to the 2D bounding box, filtering out images smaller than a preset pixel, and then using the ESRGAN method for super-resolution processing;

[0082] Alternatively, the pre-built view predictor in model training module 11 uses ResNet-101 as the backbone network, and the original classification layer is replaced by the view prediction layer.

[0083] Alternatively, the loss function L of the pre-established view predictor in model training module 11. predictor as follows:

[0084]

[0085] Where N is the number of training samples, It is the true viewpoint value of the i-th sample, α i It is the predicted viewpoint value;

[0086] The pre-built multi-task feature extractor of the feature extraction module 12 uses ResNet50-IBN-a as the backbone network and combines it with the SE channel attention module to enhance the feature representation capability.

[0087] The comprehensive loss function of the comprehensive loss module 13 includes cross-entropy loss for identity classification, cross-entropy loss for view prediction, triplet loss, center loss, and orthogonality loss.

[0088] The formula for calculating the comprehensive loss function L in the comprehensive loss module 13 is as follows:

[0089] L = L id +λ1L viewpoint +λ2L triplet +λ3L center +λ4L orth ,

[0090] Where λ1, λ2, λ3, and λ4 are hyperparameters controlling the relative importance of each loss term, L id For cross-entropy loss in identity classification, L viewpoint For the cross-entropy loss of view prediction, L triplet For triplet loss, L center For the central loss, L orth Orthogonal loss;

[0091] Orthogonal loss of the integrated loss module:

[0092]

[0093] Where r id (x i ) represents the i-th sample x i Identity characteristics representation, r viewpoint (x i ) represents the i-th sample x i The perspective feature is represented by |B|, where |B| is the number of samples in the batch.

[0094] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0095] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0096] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0097] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A view-perception vehicle re-identification method based on multi-task learning, characterized in that, The method includes the following steps: Receive a 3D object detection dataset, input the LiDAR annotations in the 3D object detection dataset into a pre-built view predictor, and obtain the trained view predictor. The pre-established view predictor uses ResNet-101 as the backbone network, and the original classification layer is replaced by the view prediction layer. The loss function of the pre-established viewpoint predictor as follows: in, It is the number of training samples. It is the first The true perspective value of each sample It is the predicted viewpoint value; The trained viewpoint predictor generates viewpoint labels for the ReID dataset. The viewpoint labels of the ReID dataset are then input into a pre-built multi-task feature extractor, which outputs viewpoint feature vectors and corresponding predicted viewpoints, identity feature vectors, and corresponding identity IDs. By using the viewpoint feature vector and the identity feature vector to calculate the comprehensive loss function, the total loss function can be obtained, thereby improving the performance of vehicle re-identification. The comprehensive loss function includes cross-entropy loss for identity classification, cross-entropy loss for viewpoint prediction, triplet loss, center loss, and orthogonality loss. The formula for calculating the comprehensive loss function L is as follows: in and It is a hyperparameter that controls the relative importance of each loss term. For identity classification, cross-entropy loss, The cross-entropy loss is the prediction of the viewpoint. For triple loss, Loss at the center This is an orthogonal loss.

2. The view-perception vehicle re-identification method based on multi-task learning according to claim 1, characterized in that, The 3D object detection dataset cropped the object images based on 2D bounding boxes, filtered out images smaller than a preset number of pixels, and then used the ESRGAN method for super-resolution processing.

3. The view-perception vehicle re-identification method based on multi-task learning according to claim 1, characterized in that, The pre-established multi-task feature extractor uses ResNet50-IBN-a as the backbone network and combines it with the SE channel attention module to enhance feature representation capabilities.

4. The view-perception vehicle re-identification method based on multi-task learning according to claim 1, characterized in that, The orthogonal loss: in Indicates the first Sample Identity characteristics representation, Indicates the first Sample The perspective features are represented. This is the number of samples in the batch.

5. A view-perception vehicle re-identification system based on multi-task learning, employing the adaptive transmission method for a multi-carrier system based on sliding code rate as described in claim 1, characterized in that... include: The model training module receives the 3D object detection dataset, inputs the LiDAR annotations in the 3D object detection dataset into the pre-built view predictor, and obtains the trained view predictor. The feature extraction module is used to generate view labels for the ReID dataset using the trained view predictor. The view labels of the ReID dataset are input into the pre-built multi-task feature extractor, and the output is the view feature vector and the corresponding predicted view and identity feature vector and the corresponding identity ID. The comprehensive loss module is used to calculate the comprehensive loss function using the view feature vector and the identity feature vector, thereby improving the performance of vehicle re-identification.

6. The view-perception vehicle re-identification system based on multi-task learning according to claim 5, characterized in that, The 3D object detection dataset of the model training module is cropped according to the 2D bounding box, and images smaller than a preset pixel are filtered out. Then, the ESRGAN method is used for super-resolution processing. The pre-built view predictor in the model training module uses ResNet-101 as the backbone network, and the original classification layer is replaced by the view prediction layer. The loss function of the pre-built view predictor in the model training module as follows: in, It is the number of training samples. It is the first The true perspective value of each sample It is the predicted viewpoint value; The pre-built multi-task feature extractor of the feature extraction module uses ResNet50-IBN-a as the backbone network and combines it with the SE channel attention module to enhance the feature representation capability. The comprehensive loss function of the comprehensive loss module includes cross-entropy loss for identity classification, cross-entropy loss for view prediction, triplet loss, center loss, and orthogonality loss. The formula for calculating the comprehensive loss function L in the comprehensive loss module is as follows: in and It is a hyperparameter that controls the relative importance of each loss term. For identity classification, cross-entropy loss, The cross-entropy loss is the prediction of the viewpoint. For triple loss, Loss at the center Orthogonal loss; Orthogonal loss of the integrated loss module: in Indicates the first Sample Identity characteristics representation, Indicates the first Sample The perspective features are represented. This is the number of samples in the batch.

Images