An air-to-ground video vehicle re-identification method and device, electronic equipment and storage medium

Abstract

The application provides a kind of ground-to-air video vehicle re-identification method, device, electronic equipment and storage medium.The method comprises: using double-branch near neighbor feature enhancement network, the identity feature and perspective feature of vehicle are extracted from input image;The network includes two parallel branches of adjacent backbone network, one branch is used to extract vehicle identity attribute, and the other branch is used to capture the difference of perspective change;Identity feature and enhanced feature through near neighbor feature enhancement module PFEM are weighted and summed, and enhanced feature is accessed gradient inversion layer and perspective loss, decouples the identity information and perspective information of vehicle.The method of the application processes the intra-class feature confusion problem caused by perspective change by feature enhancement and identity and perspective feature decoupling, uses double-branch near neighbor feature enhancement network, respectively extracts identity and perspective feature, improves feature representation ability, and can better process the problem of inconsistent visual feature distribution caused by ground-to-air cross perspective.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a ground-air video vehicle re-identification method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Vehicle Re-Identification (VeRi for short) refers to finding other cameras in a gallery set for the same vehicle given a vehicle image, so vehicle re-identification can be regarded as a subtask of image retrieval. Vehicle re-identification technology can accurately track the identity of a vehicle through a multi-camera network, thereby helping to achieve tasks such as automated traffic monitoring. Therefore, vehicle re-identification technology has broad application prospects in the field of intelligent transportation.

[0003] The vehicle re-identification method and dataset in the related art mostly focus only on ground or air perspectives, and are difficult to handle the problem of inconsistent visual feature distribution caused by ground-air cross perspectives.

[0004] The above background technical information does not necessarily constitute the prior art. SUMMARY

[0005] The present application aims to provide a ground-air video vehicle re-identification method and device, electronic equipment and a storage medium.

[0006] The technical solution of the present application is as follows:

[0007] In a first aspect, the present application provides a ground-air video vehicle re-identification method, comprising:

[0008] A dual-branch near neighbor feature enhancement network is used to extract the identity features and perspective features of a vehicle from an input image; the network includes two parallel branches of a backbone network, one branch is used to extract the identity attributes of the vehicle, and the other branch is used to capture the differences in perspective changes;

[0009] The identity features and the enhanced features obtained through a near neighbor feature enhancement module PFEM are weighted and summed, and the enhanced features are input into a gradient reversal layer and a perspective loss, so as to decouple the identity information and perspective information of the vehicle.

[0010] In some embodiments, the dual-branch near neighbor feature enhancement network is used to extract the identity features and perspective features of a vehicle from an input image, comprising:

[0011] The identity feature extraction branch and the perspective feature extraction branch share the same backbone layer of a convolutional neural network, which is used to generate basic features;

[0012] The input image X is input into a backbone network Backbone iFirstly, the base feature F is converted into the identity feature F

[0013] The identity feature branch maps the base feature into the identity feature space through a convolution or fully connected layer, generating a feature F id that is strongly related to the identity of the vehicle.

[0014] The view feature branch extracts a feature F view .

[0015] In some embodiments, the identity feature is weighted and summed with the enhanced feature F

[0016] The proximity feature enhancement module PFEM uses similar feature information within the same batch of processed data to fuse the identity and view features. This module is based on a K-NN search process that retrieves the K nearest neighbors from the feature database to the current sample, and uses these neighbor features for feature enhancement, where K is a positive integer.

[0017] In some embodiments, for each input ID feature, K nearest neighbor features are retrieved from the feature database . This process is represented as:

[0018]

[0019] The K-NN search finds the K nearest neighbor features from the identity feature database to the current sample feature using a metric such as Euclidean distance or cosine similarity.

[0020] In some embodiments, for the view feature K nearest neighbor features are retrieved from the database

[0021] After retrieving the nearest neighbor features for identity and view, the PFEM first performs an intersection operation to extract the common part of the identity feature neighbors and the view feature neighbors:

[0022]

[0023] The intersection operation extracts the common features between the identity and view features, which represent the feature part that has both identity discriminability and view information.

[0024] In some embodiments, these common features are fused with the original identity feature through a feature aggregation operation to generate an enhanced feature representation:

[0025]

[0026] wherein the aggregation operation The weighted average is adopted.

[0027] In some embodiments, the identity recognition task is optimized by cross-entropy loss and triplet loss, and the view prediction task is optimized by cross-entropy loss; the cross-entropy loss for the identity recognition task, the triplet loss, and the cross-entropy loss for the view prediction task are represented as follows:

[0028]

[0029] wherein, and represent the identity probability distribution and the view probability distribution predicted by the model respectively, and represent the real identity label and the view label respectively, is the positive sample feature, is the negative sample feature, d(·, ·) is a distance function between features, and m is a hyperparameter in the triplet loss, used to control the minimum distance difference between the positive sample and the negative sample, and the total optimization objective of the model is:

[0030]

[0031] wherein λ is a hyperparameter for balancing the weight of the gradient reversal layer, used to control the strength of the gradient reversal.

[0032] In a second aspect, the present application provides a ground-to-air video vehicle re-identification device, comprising:

[0033] A dual-branch nearest neighbor feature enhancement network is adopted to extract the identity feature and the view feature of the vehicle from the input image; the network comprises two parallel branches adjacent to the backbone network, one branch is used to extract the identity attribute of the vehicle, and the other branch is used to capture the difference of the view change;

[0034] An identity feature and an enhanced feature passing through a nearest neighbor feature enhancement module PFEM are weighted and summed to access a gradient reversal layer and a view loss, thereby decoupling the identity information and the view information of the vehicle.

[0035] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; the processor implements the method of any one of the embodiments of the present application when executing the running program stored in the memory.

[0036] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method of any of the embodiments of the present application.

[0037] The ground-air video vehicle re-identification method provided by the present application processes the intra-class feature confusion problem caused by the change of the view angle through feature enhancement and identity and view angle feature decoupling, adopts a double-branch nearest neighbor feature enhancement network to extract identity and view angle features respectively, improves the feature representation capability, introduces a nearest neighbor feature enhancement module PFEM to fuse similar neighbor features, captures common features under different view angles to enhance the original features, realizes the decoupling of the view-invariant features, and can better process the problem of inconsistent visual feature distribution caused by ground-air cross-view angles. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A view of a ground-air camera in the related art is shown.

[0039] Figure 2 A flowchart of a ground-air video vehicle re-identification method according to an embodiment of the present application is shown.

[0040] Figure 3 A diagram of a double-branch nearest neighbor feature enhancement network overall model according to an embodiment of the present application is shown.

[0041] Figure 4 A visualization diagram of output feature maps of three methods of Baseline, AdaSp and DPFE is shown.

[0042] Figure 5 A structural block diagram of an electronic device according to an embodiment of the present application is shown.

[0043] Figure 6 A schematic diagram of a computer readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] In order to more clearly illustrate the features and technical contents of the embodiments of the present application, the technical solutions of the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. The accompanying drawings are only used for reference and do not limit the scope of the embodiments of the present application.

[0045] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have meanings consistent with the meanings commonly understood by one of ordinary skill in the art. The use of terms is only for the purpose of describing the embodiments of the present application and is not intended to limit the present application in any way.

[0046] In the following description, "some embodiments" represents a subset of all possible embodiments, but it should be understood that "some embodiments" can be the same or different subsets of all possible embodiments, and can be combined with each other as long as there is no conflict. It should be noted that if the terms "first", "second", "third" and the like are used in the embodiments of the present application, these terms are only used to distinguish different similar objects, and do not represent a specific order or sequence. Therefore, the terms "first", "second", "third" and the like can be used interchangeably as long as it is allowed, and the embodiments of the present application do not have to be strictly performed in the order described in the description.

[0047] In recent years, with the rapid development of deep learning, traditional vehicle re-identification technology has made remarkable achievements. However, with the increasing intelligence of cities, real-world monitoring systems can appear in every corner, which is far from traditional vehicle re-identification. The reasons are as follows: (1) Almost all vehicle re-identification datasets in related technologies are separately from ground monitoring cameras or aerial monitoring cameras, such as VeRi-776, VehicleID, VERI-Wild, CompCars, etc. for ground view only or VRAI, UAV-ReID, etc. for aerial view only. Such camera systems emphasize that the camera remains at the same height to capture the same scene or object from different angles. In the real world, with the vigorous development of unmanned aerial vehicles in recent years, aerial monitoring systems have become active in the public eye, forming a linkage with traditional monitoring systems. This ground-air collaborative monitoring system combines the wide-area dynamic monitoring capability of aerial unmanned vehicles and the fixed-point high-precision monitoring capability of ground traditional cameras, which can cover a wider monitoring range and effectively improve the overall monitoring accuracy and real-time performance. Therefore, in the face of multi-view, multi-height monitoring scenes, ground-air cross-view vehicle re-identification poses some new challenges in matching. (2) The vehicle re-identification methods in related technologies almost all use images from the same type of camera for training. When querying and matching, the matching is also different angle vehicle pictures taken by the same type of camera, such as ground-ground view matching and air-air view matching, and few studies on ground-air bidirectional view vehicle matching. Figure 1 Vehicle pictures from different ground cameras and aerial cameras are shown. Aerial camera is an aerial camera, and ground camera is a ground camera. Due to the significant view difference between the two views, the discriminative identity features are destroyed, plus different lighting conditions and environmental changes at different time periods, etc. factors, making the ground-air vehicle re-identification technology a more challenging task.

[0048] Ground surveillance view vehicle re-identification is one of the key directions in current vehicle re-identification research. Such tasks usually involve vehicle images captured by fixed ground surveillance cameras at different positions and angles. The methods in the related art are mostly based on ground surveillance view datasets and mainly focus on the cross-angle changes of vehicles under ground surveillance cameras, which are difficult to apply to cross-ground-air view matching scenarios.

[0049] The vehicle re-identification task of ground-air collaborative surveillance view is difficult for traditional vehicle re-identification methods to apply due to the large view angle change and the significant difference in the appearance of vehicles at different views. Ground views usually provide horizontal angle details, while air views are mainly top-down, resulting in significant differences in visual features of the same vehicle at ground and air views. The vehicle re-identification methods and datasets in the related art mostly focus only on ground views or air views, making it difficult to handle the problem of inconsistent visual feature distribution caused by cross-ground-air view. In terms of vehicle re-identification, the cross-view re-identification task of ground-air collaboration is almost zero.

[0050] To fill the technical gap in the field of cross-view re-identification task of ground-air collaboration, in the embodiments of the present application, the first ground-air cross-view vehicle re-identification dataset GAVRI is constructed, which provides a new identification technology research and development aid for vehicles under different height monitoring views. It provides a challenging test platform for the cross-view re-identification task of ground-air collaboration, and provides a reliable experimental basis for future vehicle re-identification research, thereby promoting further development in this field.

[0051] Feature decoupling has a wide range of applications in the fields of deep learning and computer vision, and its core purpose is to separate different features in the data, thereby improving the performance of the model in multi-task learning. In domain adaptation tasks, feature decoupling is used to address the problem of inconsistent data distribution between source and target domains, and by separating domain features and task features, the generalization ability of the model is enhanced. This method performs significantly in multi-task learning, and can effectively prevent information confusion between tasks and improve the feature learning effect of each task.

[0052] Vehicle re-identification (Re-ID) is crucial for urban surveillance and intelligent transportation. However, most studies focus on matching from ground views, while matching from ground-to-aerial views is rarely studied, mainly due to two reasons: on one hand, there is a lack of available ground-to-aerial view datasets to support the research; on the other hand, the significant difference between ground and aerial views makes it difficult to effectively apply traditional matching methods. To address this challenge, a network based on dual-branch proximal feature enhancement (DPFE) is proposed, which uses feature enhancement and identity and view feature decoupling to deal with the intra-class feature confusion problem caused by view changes. DPFE uses a dual-branch architecture to extract identity and view features, improving feature representation. In addition, a proximal feature enhancement module (PFEM) is introduced to fuse similar proximal features using K-nearest neighbor algorithm to capture common features under different views to enhance the original features. With the help of gradient reversal layer (GRL), view-invariant feature decoupling is achieved. To verify the effectiveness of DPFE, a new dataset GAVRI is provided, which contains 2159 vehicle IDs and 17437 images, covering different aerial and ground camera images. Experimental results show that DPFE achieves significant performance improvement on GAVRI, verifying its robustness and superiority in complex scenarios.

[0053] To be more realistic, especially for vehicle re-identification in ground-air camera network, a dual-branch proximal feature enhancement network DPFE is proposed in the embodiments of the present application to solve the view difference caused by extreme view changes. DPFE aims to separate the view-dependent features from the view-independent features so as to better learn the identity features with discriminability from those view-independent features. The workflow is roughly as follows: the input image is subjected to a backbone network to extract the identity features and the view features of the vehicle. The identity features are used for vehicle identity recognition, and the view features capture the view information of the vehicle and the appearance changes under different angles. In the proximal feature enhancement module PFEM, the model uses two kinds of features to perform K-nearest neighbor search in the feature database. The nearest neighbor features queried by the identity features are strongly related to the current vehicle identity, and the view features can also retrieve the vehicle images of the same identity due to their potential identity discriminative information. The intersection features of the two query results represent the features that play a key role in identity and view prediction. To further decouple the identity features and the view features, a gradient reversal layer GRL is introduced into the model to eliminate the interference of the view information in the intersection features on the identity classification, so that the model can learn more discriminative identity features.

[0054] To verify the effectiveness of the proposed dual-branch proximal feature enhancement network, a new ground-air vehicle re-identification dataset GAVRI is constructed. The dataset covers vehicle images under multiple views, involving 2159 vehicle identities and 17437 images, including vehicle images taken by different ground and air monitoring cameras under different lighting and weather conditions. This diversity helps to improve the robustness of the model in different real-world scenarios. In addition, in the evaluation setting, the GAVRI dataset does not use a single matching mode (such as ground-air or air-ground), but uses a bidirectional mixed matching, that is, the query set and the gallery set both contain vehicle images of ground and air views. This design provides a richer and more challenging test environment, which is more realistic for the multi-view cross-environment vehicle re-identification problem in the real world. The method of the embodiments of the present application proposes a dual-branch feature extraction and fusion network, which decouples the identity and view features through the PFEM module and the GRL, and thus learns more discriminative identity features, improving the recognition ability of the model.

[0055] A vehicle re-identification dataset is adopted in the embodiments of the present application, which contains 2159 vehicle identities and 17437 vehicle images across different view angles, covering different environmental conditions and shooting angles to help solve the challenges in the vehicle re-identification task.

[0056] Systematic experimental evaluation is carried out on the GAVRI and AG-ReID two ground-air datasets. By comparing with other state-of-the-art methods, the method proposed in the embodiments of the present application has obvious performance improvement.

[0057] In the method of the embodiments of the present application, the feature decoupling in the DFPE is performed with the aid of the gradient reversal layer to further separate the identity and view angle features. By introducing the adversarial learning mechanism, the identity feature learning and the view angle feature classification are mutually antagonistic, and finally the purpose of suppressing the view angle information interference is achieved, thereby promoting the model to learn more discriminative identity features. This strategy is particularly important in the vehicle re-identification task across ground and air view angles, and helps the model to extract more robust identity features under extreme view angle changes.

[0058] In the embodiments of the present application, a model architecture for solving ground and air view angle vehicle re-identification is proposed, which aims to improve the re-identification performance by combining the decoupling of identity features and view angle features and using the near neighbor feature enhancement and post-fusion mode. The proposed model will be introduced from the following aspects, mainly including: the overall architecture of the model, the dual-branch feature extraction and decoupling, the near neighbor feature enhancement module PFEM and the loss function.

[0059] Reference Figure 2 As shown in the drawings, an embodiment of the present application provides a ground-air video vehicle re-identification method, which can include:

[0060] S10, a dual-branch near neighbor feature enhancement network is adopted to extract the identity features and view angle features of the vehicle from the input image; the network includes two parallel branches adjacent to the backbone network, one branch is used to extract the vehicle identity attribute, and the other branch is used to capture the difference of the view angle change;

[0061] S20, the identity features and the enhanced features passing through the near neighbor feature enhancement module PFEM are subjected to weighted sum operation, the enhanced features are accessed to the gradient reversal layer GRL and the view angle loss, and the identity information and the view angle information of the vehicle are decoupled.

[0062] Exemplarily, the adoption of the dual-branch near neighbor feature enhancement network to extract the identity features and the view angle features of the vehicle from the input image can include:

[0063] The identity feature extraction branch and the view angle feature extraction branch share the same backbone layer of the convolutional neural network, which is used to generate the basic features;

[0064] Input image X via the backbone network i First, it is converted into basic feature F;

[0065] The identity feature branch maps basic features into the identity feature space through convolutional or fully connected layers, generating features F that are strongly correlated with vehicle identity. id This feature is used to distinguish different vehicle identities;

[0066] The view feature branch extracts features F related to the vehicle's view. view .

[0067] For example, the step of performing a weighted summation operation on the identity features and the enhanced features processed by PFEM, and then incorporating the enhanced features into GRL and view loss to decouple the vehicle's identity information from its view information, includes:

[0068] The nearest neighbor feature enhancement module PFEM uses similar feature information within the same batch of processed data to fuse identity and viewpoint features. This module is based on a K-NN search process, which retrieves the K nearest neighbors to the current sample from the feature library and uses the features of these neighbors for feature enhancement, where K is a positive integer.

[0069] For example, for each input ID feature, in the feature database Retrieve K nearest neighbor features This process can be represented as:

[0070]

[0071] K-NN search uses metrics such as Euclidean distance or cosine similarity to find the K nearest neighbors to the current sample from an identity feature database.

[0072] For example, for viewpoint features Retrieve the K nearest neighbor features from the database.

[0073] After retrieving the nearest neighbor features for identity and perspective, PFEM first performs an intersection operation. Extract the common components between neighbors with identity features and neighbors with viewpoint features:

[0074]

[0075] The intersection operation extracts common features between identity and perspective features, which represent feature parts that have both identity distinguishability and perspective information.

[0076] For example, through feature aggregation operation These public characteristics are compared with the original identity characteristics fusion is performed to generate an enhanced feature representation:

[0077]

[0078] wherein the aggregation operation is a weighted average.

[0079] Exemplarily, the identity recognition task is jointly optimized by a cross-entropy loss and a triplet loss, and the view prediction task is optimized by a cross-entropy loss; the cross-entropy loss for the identity recognition task, the triplet loss, and the cross-entropy loss for the view prediction task are represented as:

[0080]

[0081] wherein, and represent the identity probability distribution and the view probability distribution predicted by the model respectively, and represent the real identity label and the view label respectively, is a positive sample feature, is a negative sample feature, d(·, ·) is a distance function between features, and m is a hyperparameter in the triplet loss, used to control the minimum distance difference between the positive sample and the negative sample, and the total optimization objective of the model is:

[0082]

[0083] wherein λ is a hyperparameter for balancing the weight of the gradient reversal layer, used to control the strength of the gradient reversal.

[0084] The ground-air video vehicle re-identification method proposed in the embodiments of the present application is a ground-air video vehicle re-identification method based on double-branch near neighbor feature enhancement.

[0085] The embodiments of the present application propose an image double-branch near neighbor feature enhancement network based on a K-NN search algorithm, and the overall network architecture is as shown in Figure 3 . Given a set of input images X, after preprocessing, the basic features F are extracted by a standard convolutional neural network backbone. The architecture of the model mainly includes two branches: one is used to extract identity features F id , and the other is used to extract view features F viewThe two branches each perform feature extraction through specific convolutional layers or other deep learning modules and are separately fed into subsequent modules. In the identity feature branch, the extracted features are used to predict the identity of the vehicle and are optimized through identity loss and triplet loss. In the view feature branch, the extracted features are used to predict the view of the vehicle. At the initial stage of model training, to train the view classifier and the mapping layer that maps the backbone features to the view feature space, the view features are directly used for supervised learning of the view loss. After training, the parameters of the view classifier and the mapping layer are fixed, and the view features are input into the nearest neighbor feature enhancement module. PFEM inputs the identity and view features into the feature database and obtains a top-k similar feature list through a K-NN search algorithm. These similar features contain the same identity under different views. The selected features from the top-k list are fused through further intersection operation and averaging to generate more representative enhanced features.

[0086] The extracted identity features and enhanced features are further fused to strengthen the feature representation capability. At the same time, the enhanced features decouple the identity features and view features through GRL, eliminating the influence of the view on the identity prediction task, and thus learning more discriminative identity features to improve the overall recognition accuracy of the model.

[0087] Dual-branch feature extraction and fusion:

[0088] The model designs two parallel branches adjacent to the backbone network, aiming to deeply mine and separate the identity features and view features of the vehicle from the input image. The two branches each perform its own function: one focuses on extracting unique attributes related to the identity of the vehicle to ensure accurate identification of different vehicles; the other is committed to capturing the differences in view changes to cope with the diversity of vehicle poses in the image. On this basis, the identity features and enhanced features after PFEM are further weighted and summed, and then the enhanced features are input into GRL and the view loss, so that the model can effectively decouple the identity information and view information. Gradient reversal causes the model to be unable to distinguish the view information, forcing the model to learn more robust identity features during the backpropagation process.

[0089] Dual-branch feature extraction and decoupling:

[0090] The identity feature extraction branch and the view feature extraction branch share the same backbone layers of a convolutional neural network to generate basic features. Through the backbone network Backbone, the input image X i is first converted into basic features F.

[0091] F = Backbone(X i )(1)

[0092] On this basis, the identity feature branch maps the basic feature F into the identity feature space through a convolution or fully connected layer, generating features F that are strongly related to the vehicle identity id . This feature is used to distinguish different vehicle identities and is optimized through cross-entropy loss and triplet loss. At the same time, the view feature branch extracts features F related to the vehicle view view . Since there are significant differences between ground views and aerial views, the basic model cannot effectively prevent the influence of these view differences on identity representation when processing them, resulting in too large intra-class feature differences that cannot be narrowed. Specifically, when learning identity features, the model may mistakenly mix view features into identity features, causing a coupling relationship between view features and identity features, thereby hindering the effective extraction of identity features. To solve this problem, a gradient reversal layer is introduced during view feature extraction, which reverses the gradient so that the identity feature branch can 'ignore' view changes when learning, thereby avoiding the interference of view information on identity feature learning. Through this mechanism, the identity feature branch can focus on learning more discriminative identity features without being disturbed by view features. Specifically, the introduction of GRL makes the optimization target of identity features and the optimization target of view features independent of each other during training, thereby helping the model to more accurately separate and extract identity features and view features.

[0093] Proximal feature enhancement module

[0094] To further enhance feature representation, a proximal feature enhancement module PFEM (Proximal Feature Enhancement Module) is proposed, which uses similar feature information within the same batch of processed data to fuse identity and view features. This module is based on a K-NN search process to retrieve the K nearest neighbors from the feature database, and uses these neighbor features for feature enhancement, as shown in Figure 3 .

[0095] Specifically, for each input ID feature, K nearest neighbor features are retrieved from the feature database This process is represented as:

[0096]

[0097] The K-NN search finds the K nearest neighbor features from the identity feature database using Euclidean distance or cosine similarity as a metric. Similarly, for the view feature K nearest neighbor features

[0098] After retrieving the nearest neighbors of identity and view features, PFEM first performs an intersection operation The common part of identity feature neighbors and view feature neighbors is extracted:

[0099]

[0100] The intersection operation extracts the common features between identity and view features, which represent the feature part that has both identity discriminativeness and view information. Next, through a feature aggregation operation These common features are fused with the original identity features to generate the enhanced feature representation:

[0101]

[0102] where the aggregation operation adopts weighted average, and other aggregation manners are explained in the ablation experiment section 5.4.

[0103] Loss function

[0104] The identity recognition task is optimized by cross-entropy loss and triplet loss, and the view prediction task is optimized by cross-entropy loss. The above three losses can be represented as follows:

[0105]

[0106] where and represent the identity probability distribution and the view probability distribution predicted by the model, and represent the real identity label and the view label, is the positive sample feature, is the negative sample feature, and d(·, ·) is the distance function between features, m is a hyperparameter in the triplet loss, used to control the minimum distance difference between positive samples and negative samples. In summary, the total optimization objective of the model is as follows:

[0107]

[0108] where λ is a hyperparameter for balancing the weight of the gradient reversal layer, used to control the strength of gradient reversal.

[0109] Dataset

[0110] As shown in Table 1, the related art ground view vehicle re-identification dataset (i.e., VeRi-776, VehicleID, etc.) provides comprehensive ground camera view pictures and does not involve unmanned aerial vehicle view shooting. The related art aerial unmanned aerial vehicle view vehicle re-identification dataset only contains a single unmanned aerial vehicle view and lacks planar vehicle detail information. In order to provide a publicly available benchmark, a more realistic world scenario-compliant ground-air view vehicle re-identification dataset GAVRI (Ground Areial Vehicle Re-identification) is constructed in combination with unmanned aerial vehicles and ground cameras.

[0111] Table 1 Comparison of related art publicly available vehicle re-identification datasets

[0112]

[0113] The constructed dataset is from 6 unmanned aerial vehicle and ground camera videos, and the ground view video is decimated once every 32 frames, and the aerial view video is decimated once every 16 frames. After decimation, the labelme labeling software is used to select five lanes for labeling. The vehicle image size after unmanned aerial vehicle camera cropping ranges from a maximum of 668x647 pixels to a minimum of 56x70 pixels. The vehicle image size after cropping from the ground camera ranges from a maximum of 719x821 pixels to a minimum of 58x47 pixels. GAVRI is collected under different weather conditions and contains a large number of illumination changes, motion blurs, and a small amount of occlusions. The final dataset includes 17437 images and 2159 different vehicles. Due to the difference between the unmanned aerial vehicle view and the ground view, the dataset constructed in this paper has certain challenges, reflects the limitations of the re-identification task in real scenarios, and increases the difficulty compared to traditional aerial or ground multi-camera views.

[0114] In the evaluation stage, 1056 vehicle identities include 9610 pictures, and a ground-air dual view mixed query is used, that is, the query set and the gallery set both include ground view and aerial view pictures, rather than single ground-ground matching or air-air matching. Unlike each pedestrian, many different vehicles can look extremely similar at a certain angle, as if they are the same vehicle. Due to the dual view mixed query, in the gallery set, some vehicles not only have different IDs, but also have different views, but have high visual similarity with the current query picture, which further increases the difficulty of distinguishing similar vehicles in the gallery. The model is required to fully learn the overall features and local fine-grained features of the vehicle.

[0115] Experiments

[0116] Dataset and evaluation index

[0117] The proposed method for ground-air cross-view re-identification is verified on the pedestrian re-identification dataset of ground-air view angle. The standard data division strategy is adopted for the AG-ReID dataset.

[0118] The AG-ReID dataset comes from real-world outdoor environments, containing 21983 images of 2 different cameras, 388 pedestrian identities. The GAVRI dataset contains 17437 images from 4 different cameras, 2159 vehicles. In the evaluation, the cumulative match characteristic (CMC) and the mean average precision (mAP) are used as evaluation indicators.

[0119] In all experiments, ResNet-50 pre-trained on ImageNet is used as the backbone. To facilitate the integration of custom modules and enrich multi-scale information, the last down-sampling of ResNet-50 is removed, and global average pooling and global maximum pooling are used to integrate the output feature maps of the backbone. The two are spliced and BatchNorm is used to standardize the image features. For the GAVRI vehicle dataset, the input image size is set to 256x256, and the batch size is set to 128; the AG-ReID pedestrian dataset input image size is set to 256x128, and the batch size is set to 64. A random region center cropping is designed to randomly crop important local feature information around the center of the image, combined with random horizontal flipping and color jittering for data augmentation. The model is trained for a total of 150 epochs, with an initial learning rate of 3.5e-4, and divided by 10 every 50 epochs.

[0120] All experiments are coded using Pytorch, and experiments are conducted on a server configured with Intel(R) Xeon(R) Gold 5118 CPU @ 2.30 GHz and NVIDIA Tesla A40.

[0121] The proposed image double-branch nearest neighbor feature enhancement network DPFE is compared with several advanced ReID methods in Table 2 on GAVRI. As shown in Table 2, DPFE is compared with the most advanced method.

[0122] Table 2 Performance comparison of DPFE and other advanced algorithms on GAVRI dataset

[0123]

[0124] The mAP score of the method proposed herein reaches 65.0%, and the Rank-1 accuracy reaches 65.6%, which is better than other traditional methods, and exceeds the baseline method by 6.6% and 7.3%, exceeds CLIP-ReID by 4.8% and 6.5%, and exceeds AdaSP by 2.7% and 1.4%, and achieves a significant performance improvement. Table 3 shows the performance comparison of the AG-ReID dataset with mainstream methods under two settings. "Empty to ground" and "ground to empty" represent two specific evaluation modes. The performance in Table 3 is derived from the results in the related art, and the performance of DPFE on AG-ReID is better than other methods, including the Explain interpretable re-identification method proposed in the related art, which strongly proves the importance of DPFE in reducing the damage of view angle difference to identity distinguishing features. The results show that the method of the embodiments of the present application achieves excellent performance on both GAVRI and AG-ReID datasets. Compared with traditional vehicle re-identification methods, DPFE can capture more discriminative fine-grained local information features, and by decoupling the identity information and the view angle information, more features related to identity prediction are obtained, which significantly improves the accuracy of identity prediction.

[0125] Table 3 Comparison of DPFE and existing algorithms on AG-ReID dataset

[0126]

[0127] To verify the effectiveness of the DPFE method proposed herein and its various components, ablation experiments were conducted on the GAVRI dataset. Table 4 shows the ablation experiment results of DPFE on the GAVRI dataset, comparing the Rank-1, 5, 10 and mAP values of adding a double-branch feature extraction, a near neighbor feature enhancement module and a gradient reversal layer. The results show that using a double-branch to extract identity and view angle features, different features can be learned independently, avoiding confusion between the two. Without adding the PFEM module, the model mainly relies on global features for recognition, which may be difficult to distinguish in complex scenes (such as partial occlusion, similar details but different vehicles), resulting in performance degradation. When the model lacks GRL, the identity features and view angle features cannot be further decoupled, and the generalization ability between ground and air views is limited, and the model cannot learn more features related to vehicle identity, resulting in poor performance in cross-view scenarios. In addition, the two different feature fusion methods of the identity features and the features enhanced by PFEM are compared: DPFE using learnable parameter weighted summation and ordinary feature average summation DPFE_(avg).

[0128] Table 4 Ablation experiment of DPFE on GAVRI dataset

[0129]

[0130] From the experimental results, it can be seen that the DPFE model using learnable parameter weighted summation is better than the DPFE_(avg) using average summation in all evaluation indicators, which shows that the learnable weight plays an important role in improving the effectiveness of feature fusion.

[0131] Parameter analysis: To further verify the effectiveness of the PFEM module in this method, the number of K of the near neighbor feature in PFEM and λ in the gradient reversal process were experimented, and how they affect the performance of the model was analyzed:

[0132] (1) Five values of K were tested, which were 3, 5, 10, 15, and 20, and the experimental results are shown in Table 5. When K is 5, the overall performance of the model improves most significantly, and as K increases, the overall performance shows a downward trend. This phenomenon shows that the quality of the near neighbor feature is positively correlated with the quality of the enhanced feature. When the value of K is too small, the further screening of the intersection operation may lead to the inability to query effective neighbor features to enhance the original features; when the value of K is too large, the probability of the near neighbor features containing negative samples will be larger, and the influence of other negative samples will also be larger, resulting in a decrease in performance.

[0133] Table 5 Influence of super parameter K on model performance

[0134]

[0135] (2) In the process of gradient reversal layer, the super parameter λ controls the strength of gradient reversal. Specifically, λ determines the scaling factor when reversing the gradient, affecting the decoupling effect of the view feature and the identity feature. Set the super parameter K in PFEM to 5, and evaluate several values of λ, which are 0.05, 0.1, 0.3, and 0.5, and the experimental results are shown in Table 6. When λ = 0.1, the model performance is best. The reason is that when the value of λ is very small, the gradient reversal effect is weak, and the decoupling degree between the identity feature and the view feature is insufficient, and the identity feature still contains too much information irrelevant to the identity, affecting the overall discriminability of the model. When λ is too large, the view information will be weakened too much, which may cause the model to lose some fine-grained information related to the identity distinction, resulting in a decrease in performance.

[0136] Visualization: For the feature map visualization analysis of different methods on the dataset, Figure 4The second convolutional layer of the third bottleneck of the fourth stage of the model is visualized by the Grad-CAM tool, showing the effects of the baseline method, AdaSp and DPFE. The attention distribution of each model in different parts of the vehicle can be observed intuitively through the heat map. Through comparison, it can be seen that: the feature map of the baseline method is mainly concentrated in the local area of the vehicle, such as the headlight or license plate, indicating that its effect is limited when dealing with complex background or occlusion, and the attention area is limited. Second, the AdaSp method has improved significantly compared with the baseline method, and the feature map captures the important areas of the vehicle in a larger range, such as the roof and tail. However, the features still have certain local concentration phenomenon, and the whole vehicle information is not fully covered. Compared with the former two, the feature map of DPFE shows more uniform feature distribution. With the help of PFEM, the expression of local features is further strengthened, and combined with feature decoupling, the feature map highlights more information useful for vehicle identity prediction. The model can cover the key parts of the vehicle more comprehensively, especially in complex backgrounds, it can still effectively capture the global information of the vehicle, avoid the feature being too concentrated in a small range, and show stronger vehicle appearance feature representation ability. Therefore, DPFE can achieve higher accuracy.

[0137] Table 6 Influence of hyperparameter λ on model performance

[0138]

[0139] Eliminating the view difference between the ground and the UAV perspective is an important factor for improving the performance of ground-air vehicle re-identification. DPFE reduces the sensitivity of the model to the perspective, decouples the identity features and the perspective features, and forces the model to learn more identity robust features. By learning the relevance and difference between different perspectives, more accurate query results are provided under complex and extreme perspective changes. Experiments on the newly constructed GAVRI dataset and other related datasets show that DPFE has a larger performance increase than the baseline method, and is superior to the most advanced vehicle re-identification method in related technologies on the GAVRI dataset.

[0140] Another embodiment of the present application provides a ground-air video vehicle re-identification device, which can include:

[0141] A dual-branch near neighbor feature enhancement network is used to extract the identity features and perspective features of the vehicle from the input image; the network includes two parallel branches adjacent to the backbone network, one branch is used to extract the identity attributes of the vehicle, and the other branch is used to capture the difference of the perspective change;

[0142] The identity features and the enhanced features through the near neighbor feature enhancement module PFEM are weighted and summed, the enhanced features are accessed to the GRL and the perspective loss, and the module decouples the identity information and the perspective information of the vehicle.

[0143] Exemplarily, the dual-branch near neighbor feature enhancement network extracts the identity feature and the perspective feature of the vehicle from the input image, including:

[0144] The identity feature extraction branch and the perspective feature extraction branch share the same backbone layer of the convolutional neural network, which is used to generate the basic feature;

[0145] Through the backbone network Backbone, the input image X i is first converted into a basic feature F;

[0146] The identity feature branch maps the basic feature to the identity feature space through the convolution or fully connected layer, and generates a feature F id which is strongly related to the identity of the vehicle and is used to distinguish different vehicle identities;

[0147] The perspective feature branch extracts a feature F view .

[0148] Exemplarily, the weighted sum operation of the identity feature and the enhanced feature through the PFEM is performed, the enhanced feature is accessed to the GRL and the perspective loss, and the identity information and the perspective information of the vehicle are decoupled, including:

[0149] The near neighbor feature enhancement module PFEM uses similar feature information in the same batch of processing data to fuse the identity and perspective features; the module is based on a K-NN search process to retrieve the K nearest neighbors from the feature database, and uses the neighbor features for feature enhancement, K is a positive integer.

[0150] Exemplarily, for each input ID feature, K nearest neighbor features are retrieved from the feature database The process is represented as:

[0151]

[0152] The K-NN search finds the K nearest neighbor features from the identity feature database through the Euclidean distance or cosine similarity metric, which are the closest to the current sample feature.

[0153] Exemplarily, for the perspective feature K nearest neighbor features are retrieved from the database

[0154] After retrieving the nearest neighbor features of the identity and perspective, the PFEM first performs an intersection operation extracting the part shared by the identity feature neighbors and the perspective feature neighbors:

[0155] ​

[0156] The intersection operation extracts the common features between the identity and view features, which represent the feature parts that are both discriminative and contain view information.

[0157] Exemplarily, the feature aggregation operation is The common features are fused with the original identity features to generate the enhanced feature representation:

[0158]

[0159] The aggregation operation adopts a weighted average.

[0160] Exemplarily, the identity recognition task is optimized by the cross-entropy loss and the triplet loss, and the view prediction task is optimized by the cross-entropy loss; the cross-entropy loss for the identity recognition task, the triplet loss, and the cross-entropy loss for the view prediction task are represented as follows, respectively:

[0161]

[0162] wherein, and represent the identity probability distribution and the view probability distribution predicted by the model, respectively, and represent the real identity label and the view label, respectively, is the positive sample feature, is the negative sample feature, d(·, ·) is a distance function between features, and m is a hyperparameter in the triplet loss, used to control the minimum distance difference between the positive sample and the negative sample, and the total optimization objective of the model is:

[0163]

[0164] wherein λ is a hyperparameter for balancing the weight of the gradient reversal layer, used to control the strength of the gradient reversal.

[0165] Another embodiment of the present application provides an electronic device comprising a processor and a memory; the processor implements the method of any embodiment of the present application when executing the running program stored in the memory.

[0166] Exemplarily, as Figure 5 shown, Figure 5A structural diagram of an electronic device provided for a specific example includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602 and the memory 603 communicate with each other through the communication bus 604. The memory 603 is used to store a computer program; the processor 601 is used to execute the program stored in the memory 603, so as to realize the method provided by any of the embodiments of the present application.

[0167] In the above electronic device, the communication bus can be a bus of a peripheral component interconnect standard or an extended industry standard architecture bus, etc. The communication bus can include an address bus, a data bus and a control bus, etc. The communication interface is used to realize the data exchange between the electronic device and other devices.

[0168] The memory 603 can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. In addition, the memory 603 can also include other storage devices located remotely from the processor 601. The processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other types of programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0169] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the embodiments of the present application. Referring to Figure 6 as shown, Figure 6 The computer-readable storage medium shown is an optical disc 20, and the optical disc 20 stores a computer program (i.e., a program product), which, when executed by a processor, can implement the method described in any of the embodiments of the present application.

[0170] The computer-readable storage medium of the present embodiment can be any type of medium accessible by a computer, or a server, a data center or other data storage device integrated with multiple media. Common types of storage media include magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs) or semiconductor media (such as solid state disks (SSD)), etc.

[0171] In the above embodiments, all or part of the functions can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the functions can be realized by one or more computer instructions contained in a computer-readable storage medium. When the computer loads and executes these computer program instructions, part or all of the processes or functions described in the embodiments of the present application can be executed.

[0172] The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer readable storage medium, or transmitted from one computer readable storage medium to another medium through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. For example, computer instructions can be transmitted from a website, computer, server or data center to another computer, website, server or data center.

[0173] It should be noted that:

[0174] In the embodiments of the present application, the terms "comprise", "include" and any other variants thereof are intended to refer to a non-exclusive inclusion, meaning that when referring to a process, method, article or device comprising certain elements, the process, method, article or device not only includes these elements, but also can include other elements not explicitly listed, or includes inherent elements related to the process, method, article or device. Without further limitation, the use of "comprising a" expression of elements does not exclude the possibility of other same elements existing in the process, method, article or device containing the element.

[0175] Through the description of the above-mentioned embodiment method, those skilled in the art can clearly understand that these methods can be realized by software plus necessary general hardware platform. Of course, it can also be realized by pure hardware, but in many cases, the former is usually a more optimal implementation. Based on this understanding, the technical solutions of the embodiments of the present application, or the technical contributions made by the present application, can be embodied in the form of software products, which are stored in a storage medium (such as ROM / RAM, disk, optical disk, etc.), and contain a number of instructions to make image display devices (such as mobile phones, computers, servers, air conditioning devices or network devices, etc.) execute the methods described in the embodiments of the present application.

[0176] The above is a specific implementation of the embodiments of the present application, but does not limit the protection scope of the present application. Any skilled person in the art can think of changes or alternatives within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be determined according to the content of the claims.

Claims

1. A method for re-identifying vehicles in ground-to-air video, characterized in that, include: A dual-branch nearest neighbor feature enhancement network is used to extract vehicle identity features and viewpoint features from the input image. The network includes two parallel branches of the backbone network, one branch is used to extract vehicle identity attributes, and the other branch is used to capture differences in viewpoint changes. The identity feature extraction and viewpoint feature extraction branches share the same backbone layer of the convolutional neural network, which is used to generate basic features; Input image X via the backbone network i First, it is converted into basic feature F; The identity feature extraction branch maps the basic features F to the identity feature space through convolutional or fully connected layers, generating features F that are strongly correlated with the vehicle's identity. id This feature is used to distinguish different vehicle identities; The viewpoint feature extraction branch extracts features F related to the vehicle's viewpoint. view ; The vehicle's identity information and view information are decoupled by performing a weighted summation operation on the identity features and the enhanced features through the nearest neighbor feature enhancement module PFEM. The enhanced features are then fed into the gradient inversion layer and the view loss. The nearest neighbor feature enhancement module PFEM utilizes similar feature information within the same batch of processed data to fuse identity and viewpoint features. This module is based on a K-NN search process, which retrieves the K nearest neighbor features of the current sample from the feature library and uses these neighbor features for feature enhancement, where K is a positive integer. For each input identity feature, in the feature database Retrieve K nearest neighbor features This process can be represented as: K-NN search uses Euclidean distance or cosine similarity as a metric to find the K nearest neighbors in an identity feature database that are most similar to the current sample's features. For viewpoint features Retrieve the K nearest neighbor features from the database. After retrieving the nearest neighbor features for identity and perspective, PFEM first performs an intersection operation. Extract the common components from the identity nearest neighbor features and the viewpoint nearest neighbor features: The intersection operation extracts the common features between identity and viewpoint nearest neighbor features. These features represent feature parts that have both identity distinguishability and viewpoint information. Through feature aggregation operations These public characteristics are compared with the original identity characteristics Perform fusion to generate enhanced feature representations: Among them, aggregation operation A weighted average is used.

2. The method according to claim 1, characterized in that, The identity recognition task is optimized by both cross-entropy loss and triplet loss, while the viewpoint prediction task is optimized by cross-entropy loss. The cross-entropy loss for the identity recognition task, the triplet loss for the viewpoint prediction task, and the cross-entropy loss for the viewpoint prediction task are expressed as follows: in, and Let these represent the identity probability distribution and the viewpoint probability distribution predicted by the model, respectively. and These represent real identity tags and perspective tags, respectively. These are the original characteristics of identity. These are positive sample features. Let be the negative sample features, d(·,·) be the distance function between features, and m be the hyperparameter in the triplet loss, used to control the minimum distance difference between positive and negative samples. The overall optimization objective of the model is: Where λ is a hyperparameter for balancing the weights of the gradient inversion layer, used to control the strength of gradient inversion.

3. A ground-to-air video vehicle re-identification device, characterized in that, Performing the method as described in claim 1 or 2 includes: A module employing a dual-branch nearest neighbor feature enhancement network extracts vehicle identity features and viewpoint features from the input image; the network comprises two parallel branches adjacent to the backbone network, one branch for extracting vehicle identity attributes and the other branch for capturing differences in viewpoint changes; The module decouples vehicle identity information from view information by performing a weighted summation operation on the identity features and the enhanced features obtained through the nearest neighbor feature enhancement module PFEM, and then connecting the enhanced features to the gradient inversion layer and view loss.

4. An electronic device, characterized in that, It includes a processor and a memory; the processor, when executing a running program stored in the memory, implements the method as described in any one of claims 1 and 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 and 2.

Images