Air-ground cross-platform target re-identification method based on semantic alignment and prompt learning
By employing semantic alignment and cue learning methods, a visual encoder and a text encoder are constructed using a large-scale visual language model. This allows for the learning of cross-platform invariant semantic features of pedestrians in both air and ground environments. Furthermore, platform cue vectors are introduced to address the visual feature misalignment problem in cross-platform video target re-identification, thereby improving recognition accuracy.
Patent Information
- Application Number
- CN202411008010.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-25
AI Technical Summary
Existing cross-platform video target re-identification methods suffer from severe feature misalignment due to the large differences in perspective between ground monitoring and drone cameras. Traditional methods cannot effectively align visual features across air and ground platforms.
We employ a semantic alignment and cue learning approach, utilizing a large-scale visual language model to construct a visual encoder and a text encoder. By learning cross-platform invariant semantic features of pedestrians in space and ground, and introducing platform cue vectors during training, we guide the model to pay attention to platform differences and achieve feature alignment.
It effectively solved the problem of misaligned visual features across platforms and improved the average accuracy of cross-platform video pedestrian re-identification from 74.5% to 76.5%.
Smart Images

Figure CN119007241A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and specifically designs a method for solving the problem of severe misalignment of image features in air-ground cross-platform video target re-identification based on semantic alignment and prompt learning. Background Art
[0002] Video-based object re-identification (VReID) has attracted much attention in recent years, and most of the existing video-based ReID research is based on data captured by the same platform (such as ground surveillance cameras). Assume that in this case, the suspect committed a crime in a city where a large number of surveillance cameras are deployed, and fled to an area where no ground surveillance cameras were deployed in advance. A feasible solution is to use aerial drones to capture the trajectory of the target object, so the requirement scenario becomes air-to-ground cross-platform (ground surveillance-to-air drone) video object re-identification. Air-to-ground cross-platform pedestrian re-identification faces a serious problem. Due to the huge difference in viewing angles between ground surveillance and drone cameras, there is a serious feature misalignment problem in the images of the two platforms, resulting in poor performance of traditional single-platform methods. Therefore, how to effectively align air-to-ground cross-platform visual features is a key issue in the task of air-to-ground cross-platform video pedestrian re-identification.
[0003] Traditional pedestrian re-identification methods mainly use the block alignment method to divide pedestrians into different regions and realize block-level feature alignment. For example, block convolution (PCB) cuts the pedestrian image horizontally into different regions and aligns the features of the corresponding regions of the two images in turn; BPBreID uses the joint point detection network to extract pedestrian joint point information, divides the image into different regions according to the pedestrian joint points, and then realizes feature alignment on this basis. However, these methods have obvious disadvantages. The PCB method is simple, and the region division is too rough, which is only suitable for application scenarios with small differences in viewing angles of a single platform. Although BPBreID uses a more accurate human joint point detection network, from the perspective of the drone, the pedestrian image has serious self-occlusion problems due to the downward viewing angle (such as: the legs are completely blocked by the upper body), so it is impossible to effectively extract pedestrian joint point information, and thus it is impossible to effectively realize feature alignment.
[0004] Recently, with the emergence of large-scale pre-trained visual language models (CLIP), a well-aligned visual semantic space can be obtained by cross-modal contrastive learning of large-scale network visual data and high-level language descriptions. Although in the pedestrian re-identification task, each person's identity is only represented by an index number without a language description, a set of learnable text vectors can also be introduced to roughly describe the semantic features of each pedestrian. Therefore, this paper proposes to use the basic visual language model CLIP to extract the cross-platform invariant semantic features of pedestrians and spaces, and to implement an efficient constraint model by aligning the dual-platform visual features with the cross-platform invariant semantic features of spaces and spaces respectively. Furthermore, prompt learning uses prompt vectors to guide the model to pay attention to task differences. Therefore, on the basis of semantic alignment, the present invention introduces platform prompt vectors to guide the model to obtain accurate and targeted platform information, thereby further bridging the gap in visual features. Summary of the invention
[0005] In order to overcome the shortcomings of the prior art, the present invention provides a method for re-identifying cross-platform objects in space and ground based on semantic alignment and prompt learning. The problem of non-alignment of visual features in space and ground is solved by using a large visual language basic model. The model includes two branches, a visual encoder and a text encoder, and a learnable pedestrian description and platform prompt module; wherein the visual encoder is constructed based on a visual self-attention encoder Vision Transformer, and its main function is to extract global image information; the text encoder is constructed based on a self-attention encoder Transformer, and its main function is to encode text information; the training of this model is divided into two stages. In the first stage, the visual encoder and the text encoder load the weights of the pre-trained model, and freeze all the parameters of the visual encoder and the text encoder, and use the image features of different platforms to learn the cross-platform invariant semantic information of pedestrians in space and ground. In the second stage, the pedestrian text description and the text encoder weights are frozen, and the image features output by the image encoder are aligned with the cross-platform invariant semantic features of pedestrians in space and ground. At the same time, the platform prompt vector is introduced in the second stage, so as to train the image encoder to pay attention to the invariant features of pedestrians in space and ground cross-platform images, and realize the alignment of space and ground cross-platform features.
[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:
[0007] Step 1: Dataset preparation;
[0008] A large-scale air-to-ground cross-platform pedestrian re-identification dataset is collected using surveillance cameras and drones, and named G2A-VReID. The dataset G2A-VReID contains multiple scenes, and the videos are collected at six different locations: library, intersection, bus station, school, subway entrance and square. Ground surveillance cameras are used to shoot videos from the ground perspective, and drones are used to collect videos from the sky perspective. The surveillance camera is fixed at a height of two meters above the ground, and the drone flies at different heights of 20 to 60 meters. The drone flies in hovering, cruising and rotating modes, so that the captured characters include perspectives from different bird's-eye views. Then, the collected raw video data is annotated. During the annotation process, all characters appearing in the video are annotated with rectangular bounding boxes, and each person is cropped out from the scene image according to the bounding box. Finally, manual annotation is performed to associate the same pedestrian in the drone and surveillance video, and assign a unique ID number to each pedestrian with different identities. Next, all images of pedestrians with the same ID from the same video track are merged into one video track. Each pedestrian has at least two tracks, one from the surveillance camera and the other from the drone.
[0009] Step 2: Build a semantic alignment network model;
[0010] Step 3: Establish the loss function;
[0011] The loss function includes a first training stage loss function and a second training stage loss function, the first training stage loss function includes image-text contrast loss, and the second training stage loss function includes triple loss, cross entropy loss and image-text cross entropy loss;
[0012] Step 4: Use loss function The network model is trained in the first stage with the training set to learn the semantic description of pedestrians and adopt the loss function The semantic alignment network model is trained in the second stage. During the second stage of training, only all parameters in the visual encoder of the semantic alignment network model are adjusted. After each epoch of training, the average precision mAP and the first match accuracy Rank-1 evaluation indicators are used to verify the model performance in the test set of the data set, and the mAP+Rank-1 value is recorded. The model weight when the mAP+Rank-1 value is the largest during the entire training process is saved as the trained parameter. After the model training cycle is completed, the test set is used for testing to verify the performance of the model.
[0013] In step 2, the semantic alignment network model includes four parts: a visual encoder, a text encoder, a learnable pedestrian description module, and a platform prompt module; the specific construction steps are as follows:
[0014] Step 2-1: Visual encoder;
[0015] The visual encoder includes a block encoder, a self-attention encoder, and a layer normalization module;
[0016] The block encoder is a 16×16 convolution block with a step size of 16. It divides the input image into fixed 16x16 image blocks and converts the image blocks into block codes. Then, a position code is added to each image block vector. The block code and the position code are both vectors of the same dimension. The two vectors are added so that the model can distinguish image blocks at different positions. A class code is added and the class code and the image block vector are concatenated into a code sequence. Finally, the code sequence is sent to a multi-layer series self-attention encoder. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layer helps the model learn the global and local features of the image and encode the visual features into the class code.
[0017] Step 2-2: Text encoder;
[0018] The text encoder includes a text encoder, a self-attention encoder and a layer normalization module;
[0019] The input of the text encoder is a piece of text information. First, the text encoder splits a piece of text into independent words and converts each word into a word code. Then, the word code and the class code are spliced into a word code sequence, and the word code sequence is sent to a multi-layer series of self-attention encoders. Each self-attention encoder contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layer collects semantic features in the word code sequence and encodes the semantic features into the class code. After each layer of self-attention encoder, the word code sequence will pass through the layer normalization module to achieve vector normalization.
[0020] Step 2-3: Pedestrian description module can be learned;
[0021] The learnable pedestrian description module includes a pedestrian description template and a learnable vector;
[0022] The pedestrian description template and the learnable vector are concatenated into a complete pedestrian description text in the form of "A photo of [x1] ... [xm] perosn", where "A photo of ... person" is the pedestrian description template and "[x1] ... [xm]" is a learnable vector of length m.
[0023] Step 2-4: Platform prompt module;
[0024] The platform prompt module includes two sets of prompt vectors;
[0025] The platform prompt module is regarded as changing the input of each self-attention encoder in the visual encoder, and the input vector of the self-attention encoder is represented as Where N = HW / P 2 , D represents the vector dimension, N represents the vector length, H and W are the length and width of the input image, and P is the step size of the block encoder in the visual encoder. After adding the visual cue module, the self-attention encoder is expressed as:
[0026]
[0027] in d and l are the depth and length of the platform hint vector, MSA k Represents the MSA of the kth layer in the visual encoder, Set ground and Set uav There are two collections, containing samples from ground platforms and drones respectively.
[0028] In step 3, the image-text contrast loss of the loss function of the first training stage is defined as follows:
[0029]
[0030] in, Represents the text semantic features of the pedestrian with ID i, I p represents the pedestrian video image encoding with ID P, I a represents the pedestrian video image encoding with ID a, s() represents the cosine similarity, P(y i ) means including i All positive sample sets, B represents the training Batch Size;
[0031] The triplet loss is defined as follows:
[0032]
[0033] Among them, θ is The soft margin, d p and d n are the characteristic distances of positive pairs and negative pairs respectively;
[0034] The cross entropy loss is defined as follows:
[0035]
[0036] Among them, p k represents the number of ID predictions for category k, q k Distribute soft labels for targets;
[0037] The visual-to-semantic contrast loss is defined as follows:
[0038]
[0039] Among them, V i represents the video visual features of the pedestrian with ID i, Represents the text semantic features of the pedestrian with ID k, represents the text semantic features of the pedestrian with ID j, q k Distribute soft labels for targets;
[0040] The overall first-stage loss function is expressed as:
[0041]
[0042] The overall second-stage loss function is expressed as:
[0043]
[0044] Among them, β, γ and ∈ represent relative importance weights.
[0045] The first training stage uses the Adam optimizer to optimize the pedestrian independent vector in the learnable pedestrian module.
[0046] Preferably, the number of training epochs is 120.
[0047] Preferably, in the platform prompt module, the prompt vector length l=4, and the prompt vector depth d=9.
[0048] Preferably, the image size of the training set and the test set is 256×128, and each video clip contains 8 frames of images.
[0049] Preferably, the relative importance weights β, γ and ∈ of the second stage loss function are 1.0, 0.25 and 1.0 respectively.
[0050] An electronic device comprises one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described above.
[0051] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.
[0052] The beneficial effect of the present invention lies in that it adopts the technical route of semantic alignment and prompt learning, effectively solving the challenge of cross-platform visual feature misalignment in the cross-platform video pedestrian re-identification task. The model of the present invention has a carefully designed learnable pedestrian description module. Through the first stage of comparative training, the learnable pedestrian description module learns the pedestrian's cross-view pedestrian invariant semantic features. In the second stage, the visual features of cross-platform pedestrians in the air and ground are aligned with the invariant semantic features of pedestrians, thereby solving the problem of serious misalignment of visual features in cross-platform video pedestrian re-identification in the air and ground. On the G2A-VreID dataset, due to the misalignment of cross-platform visual features, the performance of traditional methods is poor. For example, SINet only achieves a mean accuracy (mAP) of 74.5%. The present invention effectively compensates for the problem of cross-platform visual image feature misalignment by aligning the visual features of cross-platform pedestrians in the air and ground with the invariant semantic features of pedestrians. This method achieves a mean accuracy (mAP) of 76.5% in G2A-VreID.
[0053] The present invention proposes a platform prompt module, which is introduced in the second stage of model training. The model visual encoder is guided to pay attention to platform differences through platform-independent prompt vectors, thereby further bridging the platform differences. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a training flow chart of the method of the present invention.
[0055] Figure 2 It is a structural diagram of the network model constructed by the method of the present invention. DETAILED DESCRIPTION
[0056] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0057] The purpose of the present invention is to overcome the shortcomings of the above-mentioned existing technologies, and propose a method for pedestrian re-identification in air-ground cross-platform videos based on visual semantic alignment and prompt learning. The problem of non-alignment of visual features in air-ground cross-platform is solved by using a large visual language basic model. The model includes two branches, a visual encoder and a text encoder, as well as a learnable pedestrian description and platform prompt module; wherein the visual encoder is constructed based on the visual self-attention encoder Vision Transformer, and its main function is to extract global image information; the text encoder is constructed based on the self-attention encoder Transformer, and its main function is to encode text information; the training of this model is divided into two stages. In the first stage, the visual encoder and the text encoder load the pre-trained model weights, and freeze all the parameters of the visual encoder and the text encoder, and use the image features of different platforms to learn the pedestrian space-ground cross-platform invariant semantic information. In the second stage, the pedestrian text description and the text encoder weights are frozen, and the image features output by the image encoder are aligned with the pedestrian space-ground cross-platform invariant semantic features. At the same time, the platform prompt vector is introduced in the second stage, so as to train the image encoder to pay attention to the pedestrian invariant features of the air-ground cross-platform image, and realize the alignment of air-ground cross-platform features.
[0058] The technical solution adopted by the present invention to solve the technical problem includes the following steps:
[0059] Step 1: Dataset preparation;
[0060] A large-scale air-to-ground cross-platform pedestrian re-identification dataset named G2A-VReID is collected using surveillance cameras and drones, including 185,907 images, 5,576 video tracks, and 2,788 pedestrians with different numbers. To increase the richness of the data, the G2A-VReID dataset contains rich scenes, and the videos are collected in six different locations, including libraries, intersections, bus stations, schools, subway entrances, and squares. Ground surveillance cameras are used to capture videos from the ground perspective, and DJI Mavic drones are used to capture videos from the sky perspective. Specifically, the surveillance camera is fixed at a height of two meters above the ground, and the drone flies at different altitudes from 20 to 60 meters. The drone flies in a hovering, cruising, and rotating manner, so that the captured characters include perspectives from different bird's-eye views. Then, the collected raw video data is annotated. During the annotation process, all characters appearing in the video will be annotated with a rectangular bounding box, and each person will be cropped out of the scene image according to the bounding box. Finally, data annotators manually annotate the same pedestrians in the drone and surveillance videos, and assign unique ID numbers to each pedestrian with different identities. Next, all images of pedestrians with the same ID from the same video track are merged into one video track, so that each person has at least two tracks, one from the surveillance camera and the other from the drone. Finally, the dataset is randomly divided into training and test sets at a ratio of 1:2. Therefore, the test set contains 1,858 pedestrians with different IDs, and the training set contains 930 pedestrians with different IDs.
[0061] Step 2: Build a semantic alignment network model;
[0062] The semantic alignment network model consists of four parts: visual encoder, text encoder, learnable pedestrian description module and platform prompt module; the specific construction steps are as follows:
[0063] Step 2-1: Visual encoder;
[0064] The visual encoder includes a block encoder, a self-attention encoder, and a layer normalization module;
[0065] The block encoder is a 16×16 convolution block with a step size of 16. It divides the input image into fixed 16x16 image blocks and converts the image blocks into a vector representation. Then, a position code is added to each image block vector so that the model can distinguish image blocks at different positions. A class code is added and the class code and the image block vector are concatenated into a code sequence. Finally, the code sequence is fed into a multi-layer series of self-attention encoders. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layer helps the model learn the global and local features of the image and encode the visual features into the class code.
[0066] Step 2-2: Text encoder;
[0067] The text encoder includes a text encoder, a self-attention encoder and a layer normalization module;
[0068] The input of the text encoder is a piece of text information. First, the text encoder splits a piece of text into independent words and converts each word into a word code. Then, the word code and the class code are spliced into a word code sequence, and the word code sequence is sent to a multi-layer series of self-attention encoders; each self-attention encoder contains a multi-head self-attention mechanism and a feedforward neural network; the encoder layer collects semantic features in the word code sequence and encodes the semantic features into the class code. After each layer of self-attention encoder, the word code sequence will pass through the layer normalization module to achieve vector normalization;
[0069] Step 2-3: Pedestrian description module can be learned;
[0070] The learnable pedestrian description module includes a pedestrian description template and a learnable vector;
[0071] The pedestrian description template and the learnable vector are concatenated into a complete pedestrian description text in the form of "A photo of [x1] ... [xm] perosn", where "A photo of person" is the pedestrian description template and "[x1] ... [xm]" is a learnable vector of length m.
[0072] Step 2-4: Platform prompt module;
[0073] The module consists of two sets of hint vectors.
[0074] Using the platform prompt module can be seen as changing the input of each self-attention encoder in the visual encoder, and the input vector of the self-attention encoder is represented as Where N = HW / P 2 , D represents the vector dimension, N represents the vector length, H and W are the length and width of the input image, and P is the step size of the block encoder in the visual encoder. After adding the visual cue module, the self-attention encoder is expressed as:
[0075]
[0076] in d and l are the depth and length of the platform hint vector, MSA k represents the MSA of the kth layer in the visual encoder, Setg round and Set uav There are two sets, containing samples from ground platforms and drones respectively;
[0077] Step 3: Establish the loss function;
[0078] The loss function includes a first training stage loss function and a second training stage loss function, the first training stage loss function includes image-text contrast loss, and the second training stage loss function includes triple loss, cross entropy loss and image-text cross entropy loss;
[0079] The image-text contrast loss of the loss function in the first training stage is defined as follows:
[0080]
[0081] in, Represents the text semantic features of the pedestrian with ID i, I p represents the pedestrian video image encoding with ID P, s() represents the cosine similarity, P(y i ) means including i All positive sample sets, B represents the training Batch Size;
[0082] The triplet loss is defined as follows:
[0083]
[0084] Among them, θ is The soft margin, d p and d n is the characteristic distance between positive and negative pairs;
[0085] The cross entropy loss is defined as follows:
[0086]
[0087] Among them, p k Represents the number of ID predictions for category k;
[0088] The visual-to-semantic contrast loss is defined as follows:
[0089]
[0090] Among them, V i represents the video visual features of the pedestrian with ID i, represents the text semantic features of the pedestrian with ID i, q k Target distribution soft labels;
[0091] The overall first-stage loss function is expressed as:
[0092]
[0093] The overall second-stage loss function is expressed as:
[0094]
[0095] Among them, β, γ and ∈ represent relative importance weights, where β = 1.0, γ = 0.25, and e = 1.0;
[0096] Step 4: Use loss function The network model is trained in the first stage with the training set to learn the semantic description of pedestrians and adopt the loss function The semantic alignment network model is trained with the training set in the second stage. In the second stage of training, only all parameters in the visual encoder of the semantic alignment network model are adjusted; after the preset training cycle of the model is completed, the test set is used for testing to verify the performance of the model. Specific embodiment:
[0098] (1) Dataset preparation:
[0099] Using panchromatic images collected by ground-based cameras and aerial drones, the steps are as follows:
[0100] ① Dataset collection: A large-scale air-ground cross-platform pedestrian re-identification dataset was collected using surveillance cameras and drones to increase the richness of the data and make it closer to the real environment. The video collection scenes include 9 different scenes such as libraries, intersections, bus stations, and tourist attractions. Ground surveillance cameras are used to shoot videos from the ground perspective, and DJI Mavic drones are used to collect videos from the sky perspective. Specifically, the surveillance camera is fixed at a height of about two meters from the ground, and the drone flies at different altitudes from 20 to 60 meters. The drone flies in a hovering, cruising, and rotating manner, so that the captured characters contain a richer perspective.
[0101] ② Dataset annotation. During the annotation process, all people appearing in the video will be marked with bounding boxes, and each person will be cropped out of the scene image based on the bounding box. The same person in the drone and surveillance video is associated and assigned a unique ID. Next, we merge all images of a person from one camera into one track. In this way, each person has at least two tracks, one from the surveillance camera and the other from the drone. The dataset contains a total of 2778 pedestrians with different IDs, each pedestrian contains 2 video clips from different platforms, and each video clip contains an average of 33.3 frames of images.
[0102] ③ Dataset division: randomly divide the training set and test set in a ratio of 1:2. Therefore, the test set contains 1858 pedestrians with different IDs, and the training set contains 930 pedestrians with different IDs.
[0103] ④ Randomly extract 8 frames of images from the original video clip, and scale the length and width of each frame to 256×128. Perform random flipping, random mosaic and normalization on a group of video images, and finally convert them into the Pytorch Tensor data format.
[0104] At this point, the dataset preparation step is completed.
[0105] (2) Constructing a complementary fusion network model
[0106] The network structure diagram is as follows Figure 2 As shown, the parameters for building the network are:
[0107] The image encoder uses ViT-B-16 with a vector length of 768, and the video encoder also uses ViT-B-16 with a sequence length of 512. The pedestrian independent learnable vector length of the pedestrian learnable module is 4. The length of the platform prompt module is 4 and the depth is 9.
[0108] (3) Network training;
[0109] Input image: full-color video clip V, size 8×256×128 (video frame length×height×width). A set of video images are randomly flipped, randomly mosaicked, and normalized, and finally converted into the data format of the deep learning tensor library Pytorch Tensor.
[0110] Other related settings: In the PyTorch framework, the adaptive matrix estimation optimizer Adam optimizer is used to update parameters in both training stages. The initial weights of the model are the original pre-trained weights of CLIP. In the first training stage, all weights of the image encoder and text encoder are frozen, and the Adam optimizer is used to optimize the pedestrian independent vector in the learnable pedestrian module. The number of training epochs is set to 120, the batch size of batch training is set to 256, and the initial learning rate is set to 0.000035. The initial weights of the model in the second training stage are the weights of the model after the first stage training, all parameters in the learnable pedestrian description module and text encoder are frozen, and the Adam optimizer is used to optimize the image encoder. The number of training epochs is set to 120, the batch size of batch training is set to 32, and the initial learning rate is set to 0.000005. The learning rate is decayed by 90% at the 50th and 90th epochs. After each epoch training, the average precision mAP and the first match accuracy Rank-1 evaluation indicators are used to verify the model performance on the dataset test set, and the mAP+Rank-1 value is recorded. After the training is completed, the model weight when the mAP+Rank-1 value is the largest during the entire training process is saved.
[0111] The training stop condition is: completing 120 rounds of training.
[0112] (4) Network testing
[0113] ① Input image: video clip V, size 8×256×128 (video frame length×height×width), a set of video images are subjected to random flipping, random mosaic and normalization processing in turn, and finally the video clip V is converted into the data format of PytorchTensor.
[0114] ② Load the network parameters with the maximum mAP+Rank-1 value saved at the end of the second training phase, and verify the model performance on the test set data. The test set is divided into a query set and a candidate set. The query set contains N pedestrian images, and each image corresponds to K images in the candidate set. The model uses a visual encoder to encode any image in the candidate set and all images in its corresponding candidate set into a set of vectors, measures the similarity between the candidate set image and the query set image by cosine similarity, and sorts them from high to low in terms of similarity. If for a query set image, the image with the highest similarity in the candidate set is different from the ID of the query set image, then the recognition is correct. Otherwise, if for a query set image, the image with the highest similarity in the candidate set is different from the ID of the query set image, then the recognition is wrong.
[0115] ③Network output results: The final output results are displayed using the average precision mAP and first match accuracy Rank-1 evaluation indicators.
[0116] Experiments show that after introducing the platform prompt module, the performance of this method is further improved on the G2A-VreID dataset. As shown in Table 3:
[0117] Table 1 Performance comparison between the method of the present invention and different classical methods
[0118]
[0119]
[0120] The present invention achieves a mean average precision (mAP) of 78.1% in G2A-VreID.
[0121] Compared with the classic method, the present invention is based on visual semantic alignment and platform prompt learning, which effectively solves the problem of visual feature misalignment in cross-platform pedestrian re-identification and achieves better results. Compared with PiT, the present invention improves the average precision mAP by 1.8% and the first match accuracy Rank-1 by 1.6%.
Claims
1. A space-to-ground cross-platform object re-identification method based on semantic alignment and cue learning, characterized in that The steps include: Step 1: Dataset preparation; A large-scale air-to-ground cross-platform pedestrian re-identification dataset is collected using surveillance cameras and drones, and named G2A-VReID. The dataset G2A-VReID contains multiple scenes, and the videos are collected at six different locations: library, intersection, bus station, school, subway entrance and square. Ground surveillance cameras are used to shoot videos from the ground perspective, and drones are used to collect videos from the sky perspective. The surveillance camera is fixed at a height of two meters above the ground, and the drone flies at different heights of 20 to 60 meters. The drone flies in hovering, cruising and rotating modes, so that the captured characters include perspectives from different bird's-eye views. Then, the collected raw video data is annotated. During the annotation process, all characters appearing in the video are annotated with rectangular bounding boxes, and each person is cropped out from the scene image according to the bounding box. Finally, manual annotation is performed to associate the same pedestrian in the drone and surveillance video, and assign a unique ID number to each pedestrian with different identities. Next, all images of pedestrians with the same ID from the same video track are merged into one video track. Each pedestrian has at least two tracks, one from the surveillance camera and the other from the drone. Step 2: Build a semantic alignment network model; Step 3: Establish the loss function; The loss function includes a first training stage loss function and a second training stage loss function, the first training stage loss function includes image-text contrast loss, and the second training stage loss function includes triple loss, cross entropy loss and image-text cross entropy loss; Step 4: Use loss function The network model is trained in the first stage with the training set to learn the semantic description of pedestrians and adopt the loss function The semantic alignment network model is trained in the second stage. During the second stage of training, only all parameters in the visual encoder of the semantic alignment network model are adjusted. After each epoch of training, the average precision mAP and the first match accuracy Rank-1 evaluation indicators are used to verify the model performance in the test set of the data set, and the mAP+Rank-1 value is recorded. The model weight when the mAP+Rank-1 value is the largest during the entire training process is saved as the post-training parameter. After the model training cycle is completed, the test set is used for testing to verify the performance of the model.
2. The method for space-ground cross-platform object re-identification based on semantic alignment and hint learning according to claim 1 is characterized by: In step 2, the semantic alignment network model includes four parts: a visual encoder, a text encoder, a learnable pedestrian description module, and a platform prompt module; the specific construction steps are as follows: Step 2-1: Visual encoder; The visual encoder includes a block encoder, a self-attention encoder, and a layer normalization module; The block encoder is a 16×16 convolution block with a step size of 16. It divides the input image into fixed 16x16 image blocks and converts the image blocks into block codes. Then, a position code is added to each image block vector. The block code and the position code are both vectors of the same dimension. The two vectors are added so that the model can distinguish image blocks at different positions. A class code is added and the class code and the image block vector are concatenated into a code sequence. Finally, the code sequence is sent to a multi-layer series self-attention encoder. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layer helps the model learn the global and local features of the image and encode the visual features into the class code. Step 2-2: Text encoder; The text encoder includes a text encoder, a self-attention encoder and a layer normalization module; The input of the text encoder is a piece of text information. First, the text encoder splits a piece of text into independent words and converts each word into a word code. Then, the word code and the class code are spliced into a word code sequence, and the word code sequence is sent to a multi-layer series of self-attention encoders. Each self-attention encoder contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layer collects semantic features in the word code sequence and encodes the semantic features into the class code. After each layer of self-attention encoder, the word code sequence will pass through the layer normalization module to achieve vector normalization. Step 2-3: Pedestrian description module can be learned; The learnable pedestrian description module includes a pedestrian description template and a learnable vector; The pedestrian description template and the learnable vector are concatenated into a complete pedestrian description text in the form of "A photo of [x1] ... [xm] perosn", where "A photo of ... person" is the pedestrian description template and "[x1] ... [xm]" is a learnable vector of length m. Step 2-4: Platform prompt module; The platform prompt module includes two sets of prompt vectors; The platform prompt module is regarded as changing the input of each self-attention encoder in the visual encoder, and the input vector of the self-attention encoder is represented as Where n = HW / P 2 , D represents the vector dimension, N represents the vector length, H and W are the length and width of the input image, and P is the step size of the block encoder in the visual encoder. After adding the visual cue module, the self-attention encoder is expressed as: in d and l are the depth and length of the platform hint vector, MSA k Represents the MSA of the kth layer in the visual encoder, Set ground and Set uav There are two collections, containing samples from ground platforms and drones respectively.
3. The method for space-ground cross-platform object re-identification based on semantic alignment and hint learning according to claim 1 is characterized by: In step 3, the image-text contrast loss of the loss function of the first training stage is defined as follows: in, Represents the text semantic features of the pedestrian with ID i, I p represents the pedestrian video image encoding with ID P, I a represents the pedestrian video image encoding with ID a, s() represents the cosine similarity, P(y i ) means including i All positive sample sets, B represents the training Batch Size; The triplet loss is defined as follows: Among them, θ is The soft margin, d p and d n are the characteristic distances of positive pairs and negative pairs respectively; The cross entropy loss is defined as follows: Among them, p k represents the number of ID predictions for category k, q k Distribute soft labels for targets; The visual-to-semantic contrast loss is defined as follows: Among them, V i represents the video visual features of the pedestrian with ID i, represents the text semantic features of the pedestrian with ID k, represents the text semantic features of the pedestrian with ID j, q k Distribute soft labels for targets; The overall first-stage loss function is expressed as: The overall second-stage loss function is expressed as: Among them, β, γ and ∈ represent relative importance weights.
4. The method for space-ground cross-platform object re-identification based on semantic alignment and hint learning according to claim 3 is characterized by: The first training stage uses the Adam optimizer to optimize the pedestrian independent vector in the learnable pedestrian module.
5. The method for space-ground cross-platform object re-identification based on semantic alignment and hint learning according to claim 1, characterized in that: The number of training epochs is 120.
6. The method for space-ground cross-platform object re-identification based on semantic alignment and hint learning according to claim 2 is characterized by: In the platform prompt module, the prompt vector length l=4, and the prompt vector depth d=9.
7. The method for space-ground cross-platform object re-identification based on semantic alignment and hint learning according to claim 1 is characterized by: The image size of the training set and the test set is 256×128, and each video clip contains 8 frames of images.
8. The method for space-ground cross-platform object re-identification based on semantic alignment and hint learning according to claim 3 is characterized by: The relative importance weights β, γ, and ∈ of the second stage loss function are 1.0, 0.25, and 1.0, respectively.
9. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Partial pedestrian re-identification method based on visible perception texture semantic alignment
CN111797813A
Pose-guided cross-attention feature fusion sheltered pedestrian re-recognition algorithm and device
CN116740642A
Text pedestrian re-identification method based on cross-modal semantic alignment
CN118072349A
Contrastive captioning neural networks
US20230351149A1
Cited By
Pedestrian re-identification method based on extensible representation learning
CN119693877A
Target re-identification method and system based on inter-block stepped TRANSFORMER
CN119723185A
Air-ground cooperative target identification method based on self-evolution visual cue learning
CN119785387A
Pedestrian re-identification method and related device
CN119832601A
A pedestrian re-identification method and related device
CN119832601B
