Air-ground cross-platform object re-identification method based on semantic alignment and hint learning

Abstract

This invention provides a cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning. It addresses the problem of visual feature misalignment across air and ground platforms by utilizing a large-scale visual language model. This model includes two branches: a visual encoder and a text encoder, as well as learnable pedestrian description and platform cue modules. The visual encoder is built based on a Vision Transformer self-attention encoder to extract global image information; the text encoder is built based on a Transformer self-attention encoder to encode text information. Platform cue vectors are introduced to train the image encoder to focus on pedestrian-invariant features in cross-platform air-to-ground images, achieving cross-platform feature alignment. This invention effectively solves the challenge of cross-platform visual feature misalignment in cross-platform video pedestrian re-identification tasks and effectively compensates for the problem of cross-platform visual image feature misalignment.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a method for solving the problem of severe misalignment of image features in cross-platform video target re-identification based on semantic alignment and cue learning. Background Technology

[0002] Video-based object re-identification (VReID) has garnered significant attention in recent years, but existing VReID research largely relies on data captured from the same platform (e.g., ground surveillance cameras). Imagine a suspect committing a crime in a city with numerous surveillance cameras and then fleeing to an area without them. A viable solution is to use aerial drones to capture the target's trajectory, thus creating a scenario requiring cross-platform (ground surveillance to aerial drone) video object re-identification. Cross-platform pedestrian re-identification faces a critical challenge: due to the significant difference in perspective between ground surveillance and drone cameras, there is severe feature misalignment between the images from the two platforms, leading to poor performance of traditional single-platform methods. Therefore, effectively aligning visual features across air and ground platforms is a key issue in cross-platform video pedestrian re-identification.

[0003] Traditional pedestrian re-identification methods primarily utilize block alignment to divide pedestrians into different regions and achieve block-level feature alignment. For example, Block Convolutional Mapping (PCB) horizontally segments the pedestrian image into different regions and aligns the features of corresponding regions in two images sequentially. BPBreID uses a keypoint detection network to extract pedestrian keypoint information, divides the image into different regions based on these keypoints, and then performs feature alignment on this basis. However, these methods have significant drawbacks. The PCB method is simple, but the region division is too coarse, making it only suitable for application scenarios where the perspective difference between a single platform is small. While BPBreID utilizes a more precise human keypoint detection network, from a drone's perspective, the overhead angle causes severe self-occlusion problems in pedestrian images (e.g., legs completely obscured by the upper body), making it difficult to effectively extract pedestrian keypoint information and thus hindering effective feature alignment.

[0004] Recently, with the emergence of Large-Scale Pre-Trained Visual Language Models (CLIP), a well-aligned visual semantic space can be obtained through cross-modal contrastive learning on large-scale network visual data and high-level language descriptions. Although in pedestrian re-identification tasks, each person's identity is represented only by an index number without linguistic description, a set of learnable text vectors can be introduced to roughly describe the semantic features of each pedestrian. Therefore, this paper proposes to extract cross-platform invariant semantic features of pedestrians by leveraging the basic visual language model CLIP. The model is efficiently constrained by aligning the dual-platform visual features with the cross-platform invariant semantic features. Furthermore, cue learning utilizes cue vectors to guide the model to focus on task differences. Therefore, based on semantic alignment, this invention introduces platform cue vectors to guide the model to acquire accurate and targeted platform information, thereby further bridging the gap in visual features. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning. It addresses the misalignment of visual features across air and ground platforms by utilizing a large-scale visual language model. This model comprises two branches: a visual encoder and a text encoder, as well as learnable pedestrian description and platform cue modules. The visual encoder is built based on a Vision Transformer self-attention encoder, whose main function is to extract global image information. The text encoder is built based on a Transformer self-attention encoder, whose main function is to encode text information. The model training consists of two stages. In the first stage, the visual encoder and text encoder load pre-trained model weights and freeze all parameters. Using image features from different platforms, cross-platform invariant semantic information of pedestrians is learned. In the second stage, the pedestrian text description and text encoder weights are frozen. The image features output by the image encoder are aligned with the cross-platform invariant semantic features of pedestrians. Simultaneously, platform cue vectors are introduced in the second stage to train the image encoder to focus on the pedestrian-invariant features of cross-platform images, achieving cross-platform feature alignment.

[0006] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0007] Step 1: Dataset preparation;

[0008] A large-scale cross-platform pedestrian re-identification dataset, named G2A-VReID, was collected using surveillance cameras and drones. The G2A-VReID dataset includes multiple scenes, with videos captured from six different locations: a library, an intersection, a bus stop, a school, a subway entrance, and a square. Ground-based surveillance cameras captured ground-view videos, while drones captured aerial videos. The surveillance cameras were fixed at a height of two meters above the ground, while the drones flew at varying heights from 20 to 60 meters, using hovering, cruising, and rotating flight techniques to capture people from different overhead angles. The raw video data was then labeled, with all people appearing in the videos marked with rectangular bounding boxes, and each person was cropped from the scene image based on the bounding boxes. Finally, manual labeling was performed, associating the same pedestrian from both the drone and surveillance videos and assigning a unique ID to each pedestrian. Next, all images of pedestrians with the same ID from the same video trajectory were merged into a single video trajectory. Each pedestrian had at least two trajectories: one from the surveillance camera and the other from the drone.

[0009] Step 2: Construct a semantic alignment network model;

[0010] Step 3: Establish the loss function;

[0011] The loss function includes a first training phase loss function and a second training phase loss function. The first training phase loss function includes image-text contrast loss, and the second training phase loss function includes triplet loss, cross-entropy loss, and image-text cross-entropy loss.

[0012] Step 4: Use the loss function The network model is trained in the first stage using the training set to learn the semantic description of pedestrians, employing a loss function. The semantic alignment network model is trained in the second stage. During the second stage of training, only all parameters in the visual encoder of the semantic alignment network model are adjusted. After each epoch of training, the model performance is verified on the test set of the dataset using the average precision (mAP) and the first matching accuracy (Rank-1) evaluation metrics. The mAP+Rank-1 value is recorded, and the model weights with the maximum mAP+Rank-1 value during the entire training process are saved as post-training parameters. After the model training cycle ends, the test set is used to test the model to verify its performance.

[0013] In step 2, the semantic alignment network model comprises four parts: a visual encoder, a text encoder, a learnable pedestrian description module, and a platform prompting module; the specific construction steps are as follows:

[0014] Step 2-1: Visual encoder;

[0015] The visual encoder includes a block encoder, a self-attention encoder, and a layer normalization module;

[0016] The block encoder is a 16×16 convolutional block with a stride of 16. It divides the input image into fixed 16x16 image blocks and converts the image blocks into block codes. Then, a positional code is added to each image block vector. Both the block code and the positional code are vectors of the same dimension. The two vectors are added together, enabling the model to distinguish image blocks at different locations. A class code is then added and concatenated with the image block vector to form an encoding sequence. Finally, the encoding sequence is fed into a multi-layer cascaded self-attention encoder. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layers help the model learn the global and local features of the image and encode the visual features into the class code.

[0017] Step 2-2: Text Encoder;

[0018] The text encoder includes a text encoder, a self-attention encoder, and a layer normalization module;

[0019] The input to the text encoder is a piece of text information. First, the text encoder splits the text into individual words and converts each word into a word code. Then, the word codes are concatenated with the class codes to form a word code sequence, which is then fed into a multi-layered cascaded self-attention encoder. Each self-attention encoder contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layer collects semantic features from the word code sequence and encodes the semantic features into the class code. After each self-attention encoder layer, the word code sequence undergoes a normalization module to achieve vector normalization.

[0020] Steps 2-3: Learn the pedestrian description module;

[0021] The learnable pedestrian description module includes pedestrian description templates and learnable vectors;

[0022] The pedestrian description template and the learnable vector are concatenated to form a complete pedestrian description text, which is in the form of "A photo of [x1]...[xm]person", where "A photo of...person" is the pedestrian description template and "[x1]...[xm]" is a learnable vector of length m.

[0023] Steps 2-4: Platform prompt module;

[0024] The platform prompt module includes two sets of prompt vectors;

[0025] The platform prompting module can be viewed as changing the input of each self-attention encoder in the visual encoder, and the input vector of the self-attention encoder can be represented as... Where N = HW / P 2 D represents the vector dimension, N represents the vector length, H and W are the length and width of the input image, and P is the stride of the block encoder in the visual encoder. After adding the visual cue module, the self-attention encoder is represented as:

[0026]

[0027] in d and l are the depth and length of the platform cue vector, MSA k Set represents the MSA of the k-th layer in the visual encoder. ground and Set uav There are two sets, one containing samples from ground platforms and the other from drones.

[0028] In step 3, the image-text contrast loss of the loss function in the first training phase is defined as follows:

[0029]

[0030] in, I represents the textual semantic features of the pedestrian with ID i. p This represents the pedestrian video image encoding with ID P, where I a Let s() represent the pedestrian video image encoding with ID a, and s() represent the cosine similarity. P(y i ) indicates that it includes y i The set of all positive samples, where B represents the training batch size;

[0031] The triplet loss is defined as follows:

[0032]

[0033] Where θ is soft margin, d p and d n These are the feature distances of the positive pair and the feature distances of the negative pair, respectively.

[0034] The cross-entropy loss is defined as follows:

[0035]

[0036] Where, p k q represents the number of predicted ID logs for category k. k Soft labels are distributed for the target distribution;

[0037] The visual-to-semantic contrast loss is defined as follows:

[0038]

[0039] Among them, V i This represents the video visual features of the pedestrian with ID i. The textual semantic features representing the pedestrian with ID k. q represents the textual semantic features of the pedestrian with ID j. k Soft labels are distributed for the target distribution;

[0040] The overall first-stage loss function is expressed as:

[0041]

[0042] The overall second-stage loss function is expressed as:

[0043]

[0044] Where β, γ, and ∈ represent relative importance weights.

[0045] The first training phase uses the Adam optimizer to optimize the pedestrian-independent vectors in the learnable pedestrian module.

[0046] Preferably, the number of training epochs is 120.

[0047] Preferably, in the platform prompt module, the prompt vector length l = 4 and the prompt vector depth d = 9.

[0048] Preferably, the training and test set images are 256×128 in size, and each video segment contains 8 frames.

[0049] Preferably, the relative importance weights β, γ, and ∈ of the loss function in the second stage are 1.0, 0.25, and 1.0, respectively.

[0050] An electronic device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.

[0051] A computer-readable storage medium storing program code that can be invoked by a processor to perform the method described above.

[0052] The beneficial effect of this invention lies in its adoption of a semantic alignment and cue learning approach, effectively addressing the challenge of misaligned visual features across platforms in cross-platform video person re-identification tasks. The model of this invention employs a carefully designed learnable person description module. In the first stage, through comparative training, the learnable person description module learns cross-viewpoint person-invariant semantic features of pedestrians. The second stage aligns the cross-platform visual features of pedestrians in both air and ground with the person-invariant semantic features, thereby resolving the problem of severe misalignment of visual features in cross-platform video person re-identification. On the G2A-VreID dataset, traditional methods perform poorly due to cross-platform visual feature misalignment; for example, SINet only achieves a mean accuracy (mAP) of 74.5%. This invention, by aligning the cross-platform visual features of pedestrians in both air and ground with the person-invariant semantic features, effectively compensates for the problem of misalignment of cross-platform visual image features, achieving a mean accuracy (mAP) of 76.5% on G2A-VreID.

[0053] This invention proposes a platform cueing module, which is introduced in the second stage of model training. The module guides the model's visual encoder to pay attention to platform differences through platform-independent cueing vectors, thereby further bridging the gap between platforms. Attached Figure Description

[0054] Figure 1 This is a flowchart of the training process of the method of the present invention.

[0055] Figure 2 This is a diagram of the network model structure constructed by the method of this invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and examples.

[0057] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a cross-platform video pedestrian re-identification method based on visual semantic alignment and cue learning. This method addresses the misalignment of visual features across air and ground platforms by utilizing a large-scale visual language model. The model comprises two branches: a visual encoder and a text encoder, as well as learnable pedestrian descriptions and platform cues. The visual encoder is built upon a Vision Transformer self-attention encoder, whose main function is to extract global image information. The text encoder is built upon a Transformer self-attention encoder, whose main function is to encode text information. The model training consists of two stages. In the first stage, the visual encoder and text encoder load pre-trained model weights and freeze all parameters. Using image features from different platforms, the method learns cross-platform invariant semantic information of pedestrians. In the second stage, the pedestrian text description and text encoder weights are frozen. The image features output by the image encoder are aligned with the cross-platform invariant semantic features of pedestrians. Simultaneously, platform cue vectors are introduced in the second stage to train the image encoder to focus on the cross-platform pedestrian features, thus achieving cross-platform feature alignment.

[0058] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0059] Step 1: Dataset preparation;

[0060] A large-scale, cross-platform pedestrian re-identification dataset, named G2A-VReID, was collected using surveillance cameras and drones. This dataset includes 185,907 images, 5,576 video tracks, and 2,788 pedestrians with distinct IDs. To enrich the data, the G2A-VReID dataset includes diverse scenes, with videos captured from six different locations: a library, intersection, bus stop, school, subway entrance, and square. Ground-based surveillance cameras captured ground-view videos, while DJI Mavic drones captured aerial videos. Specifically, the surveillance cameras were fixed at a height of two meters above the ground, while the drones flew at varying altitudes from 20 to 60 meters. The drones hovered, cruised, and rotated to capture individuals from different overhead perspectives. The collected raw video data was then annotated. During annotation, all individuals appearing in the video were marked with rectangular bounding boxes, and each person was cropped from the scene image based on their bounding box. Finally, data labelers manually labeled the data, associating the same pedestrians from both the drone and surveillance videos and assigning a unique ID to each pedestrian. Next, all images belonging to the same pedestrian ID from the same video trajectory were merged into a single video trajectory, ensuring each person had at least two trajectories: one from the surveillance camera and the other from the drone. Finally, the dataset was randomly divided into training and test sets in a 1:2 ratio. Therefore, the test set contained 1858 pedestrians with different IDs, and the training set contained 930 pedestrians with different IDs.

[0061] Step 2: Construct a semantic alignment network model;

[0062] The semantic alignment network model comprises four parts: a visual encoder, a text encoder, a learnable pedestrian description module, and a platform prompting module; the specific construction steps are as follows:

[0063] Step 2-1: Visual encoder;

[0064] The visual encoder includes a block encoder, a self-attention encoder, and a layer normalization module;

[0065] The block encoder uses 16×16 convolutional blocks with a stride of 16 to divide the input image into fixed 16x16 image blocks, converting each block into a vector representation. Then, a positional encoding is added to each image block vector, enabling the model to distinguish image blocks at different locations. A class encoding is also added, and this class encoding is concatenated with the image block vector to form an encoded sequence. Finally, the encoded sequence is fed into a multi-layered, cascaded self-attention encoder. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layers help the model learn global and local features of the image and encode visual features into the class encoding.

[0066] Step 2-2: Text Encoder;

[0067] The text encoder includes a text encoder, a self-attention encoder, and a layer normalization module;

[0068] The text encoder takes a piece of text as input. First, it breaks the text down into individual words and converts each word into a word code. Then, the word codes are concatenated with class codes to form a word code sequence, which is then fed into a multi-layered, cascaded self-attention encoder. Each self-attention encoder contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layers collect semantic features from the word code sequence and encode these features into the class code. After each self-attention encoder layer, the word code sequence undergoes a normalization module to achieve vector normalization.

[0069] Steps 2-3: Learn the pedestrian description module;

[0070] The learnable pedestrian description module includes pedestrian description templates and learnable vectors;

[0071] The pedestrian description template and the learnable vector are concatenated to form a complete pedestrian description text, which is in the form of "A photo of [x1]...[xm]perosn", where "A photo of person" is the pedestrian description template and "[x1]...[xm]" is a learnable vector of length m.

[0072] Steps 2-4: Platform prompt module;

[0073] The module consists of two sets of prompt vectors.

[0074] Using the platform suggestion module can be viewed as changing the input of each self-attention encoder in the visual encoder, representing the input vector of the self-attention encoder as follows: Where N = HW / P 2 D represents the vector dimension, N represents the vector length, H and W are the length and width of the input image, and P is the stride of the block encoder in the visual encoder. After adding the visual cue module, the self-attention encoder is represented as:

[0075]

[0076] in d and l are the depth and length of the platform cue vector, MSA k Set represents the MSA of the k-th layer in the visual encoder. ground and Set uav There are two sets, one containing samples from ground platforms and the other from drones;

[0077] Step 3: Establish the loss function;

[0078] The loss function includes a first training phase loss function and a second training phase loss function. The first training phase loss function includes image-text contrast loss, and the second training phase loss function includes triplet loss, cross-entropy loss, and image-text cross-entropy loss.

[0079] The image-text contrast loss function for the first training phase is defined as follows:

[0080]

[0081] in, I represents the textual semantic features of the pedestrian with ID i. p Let P represent the pedestrian video image encoding with ID P, s() represent the cosine similarity, and P(y i ) indicates that it includes y i The set of all positive samples, where B represents the training batch size;

[0082] The triplet loss is defined as follows:

[0083]

[0084] Where θ is soft margin, d p and d n It is the feature distance between positive and negative pairs;

[0085] The cross-entropy loss is defined as follows:

[0086]

[0087] Where, p k The number of predicted IDs for category k;

[0088] The visual-to-semantic contrast loss is defined as follows:

[0089]

[0090] Among them, V i This represents the video visual features of the pedestrian with ID i. q represents the textual semantic features of the pedestrian with ID i. k Target distribution soft labels;

[0091] The overall first-stage loss function is expressed as:

[0092]

[0093] The overall second-stage loss function is expressed as:

[0094]

[0095] Where β, γ, and ∈ represent relative importance weights, with β = 1.0, γ = 0.25, and ∈ = 1.0;

[0096] Step 4: Use the loss function The network model is trained in the first stage using the training set to learn the semantic description of pedestrians, employing a loss function. The semantic alignment network model is then trained in the second stage using the training set. During this second stage, only the parameters of the visual encoder in the semantic alignment network model are adjusted. After the model completes its preset training cycle, it is tested using the test set to verify its performance. Specific implementation examples:

[0098] (1) Dataset preparation:

[0099] The specific steps are as follows, using panchromatic images collected by ground cameras and aerial drones:

[0100] ① Dataset Collection: A large-scale, cross-platform pedestrian re-identification dataset was collected using surveillance cameras and drones to enrich the data and make it more realistic. Video capture scenarios included nine different locations such as libraries, intersections, bus stops, and tourist attractions. Ground-based surveillance cameras captured ground-view videos, while DJI Mavic drones captured aerial videos. Specifically, the surveillance cameras were fixed at a height of approximately two meters above the ground, while the drones flew at varying heights from 20 to 60 meters. The drones flew in hovering, cruising, and rotating modes, allowing for a richer range of perspectives in the captured figures.

[0101] ② Dataset annotation: During the annotation process, all people appearing in the video are labeled with bounding boxes, and each person is cropped from the scene image based on the bounding box. The drone and the same person in the surveillance video are associated and assigned a unique ID. Next, all images of one person from a single camera are merged into a single trajectory. This ensures that each person has at least two trajectories, one from the surveillance camera and the other from the drone. The dataset contains 2778 pedestrians with different IDs, and each pedestrian has two video clips from different platforms, with an average of 33.3 frames per video clip.

[0102] ③ Dataset partitioning: The training and test sets are randomly divided in a 1:2 ratio. Therefore, the test set contains 1858 pedestrians with different IDs, and the training set contains 930 pedestrians with different IDs.

[0103] ④ Randomly extract 8 frames from the original video clip, and scale each frame to 256×128. Apply random flipping, random mosaic and normalization to a set of video images, and finally convert them into PyTorch Tensor data format.

[0104] The dataset preparation steps are now complete.

[0105] (2) Constructing a complementary and integrated network model

[0106] Network structure diagram as follows Figure 2 As shown, the parameters for constructing the network are:

[0107] The image encoder uses ViT-B-16 with a vector length of 768, and the video encoder also uses ViT-B-16 with a sequence length of 512. The pedestrian learnable module has a pedestrian-independent learnable vector length of 4. The platform cueing module has a length of 4 and a depth of 9.

[0108] (3) Network training;

[0109] Input image: a full-color video clip V, with dimensions of 8×256×128 (video frame length × height × width). A set of video images is processed by random flipping, random mosaicking, and normalization, and finally converted into the data format of PyTorch Tensor, a deep learning tensor library.

[0110] Other relevant settings: In the PyTorch framework, the Adam optimizer is used to update parameters in both training phases. The initial model weights are the CLIP original pre-trained weights. In the first training phase, all weights of the image encoder and text encoder are frozen, and the Adam optimizer is used to optimize the pedestrian-independent vectors in the learnable pedestrian module. The number of training epochs is set to 120, the batch size is set to 256, and the initial learning rate is set to 0.000035. In the second training phase, the model's initial weights are the weights of the model after the first phase of training. All parameters in the learnable pedestrian description module and text encoder are frozen, and the Adam optimizer is used to optimize the image encoder. The number of training epochs is set to 120, the batch size is set to 32, and the initial learning rate is set to 0.000005. The learning rate decays by 90% at epochs 50 and 90. After each epoch of training, the model performance is validated on the test set using the mean accuracy (mAP) and first matching accuracy (Rank-1) metrics, and the mAP + Rank-1 value is recorded. After training, the model weights at which the mAP+Rank-1 value is maximized during the entire training process are saved.

[0111] The condition for stopping training is: completing 120 rounds of training.

[0112] (4) Network Testing

[0113] ① Input image: Video clip V, size 8×256×128 (video frame length×height×width). A set of video images are processed by random flipping, random mosaic and normalization in sequence. Finally, the video clip V is converted into PyTensor data format.

[0114] ② Load the network parameters saved at the end of the second training phase when the mAP+Rank-1 value was maximized, and verify the model performance on the test set data. The test set is divided into a query set and a candidate set. The query set contains N pedestrian images, and each image corresponds to K images in the candidate set. The model uses a visual encoder to encode any image in the candidate set and all images in its corresponding candidate set into a set of vectors. Cosine similarity is used to measure the similarity between the candidate set images and the query set images, and they are sorted from high to low similarity. If, for a given query set image, the image with the highest similarity in the candidate set has a different ID than the query set image, the recognition is correct. Otherwise, if, for a given query set image, the image with the highest similarity in the candidate set has a different ID than the query set image, the recognition is incorrect.

[0115] ③ Network output results: The final output results are presented using the mean accuracy (mAP) and rank-1 accuracy as evaluation metrics.

[0116] Experiments show that the performance of this method on the G2A-VreID dataset is further improved after introducing the platform hint module. See Table 3 for details.

[0117] Table 1. Performance comparison between the method of this invention and different classical methods.

[0118]

[0119]

[0120] This invention achieves an average accuracy (mAP) of 78.1% in G2A-VreID.

[0121] Compared to classical methods, this invention, based on visual semantic alignment and platform cue learning, effectively solves the problem of visual feature misalignment in cross-platform person re-identification, achieving better results. Compared to PiT, this invention improves the mean accuracy (mAP) by 1.8% and the first-match accuracy (Rank-1) by 1.6%.

Claims

1. A cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning, characterized in that... Includes the following steps: Step 1: Dataset preparation; A large-scale cross-platform pedestrian re-identification dataset, named G2A-VReID, was collected using surveillance cameras and drones. The G2A-VReID dataset includes multiple scenes, with videos captured from six different locations: a library, an intersection, a bus stop, a school, a subway entrance, and a square. Ground-based surveillance cameras captured ground-view videos, while drones captured aerial videos. The surveillance cameras were fixed at a height of two meters above the ground, while the drones flew at varying heights from 20 to 60 meters, using hovering, cruising, and rotating flight techniques to capture people from different overhead angles. The raw video data was then labeled, with all individuals appearing in the videos marked with rectangular bounding boxes, and each person was cropped from the scene image based on these bounding boxes. Finally, manual labeling was performed, associating the same pedestrian from both the drone and surveillance videos and assigning a unique ID to each pedestrian. Next, all images of pedestrians with the same ID from the same video trajectory were merged into a single video trajectory, with each pedestrian having at least two trajectories: one from the surveillance camera and the other from the drone. Step 2: Construct a semantic alignment network model; Step 3: Establish the loss function; The loss function includes a first training phase loss function and a second training phase loss function. The first training phase loss function includes image-text contrast loss, and the second training phase loss function includes triplet loss, cross-entropy loss, and image-text cross-entropy loss. In step 3, the image-text contrast loss of the loss function in the first training phase is defined as follows: in, I represents the textual semantic features of the pedestrian with ID i. p This represents the pedestrian video image encoding with ID P, where I a Let s() represent the pedestrian video image encoding with ID a, and s() represent the cosine similarity. P(y i ) indicates that it includes y i The set of all positive samples, where B represents the training batch size; The triplet loss is defined as follows: Where θ is soft margin, d p and d n These are the feature distances of the positive pair and the feature distances of the negative pair, respectively. The cross-entropy loss is defined as follows: Where, p k q represents the number of predicted ID logs for category k. k Soft labels are distributed for the target distribution; The visual-to-semantic contrast loss is defined as follows: Among them, V i This represents the video visual features of the pedestrian with ID i. The textual semantic features representing the pedestrian with ID k. q represents the textual semantic features of the pedestrian with ID j. k Soft labels are distributed for the target distribution; The overall first-stage loss function is expressed as: The overall second-stage loss function is expressed as: Where β, γ, and ∈ represent relative importance weights; Step 4: Use the loss function The network model is trained in the first stage using the training set to learn the semantic description of pedestrians, employing a loss function. The semantic alignment network model is trained in the second stage. During the second stage of training, only all parameters in the visual encoder of the semantic alignment network model are adjusted. After each epoch of training, the model performance is verified on the test set of the dataset using the mean accuracy (mAP) and the first matching accuracy (Rank-1) evaluation metrics. The mAP+Rank-1 value is recorded, and the model weights with the maximum mAP+Rank-1 value during the entire training process are saved as post-training parameters. After the model training cycle ends, the test set is used to test the model to verify its performance.

2. The cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning according to claim 1, characterized in that: In step 2, the semantic alignment network model comprises four parts: a visual encoder, a text encoder, a learnable pedestrian description module, and a platform prompting module; the specific construction steps are as follows: Step 2-1: Visual encoder; The visual encoder includes a block encoder, a self-attention encoder, and a layer normalization module; The block encoder is a 16×16 convolutional block with a stride of 16. It divides the input image into fixed 16x16 image blocks and converts the image blocks into block codes. Then, a positional code is added to each image block vector. Both the block code and the positional code are vectors of the same dimension. The two vectors are added together, enabling the model to distinguish image blocks at different locations. A class code is then added and concatenated with the image block vector to form an encoding sequence. Finally, the encoding sequence is fed into a multi-layer cascaded self-attention encoder. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layers help the model learn the global and local features of the image and encode the visual features into the class code. Step 2-2: Text Encoder; The text encoder includes a text encoder, a self-attention encoder, and a layer normalization module; The input to the text encoder is a piece of text information. First, the text encoder splits the text into individual words and converts each word into a word code. Then, the word codes are concatenated with the class codes to form a word code sequence, which is then fed into a multi-layered cascaded self-attention encoder. Each self-attention encoder contains a multi-head self-attention mechanism and a feedforward neural network. The encoder layer collects semantic features from the word code sequence and encodes the semantic features into the class code. After each self-attention encoder layer, the word code sequence undergoes a normalization module to achieve vector normalization. Steps 2-3: Learn the pedestrian description module; The learnable pedestrian description module includes pedestrian description templates and learnable vectors; The pedestrian description template and the learnable vector are concatenated to form a complete pedestrian description text, which is in the form of "A photo of [x1]…[xm]person", where "A photo of…person" is the pedestrian description template. "[x1]…[xm]" is a learnable vector of length m; Steps 2-4: Platform prompt module; The platform prompt module includes two sets of prompt vectors; The platform prompting module can be viewed as changing the input of each self-attention encoder in the visual encoder, and the input vector of the self-attention encoder can be represented as... Where N = HW / P 2 D represents the vector dimension, N represents the vector length, H and W are the length and width of the input image, and P is the stride of the block encoder in the visual encoder. After adding the visual cue module, the self-attention encoder is represented as: in d and l are the depth and length of the platform cue vector, MSA k Set represents the MSA of the k-th layer in the visual encoder. ground and Set uav There are two sets, one containing samples from ground platforms and the other from drones.

3. The cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning according to claim 1, characterized in that: The first training phase uses the Adam optimizer to optimize the pedestrian-independent vectors in the learnable pedestrian module.

4. The cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning according to claim 1, characterized in that: The number of training epochs is set to 120.

5. The cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning according to claim 2, characterized in that: In the platform prompt module, the prompt vector length l = 4 and the prompt vector depth d = 9.

6. The cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning according to claim 1, characterized in that: The training and test sets have an image size of 256×128, and each video segment contains 8 frames.

7. The cross-platform air-to-ground target re-identification method based on semantic alignment and cue learning according to claim 1, characterized in that: The relative importance weights β, γ, and ∈ of the loss function in the second stage are 1.0, 0.25, and 1.0, respectively.

8. An electronic device, characterized in that, include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-7.

Images