Cross-modal pedestrian re-identification method, system and product using unpaired data

Abstract

The invention discloses a cross-modal pedestrian re-identification method and system using unpaired data and a product, and the method comprises the steps: firstly obtaining a pedestrian image data set in a multi-modal scene, which comprises a visible light modal image and an infrared modal image; and then extracting visible light image and infrared image features by using an image feature extraction network to obtain a representation vector, calculating a similarity metric value with the existing representation vector, and based on a Top-K sorting strategy, selecting the most similar sample as a matching result, thereby completing cross-modal pedestrian identity recognition without pairing data. According to the method, the accuracy of pedestrian re-identification from visible light to infrared light is improved, effective alignment and feature mapping between visible light and infrared images can be realized under the condition of no pairing data, the personnel identification and tracking capability in a multi-modal monitoring environment is improved, and the intelligent level of a security and protection system is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and machine learning technologies, and in particular to an unsupervised cross-modal pedestrian re-identification method, system, and product. Specifically, the present invention relates to a cross-modal pedestrian re-identification method, system, and product using unpaired data, aiming to achieve pedestrian re-identification between visible light and infrared images using unpaired data. Background Art

[0002] Person re-identification (PRED) aims to identify and match images of the same person across multiple surveillance cameras. Traditional PRED methods are primarily based on visible light images, which have significant limitations in low-light or nighttime environments. Therefore, cross-modal PRED based on infrared images has become an important research direction.

[0003] The invention patent with publication number CN119863823A is a method for visible light-infrared pedestrian re-identification based on hypergraph-driven soft semantic dynamic learning. The method divides pedestrian images into training and test sets and performs data augmentation. The method uses a self-attention mechanism to extract pedestrian hierarchical soft semantic features. The method uses a gating mechanism to fuse cross-stage features in channel and spatial directions. The method calculates the modal alignment loss to reduce the difference between visible light and infrared features. The method uses a hypergraph neural network to dynamically explore the relationship between the extracted soft semantics. The method uses a joint loss to train the network until convergence. The method re-identifies pedestrians on the test set. By using this invention, not only can the hierarchical soft semantic features be effectively captured and dynamically integrated, modality-invariant features can be extracted, and the gap between visible light and infrared modalities can be bridged, but also the high-order relationship between semantics can be adaptively explored. The method effectively learns modality-invariant semantics, reduces the difference between modalities, and significantly improves the performance of visible light-infrared pedestrian re-identification.

[0004] The invention patent with publication number CN117392705A is a method for visible light and infrared pedestrian re-identification based on local feature optimization. The method collects a target area image, which includes a visible light modality image and an infrared modality image; inputs the target area image into a pedestrian recognition model to obtain a pedestrian identity recognition result, wherein the pedestrian recognition model includes a local feature extraction module, a local feature alignment module and a feature collaborative learning module. The local feature extraction module extracts local features of the visible light modality and the infrared modality, the local feature alignment module groups and aligns cross-modal local features to establish cross-modal connections, and the feature collaborative learning module enhances each local feature and jointly optimizes global features and local features. The present invention can adaptively mine cross-modal local features, directly establish rich alignment relationships for cross-modal local features, embed the contextual relationship of global features into local features, and significantly improve the accuracy of pedestrian recognition.

[0005] The invention patent with publication number CN117934309B: A method for fusion of unaligned infrared-visible images based on modal dictionary and feature matching, which mainly solves the current difficulty in fusing unaligned infrared-visible image pairs. First, the fixed image and the offset image are input into the feature extraction module to obtain fixed features and offset features; the fixed features and offset features are input into the cross-modal representation learning module to obtain uniformly represented fixed features and offset features; the uniformly represented fixed features and offset features are input into the feature matching module to output a matching matrix; the matching matrix and offset features are input into the feature recombination block, and the offset features are recombined using the matching matrix; the recombined features and fixed features are input into the fusion module to reconstruct the infrared-visible fused image. The present invention can correct the spatial misalignment of infrared-visible image pairs and eliminate artifacts and offsets in the fusion results, thereby improving fusion performance.

[0006] However, existing visible-to-infrared person re-identification methods either fail to meet the unsupervised requirements or rely on generating pseudo-labels to bridge identity correspondences between different modalities. These methods often rely on paired data of visible and infrared images, which requires expensive and time-consuming manual annotation, increasing costs and making them unsuitable for real-world scenarios where unpaired data is unavailable. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, and product for cross-modal person re-identification using unpaired data. These methods improve the robustness of cross-modal feature representation by generating pseudo-labels and aligning modalities in the absence of paired data. Simultaneously, they perform static-dynamic collaborative learning, effectively addressing the drawbacks of prior art techniques that rely on paired data and enabling efficient cross-modal person re-identification without paired data.

[0008] The technical solution adopted by the method of the present invention is: a cross-modal pedestrian re-identification method using unpaired data, comprising the following steps: Step 1: Obtain pedestrian image data in a multimodal scene, including images of visible light and infrared modalities; Step 2: Use the image feature extraction network to extract features from visible light images and infrared images to obtain their modality-independent representation vectors. Utilize the similarity measurement between the representation vectors and the pedestrian feature vectors extracted from the image library, and based on the Top-K sorting strategy, select the most similar samples as the matching results, thereby completing cross-modal pedestrian identification without paired data.

[0009] Preferably, in step 2, the image feature extraction network includes an image embedding module, a position encoding module, a side information embedding module, a self-attention encoder module, a puzzle feature extraction module, a multi-branch feature integration module and a supervised learning module; The image embedding module includes an image segmentation unit and a linear projection unit for segmenting the input image; the image segmentation unit divides the input image into a plurality of P×P block sequences, and the linear projection unit performs a vectorization operation on each block and concatenates all block features with a learnable classification token to form a feature sequence; The position encoding module is used to introduce position information into the feature sequence output by the image embedding module, adopt a learnable position embedding table, and adapt to different input image sizes through bilinear interpolation; The side information embedding module is used to encode the camera information and the view information into a fixed-length vector representation, wherein the camera information vector and the view information vector are concatenated to form a side information embedding vector, which is added to the output feature of the image embedding module and then input into the next module; The self-attention encoder module includes a multi-layer Transformer encoding block, each of which includes a multi-head self-attention mechanism, a feedforward network, a layer normalization unit, and a residual connection unit in sequence; and is used to perform global feature modeling on the feature sequence processed by the image embedding module, the position encoding module, and the side information embedding module; The jigsaw puzzle feature extraction module is used to disrupt the block sequence and divide it into k subgroups. After adding a shared classification token to each subgroup, each subgroup is input into a shared Transformer module to obtain k local features. The multi-branch feature integration module is used to splice the global feature with k local features; The supervised learning module includes a classifier branch and a feature distance metric branch for the training phase. The classifier branch outputs the identity category probability distribution, and the distance metric branch is used to construct positive and negative sample pairs and calculate triplet loss, optimize the feature space distribution, and improve the model's discriminability.

[0010] Preferably, the feature extraction network in step 2 is a trained network; the training process includes the following sub-steps: Step 2.1: Obtain pedestrian image data in several multimodal scenes, including visible light and infrared images, and construct a training dataset. Step 2.2: Use the image feature extraction network to extract visible light image and infrared image features; Step 2.3: Using a cross-modal feature mapping module, the features of one modality are converted into features of the other modality to generate pseudo paired modality features. The cross-modal feature mapping module includes two modality-specific mappers, each of which estimates the mean and variance of the feature distribution under the corresponding modality and uses these two statistics to generate pseudo cross-modal features of the other modality. Step 2.4: Use static learning and dynamic learning strategies to align pseudo-paired features with real features at the cluster level and instance level, and input them into the network for training; During the training process, clustering is first performed to generate pseudo labels, where the infrared feature and visible light feature representation of each cluster are stored in the infrared and visible light memory banks respectively; then they are transformed into corresponding pseudo cross-modal features through cross-modal feature mapping; the total loss used in the training process consists of static learning loss and dynamic learning loss, and is adjusted by loss weights. , balancing the two loss functions; among them, the actual infrared loss, actual visible light loss, pseudo infrared loss and pseudo visible light loss are calculated through the momentum update strategy, which together constitute the static learning loss; two types of cross-modal label preservation losses are calculated, which are the differences between the infrared modality and the visible light modality and the corresponding pseudo label features, respectively, to force the pseudo cross-modal features to be close to their corresponding actual features, and the two together constitute the dynamic learning loss; backpropagation and optimization of algorithm parameters, cyclic iteration finally obtain the optimal parameters.

[0011] As a preference, in step 2.3, a cross-modal feature mapping module is used to generate pseudo paired modality features. ; ; Among them, Mapping is a modal The following features are converted into modes The transformation function of the distribution, is the original modal feature; The characteristic distribution of each mode obeys the mode-specific Gaussian distribution, which is expressed as Indicates that Denotes the infrared or visible light modality as; each mapper estimates the mean and variance of the feature distribution under the corresponding modality:

[0012] in is the total number of samples in the Gaussian distribution, is the characteristic of each sample; and are the mode-specific mean and variance, respectively.

[0013] And use these two statistics to generate pseudo cross-modal features of another modality, Switch to another mode The pseudo cross-modal features of are:

[0014] in, and is modal The science parameters of the mapper, is a small parameter used to ensure numerical stability.

[0015] Preferably, at the beginning of each training cycle, all infrared features and visible light characteristics First, clustering is performed to generate pseudo labels, where the infrared and the visible light characteristics are expressed as and stored in infrared and visible light memory banks;

[0016] in Is the first in infrared or visible light cluster sets, represents the number of instances in each cluster; Then the actual features and Transformed into corresponding pseudo cross-modal features through the cross-modal feature mapping module and , but the identity remains unchanged, and the cluster of pseudo cross-modal features is represented as and ,constructing two additional memory banks in the cross-modal pseudo-identity space;

[0017] in Indicates the first cluster sets, Indicates the number of instances in each cluster.

[0018] Preferably, the momentum update strategy is to sample P person identities from each modality training set during training, and sample Z instances for each identity; a total of A query containing infrared and visible light images of people is used to update the actual memory using the momentum update strategy:

[0019]

[0020] At the same time, the memory of pseudo cross-modal features is also updated through a similar momentum strategy:

[0021]

[0022] in is the momentum update factor, is the number of iterations, is the characteristic of the visible light mode, It is the characteristic of infrared mode.

[0023] Preferably, the static learning loss , including actual infrared losses ,, actual visible light loss , pseudo infrared loss and pseudo-visible light loss ;

[0024]

[0025]

[0026]

[0027] in and are the positive representation vectors of the actual infrared and visible light clusters corresponding to the queried pseudo labels, and are the positive eigenvectors of pseudo-infrared and visible light clusters, respectively; is the temperature hyperparameter; is the characteristic of the visible light mode, is the characteristic of infrared mode, is the visible light modal cluster number, is the number of infrared modal clusters.

[0028] Preferably, the dynamic learning loss is

[0029]

[0030]

[0031] in and are the number of clusters of small batch data in infrared and visible light modalities, respectively; is a given infrared signature The most difficult visible sample to detect false positives, is the hardest negative sample in the cth pseudo-visible sample cluster in the mini-batch; is a given visible light characteristic The most difficult visible sample to detect false positives, It is the hardest negative sample in the cth pseudo-visible sample cluster in the mini-batch; is the temperature hyperparameter.

[0032] Preferably, the similarity measurement formula in step 2 is:

[0033] in is the cosine similarity function, is the representation vector of the image to be recognized, is the pedestrian feature vector extracted from the gallery, Represents the similarity between two images.

[0034] The technical solution adopted by the system of the present invention is: a cross-modal pedestrian re-identification system using unpaired data, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the cross-modal pedestrian re-identification method using unpaired data.

[0035] The technical solution adopted by the product of the present invention is: a cross-modal pedestrian re-identification product using unpaired data, including computer program instructions, characterized in that: when the computer program instructions are run on a computer, the computer executes the cross-modal pedestrian re-identification method using unpaired data.

[0036] Compared with the prior art, the beneficial effects of the present invention include: (1) Traditional methods usually rely on paired data of visible light and infrared images, which requires a large amount of manual annotation, which is costly and time-consuming. This paper adopts an unsupervised learning strategy to achieve cross-modal person re-identification without paired data by generating pseudo-labels and feature mapping modules, avoiding the reliance on manual annotation and reducing the cost and complexity of data preparation. (2) Traditional methods have certain limitations in cross-modal feature alignment, which may lead to large differences in feature distribution between modalities, thus affecting recognition performance. This paper designs a cross-modal feature mapping module, which achieves effective feature alignment between different modalities by analyzing the differences between visible light and infrared images in feature space, thereby improving the accuracy of cross-modal recognition; (3) Traditional methods have certain deficiencies in processing image features, resulting in poor adaptability of the model in different scenarios. This paper proposes a static and dynamic collaborative learning strategy. Static learning uses multiple memory banks to store feature clustering results under different modalities to obtain a global identity representation across modalities; dynamic learning further eliminates the differences between modalities through instance-level alignment, ensuring that the generated pseudo features are close to the real features in the feature space, thereby improving the adaptability and robustness of the model in different scenarios.

[0037] Compared with the existing technology, the present invention can significantly improve the accuracy and robustness of cross-modal re-identification without relying on expensive labeled data, especially in practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The technical solution of the present invention is further illustrated below using embodiments and specific implementation methods. In addition, some drawings are also used in the process of illustrating the technical solution. For those skilled in the art, other drawings and the intention of the present invention can be obtained based on these drawings without making any creative efforts.

[0039] Figure 1 This is a general principle framework diagram of the method according to an embodiment of the present invention; Figure 2 This is a diagram of the image feature extraction network structure according to an embodiment of the present invention; Figure 3 This is a flowchart of the image feature extraction network training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0041] In public safety management, pedestrian re-identification technology can be used to track the identities and analyze the behaviors of people in the surveillance area. Figure 1 This embodiment provides a cross-modal person re-identification method using unpaired data, comprising the following steps: Step 1: Obtain a pedestrian image dataset in a multimodal scene, including images of visible light and infrared modalities; Step 2: Use the image feature extraction network to extract features from visible light images and infrared images to obtain their modality-independent representation vectors. Utilize the similarity measurement between the representation vectors and the pedestrian feature vectors extracted from the image library, and based on the Top-K sorting strategy, select the most similar samples as the matching results, thereby completing cross-modal pedestrian identification without paired data.

[0042] Please see Figure 2 In one embodiment, the image feature extraction network includes an image embedding module, a position encoding module, a side information embedding module, a self-attention encoder module, a puzzle feature extraction module, a multi-branch feature integration module and a supervised learning module; The image embedding module includes an image segmentation unit and a linear projection unit for segmenting the input image. The image segmentation unit divides the input image into a number of P×P patches. The linear projection unit performs a vectorization operation on each patch and concatenates all patch features with a learnable classification token to form a feature sequence. The position encoding module is used to introduce position information into the feature sequence output by the image embedding module, adopt a learnable position embedding table, and adapt to different input image sizes through bilinear interpolation; The side information embedding module is used to encode the camera information and the view information into a fixed-length vector representation, wherein the camera information embedding and the view information embedding are concatenated to form a side information embedding vector, which is added to the output feature of the image embedding module and then input into the next module; The self-attention encoder module is used to globally model the feature sequence after the image embedding, position encoding and side information embedding processing, including a multi-layer Transformer encoding block, each encoding block sequentially including a multi-head self-attention mechanism, a feedforward network, a layer normalization unit and a residual connection unit; The jigsaw feature extraction module is used to enhance the local detail modeling capability. The jigsaw feature extraction module disrupts the patch sequence and divides it into k subgroups. After adding a shared classification token to each subgroup, each subgroup is input into a shared Transformer module to obtain k local features. The local features are used together with the global features for feature fusion; The multi-branch feature integration module is used to splice the global feature with k local features and perform joint learning, calculate the cross entropy loss and triple loss through multiple feature channels, supervise each channel separately and calculate the weighted average to obtain the total loss, which is used to optimize the feature expression ability; The supervised learning module includes a classifier branch and a feature distance metric branch for the training phase. The classifier branch outputs the identity category probability distribution, and the distance metric branch is used to construct positive and negative sample pairs and calculate triplet loss, optimize the feature space distribution, and improve the model's discriminability.

[0043] In one embodiment, the feature extraction network is a trained network; the training process includes the following sub-steps: Step 2.1: Obtain pedestrian image data in several multimodal scenes, including visible light and infrared images, and construct a training dataset. Step 2.2: Use the image feature extraction network to extract visible light image and infrared image features; Step 2.3: Using a cross-modal feature mapping module, the features of one modality are converted into features of the other modality to generate pseudo paired modality features. The cross-modal feature mapping module includes two modality-specific mappers, each of which estimates the mean and variance of the feature distribution under the corresponding modality and uses these two statistics to generate pseudo cross-modal features of the other modality. In one embodiment, a cross-modal feature mapping module is used to generate pseudo paired modality features. ; ; Among them, Mapping is a modal The following features are converted into modes The transformation function of the distribution, is the original modal feature; since human images from different modalities have different styles, these styles are modality-specific, this embodiment assumes that the feature distribution of each modality obeys the modality-specific Gaussian distribution, and uses Indicates that To achieve this conversion, this embodiment introduces a cross-modal feature mapping module, which contains two modality-specific mappers, each of which estimates the mean and variance of the feature distribution under the corresponding modality:

[0044] in is the total number of samples in the Gaussian distribution, is the characteristic of each sample; and are the mode-specific mean and variance, respectively.

[0045] And use these two statistics to generate pseudo cross-modal features of another modality, Switch to another mode The pseudo cross-modal features of are:

[0046] in, and is modal The science parameters of the mapper, is a very small parameter used to ensure numerical stability. Through this transformation, the features of the visible light / infrared mode are mapped to the distribution of infrared / visible light, generating corresponding pseudo cross-modal features. and All from the same image , so they share the same identity information, from which we construct the original modality-specific space and the cross-modal pseudo-identity space for further feature alignment.

[0047] Step 2.4: Use static learning and dynamic learning strategies to align pseudo-paired features with real features at the cluster level and instance level, and input them into the network for training; During the training process, clustering is first performed to generate pseudo labels, where the infrared feature and visible light feature representation of each cluster are stored in the infrared and visible light memory banks respectively; then they are transformed into corresponding pseudo cross-modal features through cross-modal feature mapping; the total loss used in the training process consists of static learning loss and dynamic learning loss, and is adjusted by loss weights. , balancing the two loss functions; among them, the actual infrared loss, actual visible light loss, pseudo infrared loss and pseudo visible light loss are calculated through the momentum update strategy, which together constitute the static learning loss; two types of cross-modal label preservation losses are calculated, which are the differences between the infrared modality and the visible light modality and the corresponding pseudo label features, respectively, to force the pseudo cross-modal features to be close to their corresponding actual features, and the two together constitute the dynamic learning loss; backpropagation and optimization of algorithm parameters, cyclic iteration finally obtain the optimal parameters.

[0048] In one embodiment, at the beginning of each training cycle, all infrared signatures and visible light characteristics First, clustering is performed to generate pseudo labels, where the infrared and the visible light characteristics are expressed as and stored in infrared and visible light memory banks;

[0049] in Is the first in infrared or visible light cluster sets, represents the number of instances in each cluster; Then the actual features and Transformed into corresponding pseudo cross-modal features through the cross-modal feature mapping module and , but the identity remains unchanged, so it is important to note that the cluster number remains the same before and after mapping. Therefore, the cluster representation of the pseudo cross-modal feature is and ,constructing two additional memory banks in the cross-modal pseudo-identity space;

[0050] in Indicates the first cluster sets, Indicates the number of instances in each cluster.

[0051] In one embodiment, the momentum update strategy is to sample P person identities from each modality training set during training, and sample Z instances for each identity; a total of A query containing infrared and visible light images of people is used to update the actual memory using the momentum update strategy:

[0052]

[0053] At the same time, the memory of pseudo cross-modal features is also updated through a similar momentum strategy:

[0054]

[0055] in is the momentum update factor, is the number of iterations, is the characteristic of the visible light mode, It is the characteristic of infrared mode.

[0056] In one embodiment, the static learning loss , including actual infrared losses ,, actual visible light loss , pseudo infrared loss and pseudo-visible light loss ;

[0057]

[0058]

[0059]

[0060] in and are the positive representation vectors of the actual infrared and visible light clusters corresponding to the queried pseudo labels, and are the positive eigenvectors of pseudo-infrared and visible light clusters, respectively; is the temperature hyperparameter; is the characteristic of the visible light mode, is the characteristic of infrared mode, is the visible light modal cluster number, is the number of infrared modal clusters.

[0061] The dynamic learning loss is

[0062]

[0063]

[0064] in and are the number of clusters of small batch data in infrared and visible light modalities, respectively; is a given infrared signature The most difficult visible sample to detect false positives, is the hardest negative sample in the cth pseudo-visible sample cluster in the mini-batch; is a given visible light characteristic The most difficult visible sample to detect false positives, It is the hardest negative sample in the cth pseudo-visible sample cluster in the mini-batch; is the temperature hyperparameter.

[0065] The total learning loss consists of static learning loss and dynamic learning loss:

[0066] in is the loss weight, balancing the two loss functions.

[0067] In the inference phase, the visible light or infrared modality image to be identified is input into the trained feature extraction network to obtain its modality-independent representation vector. The similarity metric between the representation vector and the pedestrian feature vector extracted from the image library is used:

[0068] in is the cosine similarity function, is the representation vector of the image to be identified, is the pedestrian feature vector extracted from the gallery, Represents the similarity between two images. Based on the Top-K sorting strategy, the most similar samples are selected as the matching results, thus completing cross-modal pedestrian identification without paired data.

[0069] The present invention is further described below through specific experiments.

[0070] To validate the proposed framework in a paired setting, this experiment conducts a comprehensive evaluation on two benchmark datasets for visible-infrared cross-modal person re-identification: SYSU-MM01 and RegDB. To assess the model's effectiveness in an unpaired setting, this experiment modifies the SYSU-MM01 and RegDB datasets. It is important to note that existing datasets lack cross-modal unpaired training data, necessitating adjustments to the original benchmarks. To systematically analyze the impact of the unpaired setting, a hyperparameter 𝛼 is introduced to control the proportion of identity overlaps between the visible and infrared modalities. This approach preserves the full data size without reducing the data size, thereby mitigating the impact of data reduction.

[0071] For the paired and unpaired settings of the SYSU-MM01, this experiment used two evaluation modes: all-search and indoor-search. For RegDB, the performance of the method was evaluated in two test modes: infrared to visible light and visible light to infrared. The test strictly followed the existing method, performing 10 gallery subset partitions and taking the average result. The framework was implemented on the PyTorch platform. This experiment is based on Augmented Dual-Contrastive Aggregation (ADCA) and uses the feature extraction network of TransReID as the backbone. In each mini-batch, 16 identities are sampled for each modality, and each identity contains 16 images. During training, the images are uniformly resized to 288 × 144 pixels. Standard data augmentation techniques such as random cropping, random flipping, and random erasing are used. Table 1 below shows the experimental results under different experimental environments. Tables 2 and 3 show the experimental results of the latest methods using paired data on the SYSU-MM01 and RegDB datasets, respectively.

[0072] Table 1

[0073] Table 2

[0074] Table 3

[0075] To validate the effectiveness of our approach, we conducted a comprehensive comparison with state-of-the-art methods in both unpaired and paired settings. The experimental results in Tables 1, 2, and 3 demonstrate that our approach significantly outperforms existing unsupervised methods at various ratios in both unpaired and paired settings. This is due to two key advantages: 1) Our approach effectively addresses the lack of cross-modal correlation in unpaired scenarios by synthesizing pseudo-paired data while preserving discriminative identities. 2) Our collaborative learning strategy implements a two-layer alignment paradigm for both cluster-level and instance-level optimization.

[0076] This paper proposes a cross-modal visible-to-infrared person re-identification method that utilizes unpaired data. By leveraging technologies such as cross-modal feature mapping and static-dynamic collaborative learning, it addresses the high cost and unsuitability of existing technologies due to their reliance on paired data. This method improves the accuracy of visible-to-infrared person re-identification. It also enables effective alignment and feature mapping between visible and infrared images without paired data, enhancing the ability to identify and track people in multimodal surveillance environments and strengthening the intelligence of security systems. This invention has significant technical and social significance, playing a key role in locating target individuals at night or in low-light environments.

[0077] It should be understood that the embodiments described above are only some of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention may be arbitrarily combined with each other to form a feasible technical solution. Such combination is not restricted by the order of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually inconsistent or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0078] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A cross-modal person re-identification method using unpaired data, characterized in that: The following steps are involved: Step 1: Obtain pedestrian image data in a multimodal scene, including images of visible light and infrared modalities; Step 2: Use the image feature extraction network to extract features from visible light images and infrared images to obtain their modality-independent representation vectors. Utilize the similarity measurement between the representation vectors and the pedestrian feature vectors extracted from the image library, and based on the Top-K sorting strategy, select the most similar samples as the matching results, thereby completing cross-modal pedestrian identification without paired data.

2. The cross-modal person re-identification method using unpaired data according to claim 1, characterized in that: In step 2, the image feature extraction network includes an image embedding module, a position encoding module, a side information embedding module, a self-attention encoder module, a puzzle feature extraction module, a multi-branch feature integration module and a supervised learning module; The image embedding module includes an image segmentation unit and a linear projection unit for segmenting the input image; the image segmentation unit divides the input image into a plurality of P×P block sequences, and the linear projection unit performs a vectorization operation on each block and concatenates all block features with a learnable classification token to form a feature sequence; The position encoding module is used to introduce position information into the feature sequence output by the image embedding module, adopt a learnable position embedding table, and adapt to different input image sizes through bilinear interpolation; The side information embedding module is used to encode the camera information and the view information into a fixed-length vector representation, wherein the camera information vector and the view information vector are concatenated to form a side information embedding vector, which is added to the output feature of the image embedding module and then input into the next module; The self-attention encoder module includes a multi-layer Transformer encoding block, each of which includes a multi-head self-attention mechanism, a feedforward network, a layer normalization unit, and a residual connection unit in sequence; and is used to perform global feature modeling on the feature sequence processed by the image embedding module, the position encoding module, and the side information embedding module; The jigsaw puzzle feature extraction module is used to disrupt the block sequence and divide it into k subgroups. After adding a shared classification token to each subgroup, it is input into the shared Transformer module to obtain k local features. The multi-branch feature integration module is used to combine the global feature with k local features. The supervised learning module includes a classifier branch and a feature distance metric branch for the training phase. The classifier branch outputs the identity category probability distribution, and the distance metric branch is used to construct positive and negative sample pairs and calculate triplet loss, optimize the feature space distribution, and improve the model's discriminability.

3. The cross-modal person re-identification method using unpaired data according to claim 1 or 2, characterized in that: The feature extraction network described in step 2 is a trained network; The training process consists of the following sub-steps: Step 2.1: Obtain pedestrian image data in several multimodal scenes, including visible light and infrared images, and construct a training dataset. Step 2.2: Use the image feature extraction network to extract visible light image and infrared image features; Step 2.3: Using a cross-modal feature mapping module, the features of one modality are converted into features of the other modality to generate pseudo paired modality features. The cross-modal feature mapping module includes two modality-specific mappers, each of which estimates the mean and variance of the feature distribution under the corresponding modality and uses these two statistics to generate pseudo cross-modal features of the other modality. Step 2.4: Use static learning and dynamic learning strategies to align pseudo-paired features with real features at the cluster level and instance level, and input them into the network for training; During the training process, clustering is first performed to generate pseudo labels, where the infrared feature and visible light feature representation of each cluster are stored in the infrared and visible light memory banks respectively; then, they are transformed into corresponding pseudo cross-modal features through cross-modal feature mapping; The total loss used in the training process consists of static learning loss and dynamic learning loss, and is calculated by loss weight , balancing the two loss functions; among them, the actual infrared loss, actual visible light loss, pseudo infrared loss and pseudo visible light loss are calculated through the momentum update strategy, which together constitute the static learning loss; two types of cross-modal label preservation losses are calculated, which are the differences between the infrared modality and the visible light modality and the corresponding pseudo label features, respectively, to force the pseudo cross-modal features to be close to their corresponding actual features, and the two together constitute the dynamic learning loss; backpropagation and optimization of algorithm parameters, cyclic iteration finally obtain the optimal parameters.

4. The cross-modal person re-identification method using unpaired data according to claim 3, characterized in that: In step 2.3, the cross-modal feature mapping module is used to generate pseudo paired modality features ; ； Among them, Mapping is a modal The following features are converted into modes The transformation function of the distribution, is the original modal feature; The characteristic distribution of each mode obeys the mode-specific Gaussian distribution, which is expressed as Indicates that represents the infrared or visible light modality; each mapper estimates the mean and variance of the feature distribution under the corresponding modality: in is the total number of samples in the Gaussian distribution, is the characteristic of each sample; and are the mode-specific mean and variance, respectively; And use these two statistics to generate pseudo cross-modal features of another modality, Switch to another mode The pseudo cross-modal features of are: in, and is modal The science parameters of the mapper, is a small parameter used to ensure numerical stability.

5. The cross-modal person re-identification method using unpaired data according to claim 3, characterized in that: At the beginning of each training cycle, all infrared features and visible light characteristics First, clustering is performed to generate pseudo labels, where the infrared and the visible light characteristics are expressed as and stored in infrared and visible light memory banks; in Is the first in infrared or visible light cluster sets, represents the number of instances in each cluster; Then the actual features and Transformed into corresponding pseudo cross-modal features through the cross-modal feature mapping module and , but the identity remains unchanged, and the cluster of pseudo cross-modal features is represented as and ,constructing two additional memory banks in the cross-modal pseudo-identity space; in Indicates the first cluster sets, Indicates the number of instances in each cluster.

6. The cross-modal person re-identification method using unpaired data according to claim 5, characterized in that: The momentum update strategy is to sample P person identities from each modality training set during training, and sample Z instances for each identity; Get a total of A query containing infrared and visible light images of people is used to update the actual memory using the momentum update strategy: At the same time, the memory of pseudo cross-modal features is also updated through a similar momentum strategy: in is the momentum update factor, is the number of iterations, is the characteristic of the visible light mode, It is the characteristic of infrared mode.

7. The cross-modal person re-identification method using unpaired data according to claim 3, characterized in that: The static learning loss , including actual infrared losses ,, actual visible light loss , pseudo infrared loss and pseudo-visible light loss ; in and are the positive representation vectors of the actual infrared and visible light clusters corresponding to the queried pseudo labels, and are the positive eigenvectors of pseudo-infrared and visible light clusters, respectively; is the temperature hyperparameter; is the characteristic of the visible light mode, is the characteristic of infrared mode, is the visible light modal cluster number, is the number of infrared modal clusters.

8. The cross-modal person re-identification method using unpaired data according to claim 3, characterized in that: The dynamic learning loss is in and are the number of clusters of small batch data in infrared and visible light modalities, respectively; is a given infrared signature The most difficult visible sample to detect false positives, is the hardest negative sample in the cth pseudo-visible sample cluster in the mini-batch; is a given visible light characteristic The most difficult visible sample to detect false positives, It is the hardest negative sample in the cth pseudo-visible sample cluster in the mini-batch; is the temperature hyperparameter.

9. A cross-modal person re-identification system using unpaired data, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the cross-modal pedestrian re-identification method using unpaired data as described in any one of claims 1 to 8.

10. A cross-modal person re-identification product using unpaired data, comprising computer program instructions, characterized in that: When the computer program instructions are executed on a computer, the computer is caused to perform the cross-modal person re-identification method using unpaired data according to any one of claims 1 to 8.

Images