Cross-modal person re-identification method and system based on semantic alignment and relational reasoning

Abstract

The present invention proposes a cross-modal pedestrian re-identification method and system based on semantic alignment and similarity reasoning, which belongs to the field of computer vision technology. First, the cross-modal data set is preprocessed, and the data diversity is increased by using data augmentation technology. Then, the classification loss function, the center separation loss function and the prototype diversity loss function are used to train the feature extraction deep convolutional network. The trained deep convolutional neural network is used to extract features from pedestrian images, and semantic alignment is used to achieve alignment of potential semantic local features. Finally, similarity reasoning technology is used to calculate the distance between images based on the semantically aligned feature vectors to obtain the final matching result. The present invention effectively solves the problem of semantic misalignment of feature vectors caused by pedestrian posture misalignment, and is simple to implement, high in performance and strong in robustness. The present invention fully exploits the similarity relationship between images and optimizes distance calculation, which is conducive to achieving cross-modal pedestrian re-identification with high accuracy and high stability.

Description

Technical Field

[0001] The present invention relates to a cross-modal pedestrian re-identification method and system based on semantic alignment and similarity reasoning, and belongs to the field of computer vision technology. Background Art

[0002] In the field of computer vision, person re-identification technology uses images of pedestrians captured in different scenarios to identify and match their identities. This technology is widely used in video surveillance, public security, and traffic flow monitoring.

[0003] Under well-lit conditions, a surveillance system's visible light camera can capture clear images and match pedestrian information. However, in low-light environments, such as those at night under city lights, moonlight, and starlight, a surveillance system's visible light camera struggles to capture clear images, making visible light cameras alone inadequate for all-weather surveillance. To capture clear images even in low-light conditions, modern surveillance equipment is equipped with infrared cameras to replace visible light cameras when light intensity is low. Consequently, the surveillance system captures images in two different modalities: visible light and infrared. Conventional person re-identification techniques fail to account for the significant modal differences between visible light and infrared images, making it difficult to meet the requirements of cross-modal person matching.

[0004] Cross-modal person re-identification (CRID) technology aims to eliminate the modal differences between visible light and infrared images, thereby enabling matching and recognition of images from different modalities. Due to the significant differences between visible light and infrared images, direct feature matching is difficult to achieve effective results. Currently, mainstream CRID methods are based on deep learning and deep convolutional neural networks, using horizontal partitioning of feature maps to construct local feature vectors for matching. However, due to the uncertainty of pedestrian pose, the local feature vectors generated by horizontal partitioning suffer from semantic misalignment, making them difficult to meet practical application requirements.

[0005] Semantic alignment is a technique for aligning extracted semantic features. This technique leverages the semantic information in images to convert pedestrian features from different modalities into feature vectors in the same semantic space, thereby reducing the differences between images of different modalities and improving the accuracy and reliability of cross-modal person re-identification. Latent semantic local feature alignment utilizes deep learning techniques to adaptively extract latent semantic features from feature maps. These latent semantic features can be features of certain parts of the human body, such as important local features like the face, limbs, and torso. Semantically aligned local features are of great application value in cross-modal person re-identification, effectively resolving the misalignment problem caused by pedestrian posture and improving the accuracy and robustness of cross-modal person re-identification technology.

[0006] Similarity measurement is a technique for calculating the similarity or distance between two objects. This technique uses the feature vectors or feature descriptors of the targets to be matched to calculate the similarity or distance between them. This technique is an important technology in machine learning and data mining, widely used in clustering, classification, recommendation, and retrieval, and has high practical value and application potential. Currently, in the field of cross-modal pedestrian re-identification, mainstream similarity measurement methods do not consider the potential relationship between images and still use Euclidean distance or cosine distance for measurement. Euclidean distance refers to the distance between two points in Euclidean space. Cosine distance uses the angle between two vectors to measure their similarity. These two methods are simple and intuitive and have been widely used in the field of computer vision.

[0007] However, the above methods regard images as independent individuals and ignore the relationship between images, resulting in the matching accuracy of the cross-modal pedestrian re-identification system being unable to meet actual needs. Summary of the Invention

[0008] The purpose of this invention is to start from the needs of cross-modal pedestrian re-identification, and to address the defects and shortcomings of the existing technology such as semantic misalignment and inaccurate similarity measurement, and creatively propose a cross-modal pedestrian re-identification method and system based on semantic alignment and similarity reasoning.

[0009] Similarity reasoning, which adaptively corrects the distance between images based on their interrelationships, can effectively improve the matching accuracy and robustness of cross-modal person re-identification systems. This invention achieves fast, high-performance, end-to-end semantic alignment of cross-modal pedestrian features, addressing the technical issue of semantic misalignment of feature vectors caused by factors such as shooting angle, pedestrian posture, and detection errors. Furthermore, a similarity reasoning method is designed and implemented based on the similarity relationship between images, improving the accuracy and stability of cross-modal person re-identification.

[0010] The present invention is implemented by adopting the following technical solutions.

[0011] A cross-modal person re-identification method based on semantic alignment and similarity reasoning includes the following steps:

[0012] Step 1: Preprocess the cross-modal dataset and use data augmentation techniques to increase data diversity.

[0013] Step 2: Train the feature extraction deep convolutional network using the classification loss function, center separation loss function, and prototype diversity loss function.

[0014] Step 3: Use the trained deep convolutional neural network to extract features from pedestrian images. Use semantic alignment technology to align latent semantic local features.

[0015] Step 4: Use similarity reasoning technology to calculate the distance between images based on the semantically aligned feature vectors to obtain the final matching result.

[0016] On the other hand, in order to achieve the objectives stated in the present invention, the present invention also proposes a cross-modal pedestrian re-identification system based on semantic alignment and similarity reasoning, which includes a feature extraction module, a semantic information alignment module, and a similarity reasoning module.

[0017] Beneficial effects

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. The present invention can effectively solve the problem of semantic misalignment of feature vectors caused by pedestrian posture misalignment, and is simple to implement, high in performance, and strong in robustness.

[0020] 2. The present invention can fully exploit the similarity relationship between images and optimize distance calculation, which is conducive to achieving high-accuracy and high-stability cross-modal pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the method of the present invention.

[0022] Figure 2 It is a comparison diagram of the method of the present invention and the existing method.

[0023] Figure 3 It is a schematic diagram of feature extraction, semantic alignment and similarity reasoning of the method of the present invention.

[0024] Figure 4 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0025] In order to better illustrate the purpose and advantages of the present invention, the present invention is further described below with reference to the accompanying drawings.

[0026] like Figure 1 As shown in FIG, a cross-modal person re-identification method based on semantic alignment and similarity reasoning includes the following steps:

[0027] Step 1: Preprocess the cross-modal dataset and use data augmentation techniques to increase data diversity.

[0028] Currently, publicly available cross-modal person re-identification datasets are relatively small. Using raw data directly for training can easily lead to overfitting, resulting in networks failing to meet generalization requirements. Therefore, this paper utilizes multiple data augmentation techniques, first augmenting the raw data to expand the training data, increase data diversity, and enhance the model's generalization and performance.

[0029] Specifically, first, the random cropping technique is used to crop different areas of the original image to generate training samples of different sizes and positions, simulate pedestrian images of different scales and positions, and reduce the impact of different sizes and positions of pedestrians.

[0030] Then, using the left-right symmetry of pedestrians, the pedestrian images are randomly flipped horizontally to generate new training data.

[0031] Then, using random color transformation technology, we randomly transform the pedestrian images, adjusting brightness, contrast, and saturation to increase data diversity and mitigate the impact of image color differences. At the same time, we can also randomly erase some pixels, forcing the network to focus on more features.

[0032] Finally, part of the visible light image (for example, 50%) is converted into a grayscale image, and the generated grayscale image is used as an infrared image for training to increase the diversity of infrared images.

[0033] Through the above processing, diverse training samples can be generated to help the model better learn and identify features, enhance the model's generalization ability and robustness, and improve the model's accuracy and performance.

[0034] Step 2: Train the feature extraction deep convolutional network using the classification loss function, center separation loss function, and prototype diversity loss function.

[0035] Specifically, the total loss function L(θ) is expressed as:

[0036]

[0037] Among them, L ID 、L CS and L PD Represent the classification loss function, center separation loss function and prototype diversity loss respectively; represents the aligned semantic features; y represents the identity of the image; θ is the model parameter; c is the center feature; α is the loss function weight; P represents the learnable semantic prototype; T represents the pixel feature on the feature map.

[0038] You can use L ID Further expressed as:

[0039]

[0040] Among them, L SH_ID Use a classifier to simultaneously predict the identity information of two modalities and construct cross entropy loss; L SP_ID Use two classifiers to predict the identity information of each modality respectively and construct cross entropy loss; L CM_IDThe prediction results of the classifier are used as soft labels and the loss is constructed using KL divergence.

[0041] L ID By evaluating the gap between the classification model's prediction results and the true labels, the model parameters are updated, guiding the network to focus on features that help identify pedestrians and enhancing the network's feature discovery capabilities.

[0042] You can use L CS Further expressed as:

[0043]

[0044]

[0045] Where N and M represent the number of images and the number of pedestrian identities in the current mini-batch training, respectively; ρ1 and ρ2 represent the maximum distance from the feature vector of an image to its corresponding central feature vector and the minimum distance between different central feature vectors, respectively; The feature vector representing the i-th semantic alignment; Indicates identity y i The central eigenvector of Indicates identity y k The central eigenvector of .

[0046] L CS The first term of L aggregates feature vectors belonging to the same identity into the range of their feature center ρ1. Compared with the ordinary center loss function, this scheme can effectively increase the diversity of features and solve the problem of overfitting of the network. CS The second term pushes the feature centers belonging to different identities away from each other, making the network focus on features that are helpful in distinguishing pedestrian identities.

[0047] You can use L PD Further expressed as:

[0048]

[0049] Where p represents the number of learnable semantic prototypes. PD It can guide the network to focus on different semantic features, fully explore the potential feature information in the image, and enhance the network's ability to explore potential semantic features.

[0050] During the entire training process, no additional processing is required, and the cross-modal pedestrian re-identification deep neural network is trained in an end-to-end manner in one go.

[0051] Step 3: Use the trained deep convolutional neural network to extract features from pedestrian images. Use semantic alignment technology to align latent semantic local features.

[0052] Horizontally segmenting feature maps to achieve semantic alignment of local features is well-suited for processing aligned images. However, in practice, obtaining aligned images is difficult due to various factors, including changes in shooting angle, varying pedestrian postures, and unstable detection algorithms. Using horizontal segmentation on non-aligned images also makes it difficult to achieve effective alignment of local features, which can easily lead to matching errors.

[0053] To solve this problem, this method first extracts features from the image to obtain a two-dimensional feature map of the image, and then divides the feature map into pixels to obtain a pixel-level feature vector.

[0054] Furthermore, in order to explore the latent semantic local features in the feature map, this method designs a learnable prototype, uses the learnable prototype as a reference for the latent semantic local features, and calculates the similarity matrix between the pixel-level features and the learnable prototype: S = σ(PT T ), where S represents the obtained similarity information and σ(·) represents the Sigmoid function, which is used to ensure the stability of the numerical value during calculation.

[0055] The obtained similarity matrix is ​​used to weight and aggregate pixel-level features to form a complete latent semantic local feature:

[0056]

[0057] Among them, p i represents the i-th semantically aligned local feature, n represents the number of pixel-level features; S ij represents the similarity between the i-th learnable prototype and the j-th pixel-level feature; Represents element-wise multiplication; T j Represents the j-th pixel-level feature space on the feature map.

[0058] To align the semantic information between the two modalities, the two modalities share a common set of learnable prototypes. These shared learnable prototypes consider the features of both the infrared and visible light modalities, learning common features between the two modalities. As a result, the feature vectors of the two modalities, obtained by aggregating the learnable prototypes, are effectively aligned.

[0059] Using the learnable prototype as a reference, the latent semantic local features are extracted. At this point, the overall information of the feature map is still lacking. Therefore, the average pooling technique is further used to extract the global features of the feature map, and the latent semantic local features and global features are combined into the final feature vector

[0060]

[0061] Where F represents the feature map; avg(·) represents average pooling; and [·] represents the concatenated feature vector.

[0062] Step 4: Use the similarity inference algorithm to calculate the similarity relationship between images based on the semantically aligned feature vectors, and obtain the final matching result based on the similarity relationship.

[0063] Specifically, to optimize matching by leveraging relationships between images, we first calculate the similarity matrix within the database image and the similarity matrix between the query image and the database images. To mitigate interference from low-similarity information, we retain only the top k values ​​in the similarity matrix. Furthermore, to improve the robustness of the similarity matrix, we calculate a new similarity matrix using the mean of the image's nearest neighbors. Finally, we use the similarity matrix to recalculate the distance between the query image and each database image to complete the matching.

[0064] Compared with the traditional method, the present invention optimizes the semantic alignment module and similarity reasoning module, such as Figure 2 As shown in the figure, the present invention optimizes the semantic alignment module from the current mainstream method of horizontally dividing local features to a method of adaptively discovering local features and aligning them through the network. At the same time, it optimizes the similarity reasoning module and optimizes the derivation process by using the similarity relationship between images. The detailed structure of the semantic alignment module and the similarity derivation mode is shown in the figure. Figure 3 As shown in the figure, after the shared backbone network extracts features, the feature map is fed into the semantic alignment module, which uses learnable prototypes to discover latent semantic local features in the feature map. During inference, the relationship between semantically aligned features is used to calculate a similarity matrix, which is then used to calculate the distance between images, thereby enhancing the robustness of the matching.

[0065] On the other hand, in order to achieve the purpose of the present invention, the present invention further proposes a cross-modal person re-identification system based on semantic alignment and similarity reasoning, which includes a feature map extraction module 10, a semantic alignment module 20 and a similarity reasoning module 30. Figure 4 shown.

[0066] The feature map extraction module 10 uses a deep convolutional neural network to extract feature maps of visible light images and infrared images. This module represents the input image as a feature map containing high-level semantic information.

[0067] The semantic alignment module 20 extracts latent semantic local features from the feature map to address the issue of pedestrian posture misalignment between images. This module uses a learnable prototype as a reference for the latent semantic local features. It calculates the similarity between each pixel feature in the feature map and the learnable prototype, then weights the latent semantic local features to achieve semantic alignment of pedestrian features.

[0068] The similarity reasoning module 30 uses the similarity matrix between images to represent the relationship between images, and corrects the original distance relationship based on the similarity matrix to achieve robust distance calculation and improve matching accuracy.

[0069] The connection relationship between the above modules is as follows:

[0070] The output end of the feature map extraction module 10 is connected to the input end of the semantic alignment module 20 ; the output end of the semantic alignment module 20 is connected to the input end of the similarity reasoning module 30 .

Claims

1. A cross-modal person re-identification method based on semantic alignment and similarity reasoning, characterized by: The following steps are involved: Step 1: Preprocess the cross-modal dataset and use data augmentation techniques to increase data diversity; Step 2: Train the feature extraction deep convolutional network using the classification loss function, center separation loss function, and prototype diversity loss function; The total loss function L(θ) is expressed as: Among them, L ID 、L CS and L PD Represent the classification loss function, center separation loss function and prototype diversity loss respectively; represents the aligned semantic features; y represents the identity of the image; θ is the model parameter; c is the center feature; α is the loss function weight; P represents the learnable semantic prototype; T represents the pixel feature on the feature map; L ID Further expressed as: Among them, L SH_ID Use a classifier to simultaneously predict the identity information of two modalities and construct cross entropy loss; L SP_ID Use two classifiers to predict the identity information of each modality respectively and construct cross entropy loss; L CM_ID The classifier's prediction results are used as soft labels, and the loss is constructed using KL divergence; L ID By evaluating the gap between the classification model's predictions and the true labels, the model's parameters are updated, guiding the network to focus on features that help identify pedestrians and enhancing the network's feature discovery capabilities. L CS Further expressed as: Where N and M represent the number of images and the number of pedestrian identities in the current mini-batch training, respectively; ρ1 and ρ2 represent the maximum distance from the feature vector of an image to its corresponding central feature vector and the minimum distance between different central feature vectors, respectively; The feature vector representing the i-th semantic alignment; Indicates identity y i The central eigenvector of Indicates identity y k The central eigenvector of L CS The first term aggregates the feature vectors belonging to the same identity into the range of their feature center ρ1; CS The second term pushes the feature centers belonging to different identities away from each other, so that the network focuses on features that are helpful in distinguishing pedestrian identities; L PD Further expressed as: Where p represents the number of learnable semantic prototypes; L PD Guide the network to focus on different semantic features, fully explore the potential feature information in the image, and enhance the network's ability to explore potential semantic features; During the entire training process, no additional processing is required, and the cross-modal pedestrian re-identification deep neural network is trained in an end-to-end manner in one go. Step 3: Use the trained deep convolutional neural network to extract features from pedestrian images; use semantic alignment technology to align latent semantic local features; Step 4: Use similarity reasoning technology to calculate the distance between images based on the semantically aligned feature vectors to obtain the final matching result.

2. The cross-modal person re-identification method based on semantic alignment and similarity reasoning according to claim 1, characterized in that: Step 1 includes: First, we use random cropping techniques to crop different regions of the original image to generate training samples of different sizes and positions, simulating pedestrian images of different scales and positions, and reducing the impact of different pedestrian sizes and positions. Then, using the left-right symmetry of pedestrians, the pedestrian images are randomly flipped horizontally to generate new training data; Then, random color transformation technology is used to perform random color transformation on pedestrian images, adjusting brightness, contrast and saturation to increase data diversity and reduce the impact of different image colors; Finally, part of the visible light image is converted into grayscale images, and the generated grayscale images are used as infrared images for training to increase the diversity of infrared images.

3. The cross-modal person re-identification method based on semantic alignment and similarity reasoning according to claim 2, characterized in that: Randomly erase some pixels to force the network to focus on more features.

4. The cross-modal person re-identification method based on semantic alignment and similarity reasoning according to claim 1, characterized in that: In step 3, the image is firstly subjected to feature extraction to obtain a two-dimensional feature map of the image, and then the feature map is divided into pixels to obtain a pixel-level feature vector; In order to explore the latent semantic local features in the feature map, this paper designs a learnable prototype. The learnable prototype is used as a reference for the latent semantic local features and the similarity matrix between the pixel-level features and the learnable prototype is calculated: Where S represents the obtained similarity information, and σ(·) represents the Sigmoid function, which is used to ensure the stability of the numerical value during calculation; The obtained similarity matrix is ​​used to weight and aggregate pixel-level features to form a complete latent semantic local feature: Among them, p i represents the i-th semantically aligned local feature, n represents the number of pixel-level features; S ij represents the similarity between the i-th learnable prototype and the j-th pixel-level feature; Represents element-wise multiplication; T j Represents the j-th pixel-level feature space on the feature map; To align the semantic information between the two different modalities, the two modalities share the same set of learnable prototypes. The shared learnable prototypes consider the features of both the infrared modality and the visible light modality to learn the features shared by both modalities. The average pooling technique is used to extract the global features of the feature map, and the latent semantic local features and global features are combined into the final feature vector Where F represents the feature map; avg(·) represents average pooling; and [·] represents the concatenated feature vector.

5. The cross-modal person re-identification method based on semantic alignment and similarity reasoning according to claim 1, characterized in that: In step 4, the similarity matrix within the database image and the similarity matrix between the query image and the database image are first calculated, and only the first k values ​​in the similarity matrix are retained; at the same time, a new similarity matrix is ​​calculated using the mean of the image's nearest neighbors; finally, the distance between the query image and each database image is recalculated using the similarity matrix to complete the matching.

6. A cross-modal person re-identification system based on semantic alignment and similarity reasoning using the method of claim 1, characterized in that: It includes a feature map extraction module 10, a semantic alignment module 20 and a similarity reasoning module 30; The feature map extraction module 10 uses a deep convolutional neural network to extract feature maps of visible light images and infrared images; this module represents the input image as a feature map containing high-level semantic information; Semantic alignment module 20 is used to extract latent semantic local features from the feature map to address the problem of pedestrian posture misalignment between images. This module uses a learnable prototype as a reference for the latent semantic local features and calculates the similarity between each pixel feature in the feature map and the learnable prototype, thereby weighting the latent semantic local features to achieve alignment of pedestrian features at the semantic level. The similarity reasoning module 30 uses the similarity matrix between images to represent the relationship between images, and corrects the original distance relationship by the similarity matrix to achieve robust distance calculation and improve matching accuracy; The connection relationship between the above modules is as follows: The output end of the feature map extraction module 10 is connected to the input end of the semantic alignment module 20 ; the output end of the semantic alignment module 20 is connected to the input end of the similarity reasoning module 30 .

Images