Visible light-infrared pedestrian re-identification method based on block affinity reasoning
By adopting block affinity inference method in pedestrian recognition, combining dual-contrast learning and DBSCAN clustering, the problem of low cross-modal pedestrian recognition accuracy is solved, and more efficient and accurate pedestrian recognition is achieved.
Patent Information
- Application Number
- CN202510029664.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-08
- Publication Date
- 2025-05-30
AI Technical Summary
The prior art in cross-modal pedestrian recognition has low accuracy, and a simple distance measurement method ignores the potential relationship between library images, resulting in loss of affinity data.
Using a block affinity inference method, the features of visible light and infrared images are extracted through the dual-contrast learning module, DBSCAN clustering is used, and the landmark points are obtained through the central search strategy, and the two-part graphs are constructed for modal correspondence, and finally pedestrian re-identification is achieved through block affinity inference.
The accuracy of cross-modal pedestrian re-identification is improved, the calculation overhead is reduced through blocking strategy, the distinction of features and the accuracy of modal correspondence are enhanced, and the matching performance is significantly improved.
Smart Images
Figure CN120071432A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of machine vision, and specifically relates to a visible light-infrared pedestrian re-identification method based on block affinity reasoning. Background Art
[0002] Pedestrian re-identification is a task of retrieving pedestrian images through different cameras. Existing methods mainly focus on matching visible light images, solving the single-modal problem. However, conventional visible surveillance cameras cannot effectively capture pedestrian information under poor lighting conditions. To address this challenge, modern surveillance cameras can automatically switch to the infrared mode to collect infrared images under low-light conditions. Accordingly, the research on visible light-infrared pedestrian re-identification is proposed, whose goal is to identify the same person from visible light / infrared gallery images when given an image of another modality. Most existing methods make inferences through simple distance metrics, such as calculating the Euclidean distance or Mahalanobis distance between the output features to measure similarity. Although these methods are simple and intuitive, they treat each gallery image as an independent individual, ignoring the potential relationships between gallery images, resulting in the loss of valuable affinity data and thus affecting the matching performance. To solve this problem, some methods optimize the matching by calculating the Jaccard distance between gallery images. However, the Jaccard distance only focuses on the presence or absence of elements, ignoring the specific similarity scores, thus limiting its effect to a certain extent. Accordingly, some methods consider the similarity between gallery images, but due to the need to perform a large number of calculations again when querying gallery images and updating gallery image data, the overall effect is not ideal. Summary of the Invention
[0003] Object of the Invention: To overcome the above-mentioned disadvantages of the prior art, the present invention provides a visible light-infrared pedestrian re-identification method based on block affinity reasoning to solve the problem of low accuracy in cross-modal pedestrian re-identification.
[0004] Technical Solution: The present invention discloses a visible light-infrared pedestrian re-identification method based on block affinity reasoning, including the following steps:
[0005] Step 1: Use a visible light camera to collect pedestrian images as visible light domain training images, and use an infrared camera to collect pedestrian images as infrared domain training images; perform image enhancement on the acquired images;
[0006] Step 2: Use a dual contrast learning module to extract the features of visible light images and infrared images respectively, and use DBSCAN to cluster the extracted features respectively;
[0007] Step 3: In the training stage, use the center search strategy to obtain the landmark points of each class in the clustering result, and connect the landmark points by constructing a bipartite graph to achieve modality correspondence;
[0008] Step 4: Use the modal correspondence results to partition the image library. Calculate the affinity values between blocks and the affinity values between blocks and query library images during the test phase, and retrieve the images to be queried according to the affinity values.
[0009] Further, the specific method of step 1 is as follows:
[0010] Step 1.1: Use a visible light camera to obtain a visible light image dataset; perform data cleaning on the images, including linear interpolation, Gaussian filtering, standardizing the format, and resizing the pictures; perform gray-scale enhancement on the visible light images.
[0011] Step 1.2: Use an infrared camera to obtain an infrared image dataset; perform data cleaning on the images, including linear interpolation, Gaussian filtering, standardizing the format, and resizing the pictures; perform temperature mapping enhancement on the infrared images.
[0012] Further, the specific method of step 2 is as follows:
[0013] Step 2.1: Input the visible light image dataset and the infrared image dataset obtained in step 1 into the dual contrast learning module for feature extraction; the dual contrast learning module includes a ResNet50 backbone network and an AGW backbone network; the visible light image dataset is subjected to feature extraction through the ResNet50 backbone network, and the infrared image dataset is subjected to feature extraction through the AGW backbone network, and finally feature sets of the two types of images are obtained.
[0014] Step 2.2: Use DBSCAN to cluster the feature sets of the visible light images and infrared images respectively, and use the following contrast loss function to train the visible light clustering and infrared clustering respectively:
[0015]
[0016] where L ra represents the single-modal loss function; represents the feature vector of the visible light image, represents the feature vector of the infrared image; represents the Euclidean distance between the two features; a is a binary label indicating whether the sample pair comes from the same category, with a value of 1 indicating the same category and a value of 0 indicating different categories; m is a threshold used to control the minimum discrimination distance between positive and negative samples; after training is completed, identity labels are assigned to the final results formed by clustering the two modalities.
[0017] Further, the specific method of step 3 is as follows:
[0018] Step 3.1: Conduct center search first in the training phase; for each cluster, it is necessary to calculate the Euclidean distance from each point to the center point, sort the distances from small to large, and select the top ten sample points to enter the next round of screening. If there are less than ten, calculate according to the actual sample points;
[0019] Step 3.2: Calculate the distance from each selected point to the rest of the samples, and further calculate the variance of each point. The specific expression is as follows:
[0020]
[0021] where V i represents the variance of point x i ; N represents the number of actually selected sample points; D(x i , x j ) represents the Euclidean distance from point x i to point x j ; MD i represents the mean of the distances from point x i to all other selected points; after calculating the variance of each point, select the point with the smallest variance as the landmark point of this cluster, which is the representative feature of this class;
[0022] Step 3.3: Further construct two graphs G vis =(V vis , E vis ) and G ir =(V ir , E ir ). One is a bipartite graph constructed by visible light landmark points, and the other is a bipartite graph constructed by infrared landmark points; the node sets of the bipartite graphs respectively represent the representative visible light and infrared image features selected, and the edge sets of the bipartite graphs respectively represent the similarity between the features;
[0023] Step 3.4: Calculate the similarity matrix S between visible light features and infrared features. The element in the i-th row and j-th column of the matrix is expressed as using cosine similarity; based on the similarity matrix S, traverse each row, select the feature with the highest similarity in each row for matching to form a matching pair; after traversing, if there is still a certain column that has not found a matching feature, then traverse each row in this column, find the feature with the highest similarity and match it with it. Finally, until corresponding feature matches are found for each row and each column. During the entire matching process, use the modal association loss function L ma for training, which is expressed as:
[0024]
[0025] where γ represents a threshold used to control the minimum requirement of similarity; and respectively represent the features of visible light fiducial points and infrared fiducial points; represents the cosine similarity between visible light fiducial point i and infrared fiducial point j.
[0026] Furthermore, the specific method of step 4 is as follows:
[0027] Step 4.1: According to the graph matching result of step 3, the corresponding visible light data set and infrared data set of the matching nodes are grouped together to finally form N groups; when the number of visible light nodes is greater than the number of infrared nodes, the number of groups N represents the number of infrared nodes, and when the number of visible light nodes is less than the number of infrared nodes, the number of groups N represents the number of visible light nodes;
[0028] Step 4.2: Further use the double contrast learning module to extract features from the query library images, and further construct two affinity matrices, one affinity matrix A composed of the graph matching node image set v and the query library image set g gv , and the other is the affinity matrix A constructed by the graph matching node image set v vv ; where each element A gv (i,j) represents the cosine similarity between the node image v i and the query image g j , and each element A vv (j,k) represents the similarity between the image v j and the image v k ;
[0029] Step 4.3: Set a threshold θ for the affinity matrices A gv and A vv , and set the affinity values less than this threshold to zero; the affinity matrices after noise suppression are denoted as and
[0030] Step 4.4: The noise suppression part is trained using the noise suppression loss function L ns , and this function is expressed as:
[0031]
[0032] where p ij represents the affinity value, that is, the element in the i-th row and j-th column of the affinity matrix; represents the indicator function, which is 1 when the condition is satisfied and 0 otherwise;
[0033] Step 4.5: Mean Expansion: Replace the values with zero affinity with the average of the most similar neighbors; for a given gallery image, find M images that are most similar to it, and then replace the affinity value in this image with the average affinity of these M images; this operation is only for the affinity matrix The specific mean expansion can be expressed as:
[0034]
[0035] where, A mean_row (i) represents the average affinity of M images, and the expression is: Affinity matrix represents the matrix after mean expansion In the expansion process, there is no need to determine whether the element is zero. Just find M images that are most similar to it, and then replace it with the average affinity of these M images;
[0036] Step 4.6: In the early stage, A gv uses cosine similarity and performs base distance conversion on it, expressed as: d = 1 - A gv ;
[0037] Step 4.7: Correct the distance between each query image and each node image, expressed as:
[0038] Step 4.8: Subtract the corrected distance between images from the base distance to obtain the final affinity distance D between images, expressed as: D = d - d * ;
[0039] Step 4.9: According to the final distance, for any query image, find the two images with the closest distance. If both of these two images come from the same block, it is determined that the image belongs to the label identity of this block of data; if these two images come from different blocks, it is necessary to further calculate the average cosine similarity between the query image and these two blocks of data respectively. The query image belongs to the label identity of the block of data with the higher average similarity;
[0040] Step 4.10: In terms of updating data, the new data only needs to be compared with the node image dataset, calculate the corresponding cosine similarity, and the classification can be completed by comparing the similarity values.
[0041] Furthermore, calculate the affinity matrix A gv between the graph matching node image set and the query gallery, where each element A gv (i,j) represents the cosine similarity between the node image v i and the query image g j :
[0042]
[0043] Among them, represents the feature of node image v i ; represents the feature of query image g j ;
[0044] Calculate the affinity matrix A between the graph matching node images vv , and each element A in the matrix vv (j, k) represents the similarity between image v j and image v k :
[0045]
[0046] Furthermore, the total loss function L is expressed as:
[0047] L = L ra + L ma + λL ns
[0048] Among them, L ra represents the unimodal loss function, L ma is the modal correlation loss function, L ns is the noise suppression loss function, and λ is a hyperparameter that balances the contribution of the loss terms.
[0049] Beneficial effects:
[0050] (1) During the image feature extraction process of the present invention, a double contrast learning module is adopted to perform deep feature extraction on visible light and infrared images respectively. By designing a highly targeted dual-modal contrast learning strategy, the semantic correlation within and between modalities is fully explored and utilized. This framework not only optimizes the cross-modal features of the same identity, making them closer in the feature space, but also imposes a separation constraint on the features of different identities through the contrast loss function, effectively enhancing the distinctiveness of the features.
[0051] (2) In the modal correspondence part of the present invention, the corresponding methods of center search and graph matching are adopted. The center search strategy comprehensively considers the Euclidean distance and point variance, ensuring that the landmark points can represent the main features of the class, thereby reducing the interference of noise samples on modal matching and improving the accuracy of feature extraction. Calculating the similarity between features based on the graph structure, fully exploring the potential correlation relationship between modalities, making the modal correspondence more accurate; at the same time, constructing a modal correlation loss function in the graph matching stage to optimize the similarity between landmark points, making the feature distributions of visible light and infrared landmark points more consistent, thereby significantly reducing the feature distribution difference between modalities.
[0052] (3) In the cross-modal pedestrian image retrieval part of the present invention, a block affinity reasoning method is adopted. By dividing the image library into blocks, the computational overhead of comparing each image in the entire library one by one is significantly reduced, making the query process more efficient. At the same time, the block strategy provides a clear reference range for subsequent retrieval, shortening the query time and providing traversal for subsequent addition of new data. Based on the affinity values constructed from the affinity matrix, combined with the noise suppression and mean expansion mechanisms, the interference of low-similarity noise on the retrieval results can be effectively eliminated. At the same time, through the dual optimization of the base distance and the corrected distance, the cross-modal matching is made more accurate, improving the robustness of the similarity calculation between images. Description of the Drawings
[0053] Figure 1 It is the overall flowchart of the visible light-infrared pedestrian re-identification method based on block affinity reasoning;
[0054] Figure 2 It is a schematic diagram of the dual contrast learning module;
[0055] Figure 3 It is a schematic diagram of central search and bipartite graph matching;
[0056] Figure 4 It is a schematic diagram of the block affinity reasoning module. Detailed Embodiment
[0057] The following further clarifies the present invention in combination with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent forms of modification by those skilled in the art fall within the scope defined by the appended claims of this application.
[0058] The present invention discloses a visible light-infrared pedestrian re-identification method based on block affinity reasoning, which specifically includes the following steps:
[0059] Step 1: Use a visible light camera to collect pedestrian images as visible light domain training images, and use an infrared camera to collect pedestrian images as infrared domain training images; perform image enhancement on the acquired images.
[0060] Step 1.1: Use a visible light camera to obtain a visible light image dataset; perform data cleaning on the images, including linear interpolation, Gaussian filtering, standardizing the format, and resizing the pictures; perform gray-scale enhancement on the visible light images.
[0061] Step 1.2: Use an infrared camera to obtain an infrared image dataset; perform data cleaning on the images, including linear interpolation, Gaussian filtering, standardizing the format, and resizing the pictures; perform temperature mapping enhancement on the infrared images.
[0062] Step 2: Use the dual contrast learning module to extract the features of visible light images and infrared images respectively, and use DBSCAN to cluster the extracted features respectively.
[0063] Step 2.1: Input the visible light image dataset and infrared image dataset obtained in Step 1 into the dual contrast learning framework for feature extraction; the visible light image dataset is subjected to feature extraction through the ResNet50 backbone network, and the infrared image dataset is subjected to feature extraction through the AGW backbone network, and finally the feature sets of the two types of images are obtained.
[0064] Step 2.2: Use DBSCAN to cluster the feature sets of visible light images and infrared images respectively, and use the following contrast loss function to train visible light clustering and infrared clustering respectively:
[0065]
[0066] where, L ra represents the single-mode loss function; represents the feature vector of the visible light image, represents the feature vector of the infrared image; represents the Euclidean distance between the two features; a is a binary label, indicating whether the sample pair comes from the same category, with a value of 1 indicating the same category and a value of 0 indicating different categories; m is a threshold used to control the minimum discrimination distance between positive and negative samples; after training is completed, identity labels are assigned to the final results formed by clustering the two modalities.
[0067] Step 3: Use the center search strategy to obtain the landmark points of each category in the clustering results during the training phase; connect the landmark points by constructing a bipartite graph to achieve modality correspondence.
[0068] Step 3.1: First perform center search during the training phase; for each cluster, it is necessary to first calculate the Euclidean distance from each point to the center point, sort the distances from small to large, and select the top ten sample points to enter the next round of screening. If there are less than ten, calculate according to the actual sample points.
[0069] Step 3.2: Calculate the distance from each sample to the remaining samples for the selected points, and further calculate the variance of each point. The specific expression is as follows:
[0070]
[0071] where, V i represents the variance of point x i ; N represents the number of actually selected sample points; D(x i ,x j ) represents the distance from point x i to point x jEuclidean distance; MD i represents the point x i the mean of the distances from all other selected points; after calculating the variance of each point, select the point with the smallest variance as the landmark point of the cluster, which is the representative feature of the class.
[0072] Step 3.3: Further construct two graphs G vis =(V vis , E vis ) and G ir =(V ir , E ir ), one is a bipartite graph constructed by visible light landmark points, and the other is a bipartite graph constructed by infrared landmark points; the bipartite graph node sets respectively represent the selected representative visible light and infrared image features; the bipartite graph edge sets respectively represent the similarity between features.
[0073] Step 3.4: Match the two bipartite graphs to complete modality correspondence; calculate the similarity matrix S between visible light features and infrared features, where the element in the i-th row and j-th column of the matrix is expressed as Use cosine similarity; based on the similarity matrix S, traverse each row, select the feature with the highest similarity in each row to form a matching pair; after traversing, if there is still a column without a matching feature, then traverse each row in that column and find the feature with the highest similarity to match it until a corresponding feature match is found for each row and each column; during the entire matching process, use the modality association loss function L ma for training, which can be expressed as:
[0074]
[0075] where γ represents a threshold for controlling the minimum requirement of similarity; and represent the features of visible light landmark points and infrared landmark points respectively; represents the cosine similarity between visible light landmark point i and infrared landmark point j.
[0076] Step 4: Use the modality correspondence results to partition the image library; calculate the affinity value between blocks and the affinity value between blocks and query library images during the test phase, and perform retrieval of the images to be queried according to the affinity value.
[0077] Step 4.1: During the test phase, according to the graph matching results in Step 3, form a block by combining the matching node-corresponding visible light dataset and infrared dataset, and finally form N blocks; when the number of visible light nodes is greater than the number of infrared nodes, the number of blocks N represents the number of infrared nodes, and when the number of visible light nodes is less than the number of infrared nodes, the number of blocks N represents the number of visible light nodes.
[0078] Step 4.2: Further use the feature extractor in the dual contrast learning module to extract features from the query library images, and further construct two affinity matrices, one affinity matrix A composed of the graph matching node image set v and the query library image gallery g gv , and the other is the affinity matrix A constructed from the graph matching node image set v vv . A bunch of visible light images are clustered into piles. Through steps 3.1 and 3.2, we find the landmark points for each pile. The same operation is also performed on a bunch of infrared images. Further, through the graph matching in steps 3.3 and 3.4, we match the photos of the same pedestrians in the two modalities. For example, after visible light clustering, a certain pile has ten points (numbered 1 - 10, and the landmark point is number 5), and after infrared clustering, a certain pile has seven points (numbered 1 - 7, and the landmark point is number 3). Through steps 3.3 and 3.4, we find that the landmark point number 5 and the landmark point number 3 are of the same category. Then, what we do in step 4.1 is to group the ten points in this pile of visible light and the seven points in that pile of infrared together. This block then has 17 features of the same category, and this is the block. The graph matching node image set is the set of landmark points. For example, landmark point number 5 and landmark point number 3. As can be seen at the end of the Figure 3 , the circled points on the left form the graph matching node image set, and the grouping of pile by pile forms the block. It can be understood that the graph matching node image set is used to represent all the blocks.
[0079] Step 4.3: Calculate the affinity matrix A between the graph matching node image set and the query library image gallery gv , where each element A gv (i, j) represents the cosine similarity between the node image v i and the query image g j :
[0080]
[0081] where, represents the feature of the node image v i , represents the feature of the query image g j ; Calculate the affinity matrix A between the graph matching node images vv , and each element A vv (j, k) in the matrix represents the similarity between the image v j and the image v k :
[0082]
[0083] Step 4.4: For the affinity matrices A gv and A vvSet a threshold θ and set the affinity values less than this threshold to zero; the affinity matrix after noise suppression is denoted as and Take the matrix as an example (the matrix is the same), and the specific noise suppression is as follows:
[0084]
[0085] Step 4.5: The noise suppression part is trained using the noise suppression loss function L ns which can be expressed as:
[0086]
[0087] where p ij represents the affinity value, that is, the element in the i-th row and j-th column of the affinity matrix; represents the indicator function, which is 1 when the condition holds and 0 otherwise.
[0088] Step 4.6: Mean expansion: Replace the values with zero affinity with the average of the most similar neighbors; for a given gallery image, find M images that are most similar to it, and then use the average affinity of these M images to replace the affinity value in this image; this operation is only for the affinity matrix The specific mean expansion can be expressed as:
[0089]
[0090] where A mean_row (i) represents the average affinity of M images, and the expression is: The affinity matrix represents the matrix after mean expansion In the expansion process, there is no need to determine whether the element is zero. Just find M images that are most similar to it, and then use the average affinity of these M images to replace it.
[0091] Step 4.7: Initially, A gv uses cosine similarity, and its base distance transformation is expressed as: d = 1 - A gv .
[0092] Step 4.8: Correct the distances between each query image and each node image, which is expressed as:
[0093] Step 4.9: Subtract the corrected distance between images from the base distance to obtain the final affinity distance D between images, which can be expressed as: D = d - d * .
[0094] Step 4.10: According to the final distance, for any image to be queried, find the two closest images. If both of these two images come from the same data block, it can be determined that the image belongs to the label identity of this block of data; if the two images come from different data blocks, it is necessary to further calculate the average cosine similarity between the query image and these two blocks of data respectively. The query image belongs to the label identity of the block of data with the higher average similarity.
[0095] Step 4.11: In terms of updating data, the new data only needs to be compared with the node image dataset, calculate the corresponding cosine similarity, and the classification can be completed by comparing the similarity values. Finally, the total loss function L of this method can be expressed as:
[0096] L = L ra + L ma + λL ns
[0097] where λ is a hyperparameter that balances the contributions of the loss terms.
[0098] Experimental part:
[0099] The proposed method is evaluated on the widely used visible-infrared datasets SYSU-MM01 and RegDB. The SYSU-MM01 dataset contains 1900 pedestrian images from 6 different viewpoints. The images of each pedestrian are taken from two viewpoints, and there is also a thermal imaging viewpoint. This dataset is one of the challenging datasets in the field of cross-modal pedestrian re-identification. On the SYSU-MM01 dataset, the test is divided into two settings: full retrieval mode and indoor retrieval mode. In the full retrieval mode, the gallery consists of visible light images, and the query set consists of infrared images. In the indoor retrieval mode, the visible light images taken from outdoor scenes (cameras 4 and 5) are discarded. The RegDB dataset contains 412 pedestrian identities. For each pedestrian, 10 RGB images and 10 thermal images are collected, including 254 females and 158 males. Among the 412 pedestrians, 156 are taken from the front and 256 are taken from the back. The RegDB dataset contains two test settings: infrared-to-visible and visible-to-infrared modes.
[0100] In the training stage, a non-local module based on AGW was adopted to enhance the network, and ResNet50
[13] was used as the feature extractor; the parameters of the backbone network were initialized with the pre-trained weights of ImageNet. In a small batch, the number P of each class and the number K of samples of each class were both 8; all pedestrian images were resized to 256×128; the SGD optimizer was used for model training, the initial learning rate of randomly initialized parameters was 0.1, and the initial learning rate of pre-trained parameters was 0.01; the learning rate was reduced by 10 times at the 20th and 50th epochs; random gray-scale enhancement was performed on visible light images; in each training epoch, DBSCAN was used to cluster the images within each modality; the maximum distance on the SYSU-MM01 dataset was set to 0.5, and on the RegDB dataset was set to 0.25; the minimum number of clusters for both datasets was set to 4. During the clustering process, the memory update rate λ was set to 0.05, the temperature factor τ was set to 0.1, and the weight parameter μ was set to 0.5.
[0101] Common evaluation metrics were used in the experiments to evaluate these two datasets, including using the Cumulative Match Characteristic (CMC) curve, the Mean Inverse Negative Penalty (mINP), and the Mean Average Precision (mAP) to measure the recognition performance of the model; the CMC curve reflects the classification accuracy of the model, usually expressed in the form of Rank-n for the first n matching results; mINP represents the proportion of correct samples among the retrieved samples up to the last correct result; mAP is the average accuracy of all returned results for a certain classification.
[0102] Table 1 Comparison of experimental results of different methods on the CUHK-MM01 dataset (%)
[0103]
[0104] In the comparative experiments of different methods on the SYSU-MM01 dataset, all evaluation metrics of the framework proposed in this paper exceeded the optimal model in both the full retrieval mode and the indoor retrieval mode; specifically, in the full retrieval mode, the mAP of the model reached 52.92%, an increase of 1.14% compared to the optimal model ACCL, and the accuracy of Rank-1 reached 58.36%, an increase of 1.09% compared to ACCL. In the indoor retrieval mode, the mAP reached 63.98%, an increase of 1.01% compared to the optimal model AGW, and the Rank-1 accuracy was 57.02%, an increase of 0.79% compared to ACCL.
[0105] Comparison of experimental results of different methods on the RegDB dataset (%)
[0106]
[0107]
[0108] In the comparative experiments on the RegDB dataset, the framework proposed in this paper outperforms other models in both test modes. In the Visible to Thermal mode, compared with the optimal model, the mAP is improved by 0.72% and the Rank-1 accuracy is increased by 0.7%; in the Thermal to Visible mode, the mAP is improved by 1.86% and the Rank-1 accuracy is increased by 1.59%. Generally speaking, the framework proposed in this paper is still highly competitive on these two datasets.
[0109] The above embodiments are only for explaining the technical concept and characteristics of the present invention, and the purpose is to enable those who are familiar with this technology to understand the content of the present invention and implement it accordingly, and it cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit and essence of the present invention should be covered within the protection scope of the present invention.
Claims
1. A visible light-infrared pedestrian re-identification method based on block affinity reasoning, characterized in that: The steps include: Step 1: Use a visible light camera to collect pedestrian images as visible light domain training images, and use an infrared camera to collect pedestrian images as infrared domain training images; perform image enhancement on the acquired images; Step 2: Use the dual contrast learning module to extract the features of visible light images and infrared images respectively, and use DBSCAN to cluster the extracted features respectively; Step 3: In the training phase, the center search strategy is used to obtain the landmark points of each category in the clustering results, and the landmark points are connected by constructing a bipartite graph to achieve modal correspondence; Step 4: Use the modal correspondence results to divide the image library into blocks. In the test phase, calculate the affinity values between blocks and the affinity values between blocks and query library images, and retrieve the query image based on the affinity values.
2. According to the method of visible light-infrared pedestrian re-identification based on block affinity reasoning in claim 1, it is characterized in that: The specific method of step 1 is: Step 1.1: Use a visible light camera to obtain a visible light image dataset; perform data cleaning on the image, including linear interpolation, Gaussian filtering, standardized format and image scaling; and perform grayscale enhancement on the visible light image; Step 1.2: Use an infrared camera to obtain an infrared image dataset; perform data cleaning on the image, including linear interpolation, Gaussian filtering, standardized format and image scaling; and use temperature mapping to enhance the infrared image.
3. The visible light-infrared pedestrian re-identification method based on block affinity reasoning according to claim 1 is characterized in that: The specific method of step 2 is: Step 2.1: Input the visible light image dataset and the infrared image dataset obtained in step 1 into the dual contrast learning module for feature extraction; the dual contrast learning module includes a ResNet50 backbone network and an AGW backbone network; the visible light image dataset is feature extracted through the ResNet50 backbone network, and the infrared image dataset is feature extracted through the AGW backbone network, and finally a feature set of the two images is obtained; Step 2.2: Use DBSCAN to cluster the feature sets of visible light images and infrared images respectively, and use the following contrast loss function to train the visible light clustering and infrared clustering respectively: Among them, L ra represents the single membrane state loss function; Represents the feature vector of the visible light image, A feature vector representing an infrared image; Represents the Euclidean distance between two features; a It is a binary label, indicating whether the sample pair is from the same category. A value of 1 indicates the same category, and a value of 0 indicates different categories. m is a threshold used to control the minimum distinguishing distance between positive and negative samples. After training, the final result formed by clustering the two modalities is labeled with an identity label.
4. The visible light-infrared pedestrian re-identification method based on block affinity reasoning according to claim 1 is characterized in that: The specific method of step 3 is: Step 3.1: In the training phase, the center search is first performed; for each cluster, the Euclidean distance from each point to the center point needs to be calculated first, and the top ten sample points are selected to enter the next round of screening by sorting the distances from small to large. If there are less than ten, the actual sample points will be used for calculation; Step 3.2: Calculate the distance between each sample and the remaining samples, and further calculate the variance of each point. The specific expression is as follows: Among them, V i Represents point x i The variance of; N represents the number of sample points actually screened; D(x i ,x j ) represents point x i To point x j Euclidean distance; MD i Represents point x i The mean of the distances to all other selected points; after calculating the variance of each point, select the point with the smallest variance as the landmark point of the cluster, which is the representative feature of the class; Step 3.3: Further construct two graphs G vis =(V vis ,E vis ) and G ir =(V ir ,E ir ), one is a bipartite graph constructed by visible light landmarks, and the other is a bipartite graph constructed by infrared landmarks; the node sets of the bipartite graphs represent the representative visible light and infrared image features selected, and the edge sets of the bipartite graphs represent the similarities between the features; Step 3.4: Calculate the similarity matrix S between the visible light feature and the infrared feature, where the element in the i-th row and j-th column of the matrix is represented as Cosine similarity is used; based on the similarity matrix S, each row is traversed, and the feature with the highest similarity in each row is selected to match to form a matching pair; after the traversal is completed, if there is still a column that has not found a matching feature, then each row in the column is traversed to find the feature with the highest similarity to match it, until each row and each column finds the corresponding feature match. In the entire matching process, the modal association loss function L is used for training, which is expressed as: Among them, γ represents a threshold value, which is used to control the minimum requirement of similarity; and Respectively represent the characteristics of visible light marker points and infrared marker points; Represents the cosine similarity between visible light landmark point i and infrared landmark point j.
5. The visible light-infrared pedestrian re-identification method based on block affinity reasoning according to claim 1 is characterized in that: The specific method of step 4 is: Step 4.1: According to the graph matching result of step 3, the visible light data set and the infrared data set corresponding to the matched nodes are combined into one block, and finally form N blocks; when the visible light node is larger than the infrared node, the block number N represents the number of infrared nodes, and when the visible light node is smaller than the infrared node, the block number N represents the number of visible light nodes; Step 4.2: Further use the double contrast learning module to extract features from the query library image and further construct two affinity matrices, one consisting of the graph matching node image set v and the query library g. gv , and the other is the affinity matrix A constructed by the graph matching node image set v vv ; where each element A gv (i,j) represents the node image v i and query image g j The cosine similarity between each element A vv (j,k) represents image v j and image v k The similarity between Step 4.3: Targeting the affinity matrix A gv and A vv Set a threshold θ and set the affinity values less than the threshold to zero; the affinity matrix after noise suppression is expressed as and Step 4.4: The noise suppression part is trained using the noise suppression loss function L, which represents for: where p ij represents the affinity value, i.e., the element in the i-th row and j-th column of the affinity matrix; Represents an indicator function, which is 1 when the condition is met, otherwise it is 0; Step 4.5: Mean expansion: Replace the value with zero affinity with the average of the most similar neighbors; for a given gallery image, find M images that are most similar to it, and then replace the affinity value in the image with the average affinity of these M images; this operation is only for the affinity matrix The specific mean expansion can be expressed as: Among them, A mean_row (i) represents the average affinity of M images, expressed as: Affinity Matrix Represents the matrix after mean expansion The expansion process does not need to determine whether the element is zero, it only needs to find M images that are most similar to it, and then replace it with the average affinity of these M images; Step 4.6: Early A gv The cosine similarity is used, and the base distance is converted to express it as: d = 1-A gv ; Step 4.7: Correct the distance between each query image and each node image, expressed as: Step 4.8: Use the base distance to subtract the corrected distance between images to get the final affinity distance D between images, expressed as: D = dd * ; Step 4.9: According to the final distance, any query image finds the first two images with the closest distance. If the two images are from the same block, the image is determined to belong to the label identity of the block data; if the two images are from different blocks, it is necessary to further calculate the average cosine similarity between the query image and the two blocks of data. The block with the higher average similarity means that the query image belongs to the label identity of the block data; Step 4.10: In terms of updating data, the new data only needs to be compared with the node image dataset, and the corresponding cosine similarity is calculated. Classification can be completed by comparing the similarity values.
6. The visible light-infrared pedestrian re-identification method based on block affinity reasoning according to claim 5 is characterized in that: Compute the affinity matrix A between the graph matching node image set and the query graph library gv , where each element A gv (i,j) represents the node image v i and query image g j The cosine similarity between: in, Represents the node image v i Features, Denotes the query image g j Features; Compute the affinity matrix A between graph matching node images vv , each element A in the matrix vv (j,k) represents image v j and image v k The similarity between:
7. The visible light-infrared pedestrian re-identification method based on block affinity reasoning according to claim 5 is characterized in that: The total loss function L is expressed as: L=L ra +L ma +λL ns Among them, L ra represents the single membrane state loss function, L ma is the modality association loss function, L ns is the noise suppression loss function, and λ is a hyperparameter that balances the contribution of the loss term.