A Multi-Spectral Vehicle Re-Identification Method Based on Cross Consistency

Abstract

The present invention discloses a multi-spectral vehicle re-identification method based on cross-consistency, and proposes a cross-consistency multi-stream network. By simultaneously constraining the consistency of multi-spectral features of the same vehicle at the modal and sample levels, it helps the multi-stream network better learn vehicle features in the multi-spectral scenario. The present invention reconstructs the dataset MSVR310 to make up for the deficiencies in data diversity of the existing datasets, which greatly enriches the data diversity of the dataset and contains more challenges. The present invention uses the newly proposed cross-consistency multi-stream network as a new multi-spectral vehicle re-identification model. This multi-spectral vehicle re-identification model achieves good re-identification effects on the MSVR310 dataset or other multi-spectral vehicle re-identification datasets, providing an effective solution idea for vehicle re-identification in complex lighting environments and bad weather.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and particularly to a multi-spectral vehicle re-identification method based on cross-consistency. Background Art

[0002] The task of vehicle re-identification is to find a specific vehicle that appears under different monitoring views from an image (video) library according to the vehicle appearance information. Since this technology gets rid of the requirement for a clear license plate image by the license plate recognition system while completing identity recognition, this technology has important application value in smart cities and sky network systems and has attracted the attention of experts and scholars.

[0003] At present, existing vehicle re-identification methods have achieved good results on some publicly available datasets. However, due to the interference of factors such as darkness, complex lighting, occlusion, motion blur, etc., and the huge appearance differences of vehicles under different perspectives, the vehicle re-identification task still faces huge challenges. Among these challenges, different lighting conditions, such as strong light, weak light, darkness, local lighting, etc., all cause serious interference to the vehicle appearance, and this situation is often difficult to avoid under the condition of a single visible light monitoring terminal. With the development of monitoring devices, infrared monitoring terminals have begun to be popularized, providing conditions for imaging using multiple spectra such as visible light, near-infrared, and thermal infrared, and realizing vehicle re-identification under multi-spectral conditions.

[0004] At present, there are relatively few technologies in the field of multi-spectral vehicle re-identification. Existing work mainly includes two multi-spectral vehicle re-identification datasets RGBN300 and RGBNT100, and a multi-stream convolutional network for heterogeneous collaborative perception. There are mainly the following two problems in the existing related technologies:

[0005] (1) The existing multi-spectral vehicle re-identification datasets RGBN300 and RGBNT100 are insufficient in data diversity.

[0006] First of all, the acquisition time of these two datasets is relatively concentrated, and the environmental interference they contain is relatively limited.

[0007] Secondly, since each vehicle is only captured once under different perspectives in the same environment, the environmental impact contained in the same vehicle itself is relatively small, resulting in a relatively low recognition difficulty and a large gap from the real scenario.

[0008] Finally, RGBN300 only contains two spectral data of visible light and near-infrared, and there is a lack of spectral diversity.

[0009] (2) The research of the current method focuses on the fusion of multi - spectral features, ignoring the impact of the huge modal differences between different spectral data on features, the large internal data differences within the modality, and the appearance differences of samples of the same vehicle under different environments and perspectives. Summary of the Invention

[0010] Object of the Invention: The object of the present invention is to solve the deficiencies existing in the prior art and provide a multi - spectral vehicle re - identification method based on cross - consistency.

[0011] Technical Solution: A multi - spectral vehicle re - identification method based on cross - consistency of the present invention includes a training stage and a testing stage;

[0012] Step 1: The training stage includes the following steps:

[0013] Step 1 - 1: Simultaneously obtain multiple multi - spectral samples of the same vehicle from a multi - spectral vehicle re - identification dataset (such as MSVR310, RGBNT100, RGBN300). Each sample includes a visible - light image, a near - infrared image, and a thermal - infrared image;

[0014] Step 1 - 2: Input the obtained multi - spectral samples into a cross - consistency multi - stream network for training;

[0015] The cross - consistency multi - stream network includes three convolutional neural networks. Through these three convolutional neural networks, three spectral features of each sample are extracted respectively. An adaptive layer normalization unit is embedded between the first part and the second part of each convolutional neural network. The input spectral sample passes through the first part, the adaptive layer normalization unit, and the second part of the corresponding convolutional neural network in sequence, and the corresponding final feature is output. Then, the obtained corresponding final feature is sent to a classifier through batch normalization operation to make a class prediction for the original input data;

[0016] Step 1 - 3: Regard different samples from the same vehicle as a group. Calculate the mean of the multi - spectral features included in each sample within the group as the sample - center feature of the sample; calculate the mean of the features under different spectra within the group as the modality - center feature of the spectrum. Finally, calculate the cross - center loss according to the calculated sample - center features and modality - center features within the group; at the same time, calculate the cross - entropy loss according to the comparison between the class prediction results of each branch and the true - value labels;

[0017] Step 1 - 4: After completing the above steps 1 - 1 to 1 - 3, add the calculated cross - center loss and the cross - entropy loss of each branch to obtain the total loss, and use the stochastic gradient descent algorithm to update all parameters in the entire model according to the total loss value;

[0018] Then repeat steps 1-1 to 1-4 until the loss value converges, the network reaches the optimal performance, and the training ends to obtain the trained cross-consistency multi-stream network;

[0019] Step 2: The testing phase includes the following steps:

[0020] Step 2-1: Load a sample of a vehicle from the multi-spectral vehicle re-identification dataset. This sample includes a visible light image, a near-infrared image, and a thermal infrared image. Input the sample into the trained cross-consistency multi-stream network obtained in Step 1;

[0021] Step 2-2: Extract the middle-level features of the input sample through the first part of the cross-consistency multi-stream network, send the middle-level features into the corresponding adaptive layer normalization unit to adjust the feature distribution, and then send the adjusted middle-level features into the second part of the cross-consistency multi-stream network to obtain the corresponding final features; then splice the obtained three final features at the channel level, and the spliced features are used as the final feature representation of the input sample;

[0022] Step 2-3: Divide the test set into a query set and a retrieval set, and extract the corresponding feature representations for all sample images in the retrieval set through Steps 2-1 and 2-2;

[0023] Step 2-4: Extract the corresponding feature representation for the single query vehicle sample input in the query set through Steps 2-1 and 2-2, and compare it with the sample features in the retrieval set obtained in Step 2-3, so as to determine whether the query vehicle is included in the retrieval set.

[0024] Further, the multi-spectral vehicle re-identification dataset includes RGBNT100 and RGBN300. If the RGBNT100 dataset is used, when reading the data, each sample includes a visible light image, a near-infrared image, and a thermal infrared image. When constructing the network model, the cross-consistency multi-stream network sets three convolutional neural networks; if the RGBN300 dataset is used, when reading the data, each sample includes a visible light image and a near-infrared image. When constructing the network model, the cross-consistency multi-stream network sets two convolutional neural networks.

[0025] Further, the multi-spectral vehicle re-identification dataset is reconstructed to form the MSVR310 dataset. This MSVR310 dataset (i.e., Multi-spectral Vehicle Re-identification 310) includes 310 target vehicles and 2087 groups of samples. Each sample includes one visible light, near-infrared, and thermal infrared image, for a total of 6261 images. And each image includes a vehicle identity label, a sample view label, and a sample shooting time label.

[0026] Further, the first part of the convolutional neural network (i.e., the first half of ResNet50) includes the first convolutional layer conv1, the first stacked layer layer1, and the second stacked layer layer2 to obtain the middle-level features of the input samples; then the distribution of the obtained middle-level features is adjusted by the adaptive layer normalization unit; the second part (the second half of ResNet50) includes the third stacked layer layer3, the fourth stacked layer layer4, and the global average pooling layer to obtain the final features corresponding to the input samples; wherein, the middle-level features of each sample include middle-level visible light features, middle-level near-infrared features, and middle-level thermal infrared features, and the final features of each sample include final visible light features, final near-infrared features, and final thermal infrared features. The pre-trained parameters are loaded before the convolutional neural network is trained, and then all the parameters of the convolutional neural network will be dynamically updated by the backpropagation algorithm and the optimizer during the training process.

[0027] Further, when the adaptive layer normalization unit adjusts the distribution of the middle-level features, a batch of input features are processed at the single entity level without considering the influence between different image features. Specifically:

[0028] First, calculate the overall mean μ and standard deviation σ of the input middle-level features, and then subtract the mean from the features and divide by the standard deviation (a very small positive value will be added to the standard deviation here to ensure that the standard deviation is not 0) to perform the standardization operation; at the same time, the original input features will pass through two independent factor learning modules respectively to learn the scaling factor γ and the offset factor β, where each factor learning module is stacked by multiple convolutional layers and pooling layers, and a scalar will be finally learned from the original input features through multi-layer convolution and pooling; multiply the standardized features by the scaling factor γ and then add the offset factor β to realize the rescaling and offset of the features to ensure a certain degree of adaptive adjustment of different features; finally, the features after rescaling and offset are the final output features.

[0029] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0030] (1) The present invention improves the multi-stream network and proposes a cross-consistency multi-stream network, which helps the multi-stream network better learn vehicle features in the multi-spectral scenario by simultaneously constraining the consistency of multi-spectral features of the same vehicle at the modal and sample levels.

[0031] (2) The present invention proposes an adaptive layer normalization unit, which alleviates the difference in the internal feature distribution of the modality and reduces the model learning difficulty by dynamically adjusting the distribution of the internal features of the modality.

[0032] (3) The present invention reconstructs the dataset MSVR310 to make up for the lack of data diversity in existing datasets. The dataset MSVR310 contains rich data at different times of day and night, in different weather conditions, and in different scenarios. Since the same car may appear at different times and locations, this greatly enriches the data diversity of the dataset and contains more challenges.

[0033] (4) The present invention combines the cross-consistency multi-stream network with the MSVR310 dataset to implement a new classification training model, which provides an effective solution for vehicle re-identification in complex lighting environments and severe weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a diagram of the cross-consistency multi-stream network structure in the present invention;

[0035] Figure 2 Schematic diagram of cross-center loss in the present invention;

[0036] Figure 3 Schematic diagram of the adaptive layer normalization unit in the present invention;

[0037] Figure 4 Schematic diagram of the MSVR310 multispectral vehicle re-identification dataset in this invention;

[0038] Figure 5 Schematic diagram of the training and testing process in the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0040] like Figure 1 and Figure 5 As shown, a multispectral vehicle re-identification method based on cross consistency of this embodiment includes a training phase and a testing phase;

[0041] Step 1: The training phase includes the following steps:

[0042] Step 1-1: Simultaneously obtain multiple multispectral samples of the same vehicle from the multispectral vehicle re-identification dataset. Each sample includes a visible light image, a near-infrared image, and a thermal infrared image.

[0043] Step 1-2: Input the obtained multi-spectral samples into the cross-consistency multi-stream network for training;

[0044] The cross-consistency multi-stream network includes three convolutional neural networks, which respectively extract three spectral features of each sample. An adaptive layer normalization unit is embedded between the first part and the second part of each convolutional neural network. The input spectral samples sequentially pass through the first part, the adaptive layer normalization unit, and the second part of the corresponding convolutional neural network, and the corresponding final features are output. Then, the obtained corresponding final features are sent to the classifier after batch normalization operation to make a class prediction for the original input data.

[0045] Steps 1-3: Consider different samples from the same vehicle as a group. Calculate the mean of the multi-spectral features included in each sample within the group as the sample central feature of the sample; calculate the mean of the features under different spectra within the group as the modal central feature of the spectrum. Finally, calculate the cross central loss based on the calculated sample central features and modal central features within the group. At the same time, calculate the cross-entropy loss according to the comparison between the class prediction results of each branch and the true label.

[0046] Steps 1-4: After completing the above Steps 1-1 to Steps 1-3, add the calculated cross central loss and the cross-entropy loss of each branch to obtain the total loss, and use the stochastic gradient descent algorithm to update all parameters in the entire model according to the total loss value.

[0047] Then repeat Steps 1-1 to Steps 1-4 until the loss value converges, the network reaches the optimal performance, and the training ends to obtain the trained cross-consistency multi-stream network.

[0048] Step 2: The test phase includes the following steps:

[0049] Step 2-1: Load a sample of the vehicle, which includes a visible light image, a near-infrared image, and a thermal infrared image, and input the sample into the trained cross-consistency multi-stream network obtained in Step 1.

[0050] Step 2-2: Extract the middle-level features of the input sample through the first part of the cross-consistency multi-stream network, send the middle-level features into the corresponding adaptive layer normalization unit to adjust the feature distribution, and then send the adjusted middle-level features into the second part of the cross-consistency multi-stream network to obtain the corresponding final features. Then, splice the obtained three final features at the channel level, and the spliced features are used as the final feature representation of the input sample.

[0051] Step 2-3: Divide the test set into a query set and a retrieval set, and extract the corresponding feature representations for all sample images in the retrieval set through Steps 2-1 and 2-2.

[0052] Step 2-4: Extract the corresponding feature expressions for the input single query vehicle sample in the query set through Steps 2-1 and 2-2, and compare them with the sample features in the retrieval set obtained in Step 2-3, so as to determine whether the retrieval set contains the query vehicle.

[0053] The multi-spectral vehicle re-identification dataset of this embodiment uses the MSVR310 dataset. The MSVR310 dataset includes 310 target vehicles and 2,087 groups of samples. Each sample includes one visible light, one near-infrared, and one thermal-infrared image, for a total of 6,261 images. Moreover, each image includes a vehicle identity label, a sample perspective label, and a sample shooting time label. The images of each modality are collected for each target vehicle from eight directions, namely, directly in front, left front, right front, left side, right side, left rear, right rear, and directly in the rear, through three spectra. The original collected data is manually cropped, selecting the target vehicle area and including a small range of background information. During the image selection and manual cropping process, the consistency of the content of different spectral imaging results is ensured, and the vehicle license plate area information is manually removed to prevent the leakage of private information.

[0054] The MSVR310 dataset contains rich data during the day, at different times at night, under different weather conditions, and in different scenarios. Since the same vehicle will appear at different times and different locations, this greatly enriches the data diversity of the dataset and contains more challenges.

[0055] The acquisition devices for the three types of modality images in the MSVR310 dataset include:

[0056] 1) The 360D886 camera for collecting visible light data during the day, with a resolution of 1920×1080;

[0057] 2) The main rear camera of the Xiaomi 8 smartphone for collecting visible light data at night, with a resolution of 3024×4032;

[0058] 3) Another 360D886 camera for collecting near-infrared data (this camera can freely switch between visible light and near-infrared imaging modes;

[0059] 4) The FLIR SC620 camera for collecting thermal-infrared data, with a resolution of 640×480.

[0060] To reflect the vehicle identity recognition process in the real scenario, when the MSVR310 dataset is tested, it is required that when the query sample matches the retrieval set sample, the data from the same vehicle in the same time period should be avoided for matching.

[0061] In this embodiment, the first part of the convolutional neural network (i.e., the first half of ResNet50) includes the first convolutional layer conv1, the first stacked layer layer1, and the second stacked layer layer2 to obtain the middle-level features of the input samples. Then, the distribution of the obtained middle-level features is adjusted through the adaptive layer normalization unit. The second part (the second half of ResNet50) includes the third stacked layer layer3, the fourth stacked layer layer4, and the global average pooling layer to obtain the final features corresponding to the input samples. Among them, the middle-level features of each sample include middle-level visible light features, middle-level near-infrared features, and middle-level thermal infrared features, and the final features of each sample include final visible light features, final near-infrared features, and final thermal infrared features.

[0062] As Figure 3 shown, the specific adjustment process of the adaptive layer normalization unit in this embodiment is as follows:

[0063] First, calculate the overall mean μ and standard deviation σ of the input middle-level features, and then subtract the mean from the features and divide by the standard deviation (a very small positive value will be added to the standard deviation here to ensure that the standard deviation is not 0) to perform the standardization operation. At the same time, the original input features will pass through two independent factor learning modules to learn the scaling factor γ and the offset factor β respectively. Each factor learning module is stacked by multiple convolutional layers and pooling layers, and a scalar is finally learned from the original input features through multiple layers of convolution and pooling. Multiply the standardized features by the scaling factor γ and then add the offset factor β to realize the rescaling and offset of the most features, so as to ensure that different features can be adaptively adjusted to a certain extent. Finally, the features after rescaling and offset are the final output features.

[0064] Since the multi-modal data is collected from a complex environment, there are often great differences in the content and style of the pictures within each modality, resulting in huge differences in the feature distributions learned by the network, which leads to two problems: (1) The differences between different pictures of the same vehicle with the same spectrum are huge, and the feature distributions are also very different, which is not conducive to identity confirmation; (2) The overall data distribution differences are huge, increasing the learning difficulty of the network.

[0065] The adaptive layer normalization unit here greatly reduces the distribution differences of features within the same modality by dynamically adjusting the feature distribution of a single picture, reduces the learning difficulty of the network model, alleviates the differences of various internal features, and enables the model to learn better identity expressions.

[0066] Example 1:

[0067] As Figure 4 and Figure 5As shown in the figure, in this embodiment, the multi-spectral vehicle re-identification method based on cross-consistency uses the MSVR310 dataset. Each multi-spectral vehicle sample is an image triple, including one visible light image, one near-infrared image, and one thermal infrared image, rather than any single image.

[0068] The training process is as follows:

[0069] Step (A): For all the images in all vehicle samples during the training process, in this embodiment, the resolution of each sample image is first uniformly scaled to 128×256, randomly flipped left and right with a probability of 50%, and 0 values with a width of 10 pixels are filled around the image. Then the filled image is randomly cropped into a rectangular image of size 128×256. After that, the values of each pixel in the image are normalized, and a smaller area is randomly selected for random erasing with a probability of 50%. The pixel points in the final image are floating-point numbers between -1 and 1.

[0070] After completing the above data preprocessing operations, each time 8 vehicles in the training set are selected, and 4 multi-spectral samples for each vehicle (a total of 32 samples and 96 images) are used for one forward propagation.

[0071] Step (B): The convolutional neural networks in different branches of the cross-consistency multi-stream network are independent of each other. For the input multi-spectral samples, the sample images of different spectra are sent into the corresponding convolutional neural network branches to learn the features of the spectral data.

[0072] In this process, for each branch convolutional neural network, an image of a certain spectrum is input. Then the image sample passes through the first part of the network (i.e., the first half of ResNet50, including a convolutional layer, the first stack layer, and the second stack layer) to obtain the middle-level features. Then the middle-level features are sent into the corresponding adaptive layer normalization unit of this branch. The adaptive layer normalization unit normalizes the input features and adaptively rescales and outputs the adjusted features (such as Figure 3 ), and the adjusted features are sent into the second part of the network (the second half of ResNet50, including the third stack layer, the fourth stack layer, and the global average pooling layer) to obtain the feature representation of the spectral image. Then these features pass through the batch normalization operations of their respective branches and are sent into the corresponding classifiers to make predictions on the categories of the input data. This process is as Figure 1 shown.

[0073] Step (C) divides the samples from the same vehicle into a group. Based on the feature expressions obtained from each branch in step (B), the mean of the different spectral image features within the different samples in the group is calculated as the sample center feature of the sample. The mean of the image features contained in the different spectra in the group is calculated as the modal center feature of the spectrum. Finally, the distance between the different sample center features and the distance between the different modal center features are calculated. The two are the sample center loss value and the modal center loss respectively. The two are added together to form the complete cross center loss. The process is as follows: Figure 2 At the same time, the category prediction is obtained for each branch, and the cross entropy loss of each branch is obtained by comparing it with the true value.

[0074] In step (D), the cross entropy loss of each branch is added to the cross center loss to obtain the overall loss value, and the loss value is back-propagated back to the network to calculate the gradient corresponding to each parameter. The parameter model is updated through the optimizer, and then the data is continuously read and repeated for iteration until the model converges.

[0075] Here, by using sample center features and modal center features to calculate the cross-center loss, the distance between different sample centers and the distance between different modal centers of the same vehicle can be simultaneously shortened, and the sample differences in the same vehicle can be overcome, which is beneficial to vehicle identification. It can also overcome the modal differences of different spectral data, which is beneficial to enhance the utilization effect of different modal data.

[0076] Common losses in existing technologies include triplet loss, center loss, and heterogeneous center loss. The calculation of triplet loss and center loss relies on a distance metric for a single image, which can be unreliable in complex scenarios. Heterogeneous center loss only considers differences between modalities, ignoring differences between different samples of the same vehicle, making it unsuitable for direct application in the multimodal scenarios of this invention.

[0077] In this embodiment, a total of 1200 rounds of learning are performed, and the initial learning rate is set to 0.00035, which is reduced by a factor of 10 in the 550th and 800th rounds, respectively.

[0078] The testing process is as follows:

[0079] Step (a): For all sample images in the test process, this embodiment first uniformly scales the sample image resolution to 128×256, then performs a normalization operation on the scaled image, and then feeds the test sample image into the trained cross-consistency multi-stream network.

[0080] In step (b), the cross-consistency multi-stream network obtains the features of different spectral data of the input sample image through forward propagation, and splices the features of different spectra from the same sample image at the channel level to obtain the final feature expression of the sample.

[0081] Step (c): Divide the test set into a query set and a retrieval set, and extract the corresponding feature representations for all sample images in the retrieval set through steps (a) and (b).

[0082] Step (d): Extract the corresponding feature representation for a single query vehicle sample input in the query set through steps (a) and (b), and compare it with the sample features in the retrieval set obtained in step (c), so as to determine whether the query vehicle is included in the retrieval set.

[0083] Table 1 Comparison between the present invention and the prior art on MSVR310

[0084]

[0085] As shown in Table 1, when the present invention and the prior art are compared for re-identification on the MSVR310 dataset, it can be found that the technical solution of the present invention has a relatively obvious lead in both indicators of mean average precision (mAP) and cumulative match characteristics (CMC, reflected by Rank-1, Rank-5, and Rank-10).

[0086] Table 2 Comparison between the present invention and the prior art on RGBNT100 and RGBN300

[0087]

[0088] As shown in Table 2, when the present invention and the prior art are compared for re-identification on the two datasets of RGBNT100 and RGBN300, it can be found that the technical solution of the present invention is also basically in the leading position in various indicators of the two existing datasets. Among them, when applying the RGBN300 dataset, the cross-consistency multi-stream network adopted only needs to design two convolutional neural networks, and the data samples read do not contain thermal infrared pictures; while for the RGBNT100 dataset, three independent convolutional neural network branches are still used. At the same time, 120 rounds of iteration are carried out during training, the initial learning rate is 0.00035, and the learning rate is reduced by 10 times at the 35th and 70th rounds respectively. Other settings in the training and testing processes are the same as those of MSVR310, and the respective test protocols (prohibiting the matching of samples of the same vehicle from the same perspective) are used for index calculation.

Claims

1. A multi-spectral vehicle re-identification method based on cross-consistency, characterized in that: It includes a training stage and a testing stage; Step 1: The training stage includes the following steps: Step 1-1: Simultaneously obtain multiple multi-spectral samples of the same vehicle from the multi-spectral vehicle re-identification dataset. Each sample includes a visible light image, a near-infrared image, and a thermal infrared image; Step 1-2: Input the obtained multi-spectral samples into the cross-consistency multi-stream network for training; The cross-consistency multi-stream network includes three convolutional neural networks. Through these three convolutional neural networks, three spectral features of each sample are extracted respectively. An adaptive layer normalization unit is embedded between the first part and the second part of each convolutional neural network. The input spectral sample sequentially passes through the first part, the adaptive layer normalization unit, and the second part of the corresponding convolutional neural network, and the corresponding final feature is output. Then, the obtained corresponding final feature is subjected to batch normalization operation and sent to the classifier for class prediction; Step 1-3: Regard different samples from the same vehicle as a group. Calculate the mean of the multi-spectral features included in each sample within the group as the sample center feature of the sample; calculate the mean of the features under different spectra within the group as the modal center feature of the spectrum. Finally, calculate the cross-center loss according to the calculated sample center features and modal center features within the group. At the same time, calculate the cross-entropy loss according to the comparison between the class prediction results of each branch and the true label; Step 1-4: After completing the above steps 1-1 to 1-3, add the calculated cross-center loss and the cross-entropy loss of each branch to obtain the total loss and perform backpropagation. Then, use the stochastic gradient descent algorithm to update all parameters in the entire model according to the total loss value; Then repeat steps 1-1 to 1-4 until the loss value converges, the network reaches the optimal performance, and the training ends to obtain the trained cross-consistency multi-stream network; Step 2: The testing stage includes the following steps: Step 2-Id=1: Load a sample of a vehicle from the multi-spectral vehicle re-identification dataset. The sample includes a visible light image, a near-infrared image, and a thermal infrared image, and input the sample into the trained cross-consistency multi-stream network obtained in Step 1; Step 2-2: Extract the middle-level features of the input sample through the first part of the cross-consistency multi-stream network, send the middle-level features into the corresponding adaptive layer normalization unit to adjust the feature distribution, and then send the adjusted middle-level features into the second part of the cross-consistency multi-stream network to obtain the corresponding final features. Then, splice the obtained three final features at the channel level, and the spliced features are used as the final feature representation of the input sample; Step 2-3: Divide the test set into a query set and a retrieval set, and extract the corresponding feature representations for all sample images in the retrieval set through steps 2-1 and 2-2; Step 2-4: Extract the corresponding feature representation for the single query vehicle sample input in the query set through steps 2-1 and 2-2, and compare it with the sample features in the retrieval set obtained in step 2-3 to determine whether the query vehicle is included in the retrieval set.

2. The multi-spectral vehicle re-identification method based on cross-consistency according to claim 1, wherein: The multi-spectral vehicle re-identification dataset includes RGBNT100 and RGBN300. If the RGBNT100 dataset is used, when reading the data, each sample includes a visible light image, a near-infrared image, and a thermal infrared image. When constructing the network model, the cross-consistency multi-stream network sets three convolutional neural networks. If the RGBN300 dataset is used, when reading the data, each sample includes a visible light image and a near-infrared image. When constructing the network model, the cross-consistency multi-stream network sets two convolutional neural networks.

3. The multi-spectral vehicle re-identification method based on cross-consistency according to claim 1, characterized in that: The multi-spectral vehicle re-identification dataset uses the MSVR310 dataset. The MSVR310 dataset includes 310 target vehicles and 2087 groups of samples. Each sample includes one visible light image, one near-infrared image, and one thermal infrared image, for a total of 6261 images. And each image includes a vehicle identity label, a sample view label, and a sample capture time label. When reading the data of the MSVR310 dataset, each sample includes a visible light image, a near-infrared image, and a thermal infrared image. When constructing the network model, the cross-consistency multi-stream network sets three convolutional neural networks.

4. The multi-spectral vehicle re-identification method based on cross-consistency according to claim 1, characterized in that: The first part of the convolutional neural network includes the first convolutional layer conv1, the first stacked layer layer1, and the second stacked layer layer2 to obtain the middle-level features of the input sample. Then, the distribution of the obtained middle-level features is adjusted through an adaptive layer normalization unit. The second part includes the third stacked layer layer3, the fourth stacked layer layer4, and the global average pooling layer to obtain the final features corresponding to the input sample. Among them, the middle-level features of each sample include middle-level visible light features, middle-level near-infrared features, and middle-level thermal infrared features. The final features of each sample include final visible light features, final near-infrared features, and final thermal infrared features.

5. The multi-spectral vehicle re-identification method based on cross-consistency according to claim 1, characterized in that: The specific method for the adaptive layer normalization unit to adjust the middle-level feature distribution is as follows: First, calculate the overall mean μ and standard deviation σ of the input middle-level features, and then subtract the mean from the features and divide by the standard deviation for standardization. At the same time, the original input features will pass through two independent factor learning modules to learn the scaling factor γ and the offset factor β respectively. Each factor learning module is composed of multiple convolutional layers and pooling layers stacked together. Through multi-layer convolution and pooling, a scalar is finally learned from the original input features. Multiply the standardized features by the scaling factor γ and then add the offset factor β to achieve re-scaling and offset of the features. Finally, the features after re-scaling and offset are the final output features.

Images