Vehicle re-identification method, system and equipment based on spatial-temporal feature fusion

Abstract

The invention belongs to the technical field of vehicle re-identification, and discloses a vehicle re-identification method, system and device based on spatial-temporal feature fusion. The method comprises the steps that monitoring videos of adjacent point positions in the same time period are obtained, and vehicles in the monitoring videos are detected and tracked in real time; extracting a vehicle image corresponding to each vehicle ID and associated attribute data thereof, and storing the vehicle image and the associated attribute data to form a structured data set; a new same vehicle ID is given to vehicles matched with the vehicle appearances in the monitoring videos of the adjacent point locations, the structured data set is updated, and an MIFS vehicle re-identification data set is obtained and preprocessed; and through the multi-modal spatial-temporal feature fusion network model, vehicle global spatial-temporal features and vehicle attribute features are extracted from all vehicle images, vector addition operation is carried out on the vehicle global spatial-temporal features and the vehicle attribute features, multi-modal vehicle features are obtained, and vehicle re-identification is carried out. According to the method, the data utilization efficiency can be optimized to efficiently model the space-time fusion features, so that the efficiency and accuracy of vehicle re-identification are improved.

Description

Technical Field

[0001] This invention belongs to the field of vehicle re-identification technology, specifically relating to a vehicle re-identification method, system, and device based on spatiotemporal feature fusion. Background Technology

[0002] Vehicle re-identification (ReID) is one of the core technologies in intelligent transportation and video surveillance, aiming to retrieve the same vehicle across cameras or time periods. In recent years, with the development of deep learning, vehicle re-identification technology has made significant progress, but it still faces many challenges.

[0003] Existing vehicle re-identification methods primarily rely on spatial feature extraction from single-frame images, utilizing convolutional neural networks (CNNs) or Transformer architectures to learn vehicle visual features (such as color, texture, and model). However, these methods suffer from limitations including insufficient utilization of temporal information, poor robustness to viewpoint changes, and susceptibility to occlusion and lighting interference. Insufficient utilization of temporal information means that single-frame images struggle to capture vehicle motion features (such as changes in direction and speed) and changes in appearance features (such as vehicle angle and lighting changes). While temporal information in consecutive frames can improve re-identification accuracy, most existing methods do not adequately model the temporal dimension. Poor robustness to viewpoint changes means that the appearance of the same vehicle differs significantly from different camera viewpoints, and relying solely on static spatial features can easily lead to mismatches. Susceptibility to occlusion and lighting interference means that feature extraction from single-frame images is unstable when occluded or under changing lighting conditions, while multi-frame information can mitigate these problems through temporal context.

[0004] To overcome these shortcomings, some studies have attempted to introduce temporal modeling, such as using three-dimensional (3D) convolutions or recurrent neural networks (RNNs) to process video sequences. However, these methods suffer from high computational complexity and low efficiency in spatiotemporal feature fusion. Furthermore, existing methods typically rely on cross-camera data correlation, neglecting the enhancing effect of temporal consistency of consecutive frames within the same camera on feature learning. Therefore, optimizing data utilization for efficient temporal modeling is a topic requiring further research. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a vehicle re-identification method, system, and device based on spatiotemporal feature fusion, which optimizes data utilization efficiency to efficiently model spatiotemporal fusion features, thereby improving the efficiency and accuracy of vehicle re-identification.

[0006] Specifically, the present invention is implemented using the following technical solutions.

[0007] On the one hand, the present invention provides a vehicle re-identification method based on spatiotemporal feature fusion, comprising:

[0008] Construct a MIFS (Multi-Interval Frame Sequence) vehicle re-identification dataset: Acquire surveillance videos from adjacent cameras within the same time period, detect and track vehicles appearing in the surveillance videos in real time, assign the same vehicle ID to the same vehicle in consecutive frames; extract vehicle images and their associated attribute data corresponding to each vehicle ID from the surveillance videos, save the vehicle images corresponding to each vehicle ID as image files, and save the associated attribute data in an XML file to form a structured dataset; identify vehicles with matching appearance shapes in the surveillance videos of adjacent cameras as the same vehicle, assign them a new vehicle ID, and update the saved image file and XML file of the vehicle to obtain the MIFS vehicle re-identification dataset.

[0009] The MIFS vehicle re-identification dataset is preprocessed as follows: the vehicle images are standardized in size; the vehicle images in the MIFS vehicle re-identification dataset are normalized by calculating the mean and variance of the image data; image enhancement operations are applied to the normalized vehicle images; the associated attribute data of each vehicle image are encoded separately and concatenated into a vehicle attribute vector; the vehicle attribute vectors corresponding to all vehicle images with the same vehicle ID are concatenated to obtain the concatenated vehicle attribute vector for that vehicle ID.

[0010] Using a multimodal spatiotemporal feature fusion network model, the vehicle's global spatiotemporal features and vehicle attribute features are extracted from the vehicle image corresponding to the vehicle ID. Vector addition is performed on the two to obtain and save the multimodal vehicle features of the vehicle ID. The multimodal vehicle features of the vehicle ID are then compared with the multimodal vehicle features of all other vehicle IDs that have been saved but not matched by the surveillance cameras at adjacent locations using a 1-norm calculation. If the distance between the two multimodal vehicle features is less than a preset distance threshold, they are matched as the same vehicle.

[0011] Further, the step of encoding each associated attribute data of the vehicle image and concatenating them into a vehicle attribute vector includes:

[0012] One-hot encoding is used to encode vehicle color and vehicle type; integer encoding is used to encode the lane where the vehicle is located; the vehicle's position in the image is floating-point data and is not encoded, so the original data is used.

[0013] Furthermore, the multimodal spatiotemporal feature fusion network model includes a global spatiotemporal feature extraction module, an attribute feature extraction module, and a multimodal feature fusion module;

[0014] The process of extracting vehicle global spatiotemporal features and vehicle attribute features from the vehicle image corresponding to the vehicle ID using a multimodal spatiotemporal feature fusion network model, and performing vector addition on the two to obtain the multimodal vehicle features of the vehicle ID specifically includes:

[0015] The global spatiotemporal feature extraction module extracts high-level spatial semantic information of vehicle images from multiple vehicle images arranged in chronological order, and fuses it with the temporal features of the vehicle during its driving process to obtain the global spatiotemporal features of the vehicle.

[0016] The attribute feature extraction module is used to extract vehicle attribute features from the concatenated vehicle attribute vector of all vehicle IDs.

[0017] The multimodal feature fusion module is used to perform vector addition on the vehicle's global spatiotemporal features and vehicle attribute features to obtain the multimodal vehicle features.

[0018] Furthermore, the global spatiotemporal feature extraction module includes a spatiotemporal feature fusion module and an attention mechanism module; the spatiotemporal feature fusion module includes a feature extraction network module and a dynamic duration spatiotemporal feature fusion module;

[0019] The feature extraction network module is an improved ResNet-50 network, which includes a head network, a backbone network, and a classification network. The head network accepts an image with an input dimension of (224, 224, 3) and outputs a feature map with a dimension of (112, 112, 96). The backbone network takes the output of the head network as input and performs feature extraction according to the method of reducing the first two dimensions of the feature map by a factor of 16 and expanding the third dimension by a factor of 8, to obtain spatial features with an output dimension of (7, 7, 768). The classification network reconstructs the spatial features with a dimension of (7, 7, 768) into a one-dimensional vector of (1, 37632) and maps it to a vector of (1, N) dimensions for classification, where N is the total number of vehicle IDs.

[0020] Furthermore, the global spatiotemporal feature extraction module extracts high-level spatial semantic information of vehicle images from multiple vehicle images arranged chronologically, and fuses it with the temporal features of the vehicle's movement to obtain the vehicle's global spatiotemporal features, specifically including:

[0021] 3-1-1) The improved ResNet-50 network is pre-trained to extract high-level semantic information for each vehicle image. The features of the vehicle image with normalized size (224, 224, 3) are extracted and the spatial features (7, 7, 768) are obtained.

[0022] The dynamic temporal-spatial feature fusion module described in 3-1-2) uses a location encoding method to encode the temporal sequence information and spatial information of each vehicle image, generating a location encoding vector with dimensions (7,7,768); the formula for location encoding is as follows:

[0023]

[0024]

[0025] Where PE represents the position encoding function; frame index represents the sequence number of the vehicle image corresponding to each vehicle ID, counting from 0; i is the dimension index, with even-numbered dimension indices of each location encoding vector calculated using a sine function and odd-numbered dimension indices calculated using a cosine function; d model The hidden layer dimension of the attention mechanism module is represented by 768; c is the channel number of the feature map, starting from 0. The number of c is the same as the number of channels of the spatial features extracted by the improved ResNet-50, which is a total of 768 channels.

[0026] 3-1-3) Add the location encoding vector to the spatial features to obtain the spatiotemporal fusion features of each vehicle image, with the dimensions still being (7,7,768).

[0027] 3-1-4) The spatiotemporal fusion features of multiple vehicle images are stitched together to obtain a stitched spatiotemporal fusion feature map;

[0028] 3-1-5) Using a single layer of residual convolutional network, the spliced ​​spatiotemporal fusion feature map is reduced in dimensionality to obtain a vector of (7,7,768) dimensions;

[0029] 3-1-6) The (7,7,768) dimension vector obtained by the dimensionality reduction is transformed into a (49,768) dimension vector by the flattening operation, where 49 represents the first and second dimensions of the dimensionality reduction vector expanded row by row;

[0030] The attention mechanism module described in 3-1-7) maps the (49,768)-dimensional vector obtained in 3-1-6) to a (1,768)-dimensional metric space to obtain the vehicle's global spatiotemporal features.

[0031] Furthermore, the attribute feature extraction module includes a fully connected network layer and an attention mechanism module: the extraction of vehicle attribute features from the concatenated vehicle attribute vectors corresponding to all vehicle IDs using the attribute feature extraction module specifically includes:

[0032] The attribute feature extraction module concatenates the vehicle attribute vectors corresponding to each vehicle image with the same vehicle ID to obtain the concatenated vehicle attribute vector for that vehicle ID; and extracts vehicle attribute features using the concatenated vehicle attribute vector.

[0033] The fully connected layer maps the concatenated vehicle attribute vector of the vehicle ID to a 768-dimensional feature space to obtain intermediate vehicle attribute features.

[0034] The attention mechanism module maps the intermediate state vehicle attribute features to a (1,768) dimensional metric space to obtain the vehicle attribute features.

[0035] Furthermore, the multimodal spatiotemporal feature fusion network model is obtained through the following training method:

[0036] 1) Data loading and batch processing

[0037] The MIFS vehicle re-identification dataset is divided into a training set and a test set according to a set ratio. A random sampling strategy is used to construct training batches. Each batch contains N vehicle IDs. For each vehicle ID, K consecutive vehicle images and their corresponding attribute codes are selected to form an image vector of (N, K, 224, 224, 3) and an attribute vector of (N, K×22).

[0038] 2) Phased training

[0039] First, freeze the feature extraction network module and train the multimodal spatiotemporal feature fusion network model until the mAP metric stabilizes; then unfreeze all network parameters and fine-tune them.

[0040] Using the joint loss L=L id +αL triplet The cross-entropy loss L id Used for monitoring vehicle ID classification tasks; triplet loss L triplet The metric space of the multimodal vehicle features is constrained; α is a hyperparameter representing the weight of the triplet loss; AdamW is selected as the optimizer, with an initial learning rate set and dynamically adjusted using a cosine annealing strategy;

[0041] 3) Training monitoring

[0042] After each training round, the mAP and Rank-1 accuracy of the test set are calculated. If the performance does not improve after a set number of consecutive rounds, the training is terminated early and the training result corresponding to the highest mAP value is saved as the optimal result of the model.

[0043] Furthermore, the training method for the multimodal spatiotemporal feature fusion network model also includes calculating the triplet loss L. triplet At that time, the distance between the multimodal vehicle features of every two vehicle IDs is calculated using the 1-norm; for each anchor sample, the sample of the same vehicle ID that is furthest away is selected as the most difficult positive sample, and the sample of different vehicle IDs that is closest away is selected as the most difficult negative sample.

[0044] On the other hand, the present invention also provides a vehicle re-identification system based on spatiotemporal feature fusion to implement the above-mentioned vehicle re-identification method based on spatiotemporal feature fusion. The vehicle re-identification system based on spatiotemporal feature fusion includes a data acquisition module, a data preprocessing module, and a multimodal spatiotemporal feature fusion network module.

[0045] The data acquisition module acquires surveillance videos from adjacent cameras within the same time period, detects and tracks vehicles appearing in the surveillance videos in real time, assigns the same vehicle ID to the same vehicle in consecutive frames, extracts vehicle images and associated attribute data corresponding to each vehicle ID from the surveillance videos, saves the vehicle images corresponding to each vehicle ID as image files, and saves the associated attribute data in an XML file to form a structured dataset, and identifies vehicles with matching appearance shapes in the surveillance videos of adjacent cameras as the same vehicle, assigns a new vehicle ID, and updates the saved image file and XML file of the vehicle to obtain the MIFS vehicle re-identification dataset.

[0046] The data preprocessing module standardizes the size of the vehicle images; normalizes the vehicle images in the MIFS vehicle re-identification dataset by calculating the mean and variance of the image data in the MIFS vehicle re-identification dataset; applies image enhancement operations to the normalized vehicle images; encodes each associated attribute data of the vehicle images and concatenates them into a vehicle attribute vector; and concatenates the vehicle attribute vectors corresponding to all vehicle images with the same vehicle ID to obtain the concatenated vehicle attribute vector for that vehicle ID.

[0047] The multimodal spatiotemporal feature fusion network module extracts spatial features from vehicle images using an improved ResNet-50 network, and combines this with a positional encoding method that reflects the temporal order of vehicle movement to generate spatiotemporal fusion features. It dynamically stitches together the spatiotemporal fusion features of multiple frames of vehicle images with the same vehicle ID, performs residual convolution dimensionality reduction, and then maps them to global spatiotemporal features of the vehicle through an attention mechanism. The stitched vehicle attribute vectors of the same vehicle ID are refined into vehicle attribute features by the attribute feature extraction module. The global spatiotemporal features of the vehicle and the vehicle attribute features are then added together to obtain and save the multimodal vehicle features of that vehicle ID. The multimodal vehicle features of that vehicle ID are compared with the multimodal vehicle features of all other vehicle IDs that have been saved but not matched by surveillance cameras at adjacent locations using a 1-norm calculation. If the distance between two multimodal vehicle features is less than a pre-set distance threshold, they are matched as the same vehicle.

[0048] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described vehicle re-identification method based on spatiotemporal feature fusion.

[0049] The beneficial effects of the vehicle re-identification method, system, and device based on spatiotemporal feature fusion of the present invention are as follows:

[0050] This invention discloses a vehicle re-identification method, system, and device based on spatiotemporal feature fusion. First, a vehicle re-identification dataset representing temporal relationships using multiple frames of images is constructed. A YOLOv11 object detection model and Deepsort tracking algorithm are employed to detect and track vehicles, thereby acquiring vehicle images. An innovative multimodal spatiotemporal feature fusion network is designed. Spatial features are extracted using a pre-trained feature extraction network model (e.g., an improved ResNet-50 network model), and spatiotemporal feature fusion is achieved by combining innovative sine and cosine positional encoding. An attention mechanism is introduced to optimize feature representation. The multimodal spatiotemporal feature fusion network integrates vehicle visual features and attribute features, employing a weighted fusion strategy to achieve multimodal feature complementarity, and combining triplet loss and cross-entropy loss for joint optimization. This method effectively solves the problem of vehicle appearance feature variations under multiple viewpoints, significantly improving the accuracy and robustness of vehicle re-identification.

[0051] To address the issue of changes in vehicle appearance features under multiple viewing angles and varying lighting conditions, this invention employs a positional encoding method and a dynamic feature map stitching method to efficiently integrate the spatial and temporal features of the vehicle's appearance during driving, thereby improving the network's robustness in responding to changes in viewing angle and lighting.

[0052] To balance performance and accuracy, the multimodal spatiotemporal feature fusion network of this invention can dynamically change the spatiotemporal fusion feature map splicing method extracted by the feature extraction network module to adapt to different number of input frames, thereby meeting the performance and accuracy requirements of different scenarios.

[0053] This invention relates to a vehicle re-identification method, system, and device based on spatiotemporal feature fusion. Combining the advantages of Transformer in capturing long-distance dependencies and convolutional networks in capturing local details, the multimodal spatiotemporal feature fusion network not only captures detailed information such as vehicle edge textures but also focuses on the positional distance relationships between vehicle components, thereby improving the accuracy of network recognition. Attached Figure Description

[0054] Figure 1 This is a system schematic diagram according to an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the method flow according to an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the MIFS vehicle re-identification dataset directory structure according to an embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram of vehicle image storage and lane line division within the detection frame according to an embodiment of the present invention.

[0058] Figure 5 These are schematic diagrams illustrating the same vehicle from different camera perspectives in embodiments of the present invention.

[0059] Figure 6 This is a schematic diagram of the multimodal spatiotemporal feature fusion network structure according to an embodiment of the present invention.

[0060] Figure 7 This is a schematic diagram of the Resnet-50 network improvement method according to an embodiment of the present invention.

[0061] Figure 8 This is a schematic diagram of dynamic duration feature map splicing and convolution in an embodiment of the present invention.

[0062] Figure 9 This is a schematic diagram of the number of bits and splicing of vehicle attribute codes according to an embodiment of the present invention. Detailed Implementation

[0063] The present invention will now be described in further detail with reference to the embodiments and the accompanying drawings.

[0064] One embodiment of the present invention is a vehicle re-identification system based on spatiotemporal feature fusion, such as... Figure 1 As shown, it includes a data acquisition module, a data preprocessing module, and a multimodal spatiotemporal feature fusion network module.

[0065] The data acquisition module acquires surveillance videos from adjacent cameras within the same time period, detects and tracks vehicles appearing in the videos in real time (e.g., detection based on the YOLOv11 detection model and tracking based on the Deepsort tracking algorithm), extracts vehicle images and associated attribute data (e.g., vehicle color, vehicle type, lane number, and vehicle position, i.e., the center coordinates of the bottom of the vehicle detection box) for each vehicle ID from the surveillance videos, and stores this data as a structured dataset. The vehicle images corresponding to each vehicle ID are saved as image files, and the associated attribute data is saved in an XML file, forming a structured dataset. The data acquisition module then identifies vehicles with matching appearance shapes from the surveillance videos of adjacent cameras as the same vehicle, assigns them new vehicle IDs, and updates the saved image files and XML files for that vehicle, obtaining the MIFS vehicle re-identification dataset.

[0066] The data preprocessing module standardizes the extracted vehicle images to a uniform size (e.g., 224×224 pixels), calculates the mean and variance of the image data in the MIFS dataset, and normalizes the vehicle images in the MIFS vehicle re-identification dataset based on these mean and variance. Image enhancement operations such as random cropping, rotation, and flipping are applied to the normalized vehicle images. The associated attribute data of each vehicle image are encoded separately and concatenated into a vehicle attribute vector. For example, vehicle color uses 11-bit one-hot encoding, vehicle type uses 8-bit one-hot encoding, lane location uses 1-bit integer encoding, and 2-bit floating-point values ​​are used as vehicle location information, concatenating them into a 22-dimensional vehicle attribute vector. The vehicle attribute vectors corresponding to multiple frames of the same vehicle are concatenated to obtain a concatenated vehicle attribute vector, with a dimension equal to the number of vehicle images for that vehicle ID multiplied by 22. Figure 9 The method shown is used for splicing.

[0067] The multimodal spatiotemporal feature fusion network module extracts spatial features (e.g., 7×7×768 dimensions) from individual vehicle images in the surveillance video using an improved ResNet-50 network. Combined with a location encoding method reflecting the temporal order of vehicle movement, it generates spatiotemporal fusion features. The spatiotemporal fusion features of multiple frames of vehicle images with the same vehicle ID are dynamically stitched together, and after dimensionality reduction via residual convolution, they are mapped to global vehicle spatiotemporal features (1×768 dimensions) through an attention mechanism. The stitched vehicle attribute vectors for the same vehicle ID are then refined into vehicle attribute features (1×768) by the attribute feature extraction module. The global vehicle spatiotemporal features and the vehicle attribute features are added together to obtain the multimodal vehicle features for that vehicle ID, which are then saved. The multimodal vehicle features of that vehicle ID are then compared with the multimodal vehicle features of all other unmatched vehicle IDs saved by surveillance cameras at adjacent locations using a 1-norm calculation. If the distance between two multimodal vehicle features is less than a pre-set distance threshold, they are matched as the same vehicle.

[0068] The vehicle re-identification method based on spatiotemporal feature fusion of the present invention, such as Figure 2 As shown, it includes the following steps:

[0069] I. Constructing the MIFS vehicle re-identification dataset.

[0070] The system acquires surveillance videos from adjacent cameras within the same time period, detects and tracks vehicles appearing in the videos in real time, and assigns the same vehicle ID to the same vehicle in consecutive frames. It extracts vehicle images and associated attribute data corresponding to each vehicle ID from the surveillance videos, saves the vehicle images corresponding to each vehicle ID as image files, and saves the associated attribute data in an XML file, forming a structured dataset. Vehicles whose exterior shapes (e.g., side window shape, rear window shape, rear shape, and taillight shape) match those in the surveillance videos from adjacent cameras are considered the same vehicle, assigned a new vehicle ID, and the saved image files and XML files for that vehicle are updated, thus obtaining the MIFS vehicle re-identification dataset.

[0071] MIFS Vehicle Re-identification Dataset Catalog as follows Figure 3 As shown, vehicle images and their associated attribute data are extracted from the surveillance video of each camera and stored as image files (.jpg files) and XML files (.xml files), respectively. The construction of the MIFS vehicle re-identification dataset specifically includes:

[0072] 1-1) Acquire vehicle image

[0073] Two surveillance videos from adjacent locations within the same time period are acquired. Using the Deepsort tracking algorithm based on a target detection model (e.g., YOLOv11 target detection model), vehicles appearing in the two surveillance videos from the adjacent locations within the same time period are detected. Based on data such as target appearance, detection box size and position, and vehicle speed, the vehicles in the videos are tracked, and different vehicles are assigned different vehicle IDs (identifications).

[0074] Preferably, the YOLOv11 target detection model detects vehicles appearing in each frame of the surveillance video and passes the detected vehicles to the Deepsort tracking algorithm in the form of detection boxes (e.g., rectangles represented by the coordinates of the top left and bottom right corners). Based on the detection boxes, target appearance within the detection boxes, target movement speed, and other data, the Deepsort tracking algorithm matches vehicles detected by the YOLOv11 target detection model in consecutive frames, assigning the same vehicle ID to the same vehicle and different vehicle IDs to different vehicles, with vehicle IDs starting from 0 and increasing sequentially.

[0075] 1-2) Obtain related attribute data

[0076] Set the time interval for saving the corresponding vehicle images (referred to as vehicle images) within the detection frame, and the total number of vehicle images to be saved; save the vehicle image corresponding to each vehicle ID as an image file; save the vehicle ID, the center coordinate data at the bottom of the vehicle detection frame, and the lane number data of the vehicle in the tracked vehicle images in an XML file, such as saving it as a JSON object in an XML file. The lane number data of the vehicle is determined based on the center coordinate data at the bottom of the detection frame and the pre-defined lane lines, such as... Figure 4 As shown, each XML file corresponds one-to-one with a vehicle image file.

[0077] 1-3) Vehicle color classification and vehicle type classification

[0078] For each saved image file, vehicle color and vehicle type are classified, and the classification results are saved in the corresponding XML file according to the vehicle ID. Open-source vehicle type and color classification models can be used to perform vehicle color and vehicle type classification for each saved vehicle image file.

[0079] 1-4) Matching the vehicle's exterior shape with the surveillance video frames of two cameras to identify the same vehicle.

[0080] Because the same vehicle appears in the surveillance video footage from cameras at different locations at different times, the Deepsort tracking algorithm may assign different IDs to the same vehicle. Therefore, it uses the vehicle's exterior shape (e.g., the shape of the side windows, rear window, rear end, and taillights) to match vehicle images from two camera perspectives. Vehicles whose exterior shapes match in the surveillance videos from adjacent locations are considered the same vehicle and assigned a new vehicle ID. Figure 5 As shown, the system updates the saved image files and XML files of the corresponding vehicles to obtain the MIFS vehicle re-identification dataset. New vehicle IDs start from 0 and increase sequentially. The following section uses an example of 700 vehicle IDs in the MIFS vehicle re-identification dataset to illustrate the solution of this invention.

[0081] 2. Preprocessing the MIFS vehicle re-identification dataset.

[0082] The image files and XML files of the MIFS vehicle re-identification dataset are preprocessed, including unifying the size of the vehicle images; calculating the mean and variance of the image data in the MIFS vehicle re-identification dataset, normalizing the vehicle images based on the mean and variance, and applying image enhancement operations to the normalized vehicle images; encoding the associated attribute data of each vehicle image and concatenating them into a vehicle attribute vector; and concatenating the vehicle attribute vectors corresponding to all vehicle images with the same vehicle ID to obtain the concatenated vehicle attribute vector for that vehicle ID. The specific steps are as follows:

[0083] 2-1) Calculate the mean and standard values ​​of the image data in the MIFS vehicle re-identification dataset.

[0084] Calculate the mean of the image data based on the image files of the MIFS vehicle re-identification dataset. c and variance std c This facilitates subsequent normalization processing of the image data. The mean of the image data. c The calculation formula is as follows:

[0085]

[0086] Where c represents the R, G, or B color channels, n is the total number of images, and w i h is the width of the i-th image. i I is the height of the i-th image. i,c (x,y) is the pixel value at position (x,y) on channel c of the i-th image, and N is the total number of pixels:

[0087]

[0088] The mean of the image data is obtained. c Then, the variance std of the image data c The calculation formula is as follows:

[0089] .

[0090] 2-2) The vehicle images are standardized in size, and the vehicle images in the MIFS vehicle re-identification dataset are normalized by using the mean and variance of the image data in the dataset.

[0091] To resize images in the MIFS vehicle re-identification dataset, uniformly adjust their dimensions to a set value, such as (224, 224, 3), representing the image's width, height, and number of color channels, respectively. This can be done using the `Reize` function from the `torchvision` (torch vision function toolkit) package. Alternatively, the mean value of the image data can be used. c and variance std c The image data is normalized to obtain normalized image data. The pixel value of the normalized image data is... out for:

[0092]

[0093] Among them, pixel in These are the original pixel values ​​of the image.

[0094] 2-3) Apply image enhancement operations to the normalized vehicle images.

[0095] Image enhancement operations are performed on normalized image data, including random cropping, random rotation, and random horizontal flipping.

[0096] You can use the image processing functions included in the open-source torchvision toolkit to perform random cropping, random rotation, and random horizontal flipping operations.

[0097] Specifically, the random cropping operation can be implemented using the random cropping function in the torchvision toolkit. This involves setting the cropping region size parameter, randomly selecting a region of the same size as the cropping region from the original input standard-sized image, and then rescaling the cropped image to the standard size. This operation effectively enhances the model's ability to recognize local vehicle features.

[0098] Random rotation can be implemented using the random rotation function in the torchvision toolkit: Using the image center as the rotation base point, a rotation angle range (RotationDegree) is set, and the rotation angle of the image changes randomly within the range [-RotationDegree, RotationDegree]. Random rotation moderately increases the change in viewing angle while keeping the main structure of the vehicle unchanged.

[0099] The random horizontal flip operation can be implemented using the random horizontal flip function in the torchvision toolkit: set the flip probability, and perform a horizontal mirror flip operation when the value generated by the random number generator is less than the flip probability. This operation simulates the scenario characteristics of vehicles traveling in both directions in a real road environment.

[0100] 2-4) Encode each associated attribute data of the vehicle image separately and concatenate them into a vehicle attribute vector.

[0101] The XML file of the MIFS vehicle re-identification dataset records the vehicle ID, vehicle color, vehicle type, lane number, and vehicle position in the image. Since vehicle color, vehicle type, lane number, and position play a significant role in vehicle re-identification, these vehicle attributes are encoded. Preferably, one-hot encoding is used to encode the vehicle color and vehicle type; integer encoding is used to encode the lane number; and the vehicle position, being floating-point data, is not encoded and is retained as is. The final result is an 11-bit one-hot encoding for vehicle color, an 8-bit one-hot encoding for vehicle type, a 1-bit lane number, and a 2-bit position encoding, totaling 22 bits.

[0102] Third, using a multimodal spatiotemporal feature fusion network model, the global spatiotemporal features and vehicle attribute features of the vehicle are extracted from the vehicle image corresponding to the vehicle ID. Vector addition is performed on these two features to obtain and save the multimodal vehicle features for that vehicle ID. The multimodal vehicle features corresponding to that vehicle ID are then compared with the multimodal vehicle features of all other unmatched vehicle IDs saved by adjacent monitoring cameras using a 1-norm calculation. Based on a pre-set threshold, it is determined whether two vehicles match as the same vehicle. If the distance between the two multimodal vehicle features is less than the pre-set distance threshold, the two vehicles are considered a match and are considered the same vehicle.

[0103] like Figure 6As shown, the multimodal spatiotemporal feature fusion network model includes a global spatiotemporal feature extraction module, an attribute feature extraction module, and a multimodal feature fusion module. Here, x is the vehicle ID, representing a unique identifier for a vehicle in the dataset; id x_0 represents the first vehicle image of vehicle ID x; id x_1 represents the second vehicle image of vehicle ID x; and so on. The specific steps are as follows:

[0104] 3-1) The global spatiotemporal feature extraction module extracts high-level spatial semantic information (including vehicle texture, color, direction, etc.) from multiple vehicle images arranged in chronological order, and efficiently fuses the extracted high-level spatial semantic information with the temporal features of the vehicle during its driving process to obtain the global spatiotemporal features of the vehicle.

[0105] Preferably, in another embodiment, the global spatiotemporal feature extraction module includes a spatiotemporal feature fusion module and an attention mechanism module. The spatiotemporal feature fusion module includes a feature extraction network module and a dynamic duration spatiotemporal feature fusion module.

[0106] The feature extraction network module is an improved ResNet-50 network, derived from improvements to the ResNet-50 network, as follows: Figure 7 As shown.

[0107] The ResNet-50 network consists of a head network, a backbone network, and a classifier network. The head network receives an image with input dimensions of (224, 224, 3) and outputs a feature map with dimensions of (112, 112, 64). The backbone network takes the output of the head network as input and performs feature extraction by scaling down the first two dimensions by a factor of 16 and scaling up the third dimension by a factor of 8, resulting in a feature map with output dimensions of (7, 7, 512). The classifier network reconstructs the (7, 7, 512)-dimensional feature map into a one-dimensional vector of (1, 25088) (25088 is obtained from 7×7×512) and maps it to a classification vector of dimension (1, N) (N represents the number of vehicle IDs).

[0108] The improved ResNet-50 network includes a head network, a backbone network, and a classification network. The output dimension of the head network is changed to (112, 112, 96). The backbone network receives the output of the head network as input and outputs a feature map with a dimension of (7, 7, 768) according to the feature extraction method. The classification network reconstructs the feature map with a dimension of (7, 7, 768) into a one-dimensional vector of (1, 37632) (37632 is obtained from 7×7×768), and still maps it to a vector of a dimension of (1, 700) for classification. That is, in the improved ResNet-50 network, the head network accepts an image input with dimensions (224,224,3) and outputs a feature map with dimensions (112,112,96). The backbone network takes the output of the head network as input and performs feature extraction according to the method of reducing the first two dimensions of the feature map by a factor of 16 and expanding the third dimension by a factor of 8, to obtain spatial features with output dimensions (7,7,768). The classification network reconstructs the spatial features with dimensions (7,7,768) into a one-dimensional vector of dimensions (1,37632) and maps it to a vector of dimensions (1,N) for classification, where N is the total number of vehicle IDs, for example, N is 700.

[0109] Preferably, in another embodiment, the improved ResNet-50 network is trained by the following steps:

[0110] The image data from the MIFS vehicle re-identification dataset was randomly divided into training and test sets in an 8:2 ratio, with vehicle IDs as the class, resulting in 700 classes. Cross-entropy loss was used as the loss function for training the improved ResNet-50 network.

[0111] The global spatiotemporal feature extraction module extracts high-level spatial semantic information of vehicle images from multiple vehicle images arranged chronologically, and fuses it with the temporal features of the vehicle's movement to obtain the vehicle's global spatiotemporal features. The specific steps are as follows:

[0112] 3-1-1) High-level semantic information of each vehicle image is extracted using a pre-trained improved ResNet-50 network, including vehicle texture, color, orientation, etc. Features of the vehicle image with normalized size (224, 224, 3) are extracted, and spatial features of (7, 7, 768) (representing the width, height, and number of channels of the feature map, respectively) are obtained. In the feature extraction network module, the classification network of the pre-trained improved ResNet-50 network is removed, and only the head network and the backbone network are used.

[0113] 3-1-2) The dynamic temporal-spatial feature fusion module uses a location encoding method to encode the temporal and spatial information of each vehicle image; the location encoding generates a location vector with dimensions (7,7,768). The location encoding formula is as follows:

[0114]

[0115]

[0116] Where PE (Position Encoding) represents the position encoding function; frame index represents the sequence number of the vehicle image for each vehicle ID, counting from 0; i is the dimension index, with even-numbered dimension indices of each location encoding vector using a sine function and odd-numbered dimension indices using a cosine function; d model represents the hidden layer dimension of the attention mechanism module, with a value of 768; c is the channel number of the feature map, starting from 0. The count of c is the same as the number of channels of the spatial features extracted by the improved ResNet-50, a total of 768 channels.

[0117] 3-1-3) Add the location encoding vector to the spatial features to obtain the spatiotemporal fusion features of each vehicle image, with the dimensions still being (7,7,768).

[0118] 3-1-4) The spatiotemporal fusion features of multiple vehicle images are stitched together to obtain a stitched spatiotemporal fusion feature map, as shown in the stitching method. Figure 8 As shown, different stitching methods are used depending on the number of vehicle images captured. Figure 8 (a) in the image indicates the stitching method when the number of vehicle image cropped is 4. Figure 8 (b) in the figure represents the stitching method when the number of vehicle image cropped is 9. Figure 8 (b) in comparison to Figure 8 (a) in the example stitches together more features (when the vehicle image cropping interval is fixed, the more vehicle images are cropped and the longer the time span), the richer the temporal features, but the computational cost is greater.

[0119] 3-1-5) Use a single residual convolutional network to reduce the dimensionality of the spatiotemporal fusion feature map to (7, 7, 768). Adjust the stride of the residual convolutional network according to the number of vehicle image segments. For example... Figure 8 As shown in (a) above, the spatiotemporal fusion feature map obtained by concatenation has dimensions (14, 14, 768). Setting the stride of the residual convolutional network to 2 reduces the dimension to (7, 7, 768). Figure 8 As shown in (b), the spatiotemporal fusion feature map obtained by splicing has a dimension of (21,21,768). The stride of the residual convolutional network is set to 3, which reduces the dimension to (7,7,768).

[0120] 3-1-6) The (7,7,768) dimension vector obtained by the dimension reduction is transformed into a (49,768) dimension vector by the flatten operation, where 49 represents the first and second dimensions of the dimension reduction vector expanded row by row.

[0121] The attention mechanism module (3-1-7) maps the (49,768) dimension vector obtained in (3-1-6) to the (1,768) dimension metric space to obtain the vehicle's global spatiotemporal features.

[0122] 3-2) Use the attribute feature extraction module to extract vehicle attribute features from the concatenated vehicle attribute vector of all vehicle IDs.

[0123] The attribute feature extraction module consists of a fully connected network layer and an attention mechanism module:

[0124] like Figure 9 As shown, the attribute feature extraction module concatenates the vehicle attribute vectors (including 11-bit vehicle color one-hot encoding, 8-bit vehicle type one-hot encoding, 2-bit vehicle position encoding in the image, and 1-bit vehicle lane type encoding) corresponding to each vehicle image of the same vehicle ID to obtain the concatenated vehicle attribute vector of that vehicle ID, with the dimension being the number of vehicle images of that vehicle ID multiplied by 22.

[0125] The fully connected layer maps the concatenated vehicle attribute vector of the vehicle ID to a 768-dimensional feature space to obtain intermediate vehicle attribute features.

[0126] The attention mechanism module maps the vehicle attribute features of the intermediate state to a (1,768)-dimensional metric space to obtain the vehicle attribute features.

[0127] 3-3) Using the multimodal feature fusion module, vector addition is performed on the vehicle's global spatiotemporal features and vehicle attribute features to obtain and save the multimodal vehicle features for the vehicle ID. The multimodal vehicle features corresponding to the vehicle ID are then compared with the multimodal vehicle features of all other vehicle IDs that have been saved but not matched by adjacent monitoring cameras using a 1-norm calculation. If the distance between two multimodal vehicle features is less than a preset distance threshold, they are matched as the same vehicle.

[0128] The multimodal feature fusion module includes vehicle global spatiotemporal features (F... global_spatial_temporal ) and vehicle attribute characteristics (F attribute The vector addition module, attention mechanism module, and classifier are as follows:

[0129] Both the vehicle global spatiotemporal features and vehicle attribute features are vectors of dimension (1, 768). Vector addition is performed to obtain the multimodal vehicle features (F...). multimodalThe multimodal vehicle feature dimension remains (1,768), see the following formula.

[0130]

[0131] Where β is a hyperparameter representing the weight of vehicle attribute features.

[0132] The attention mechanism module maps the (1,768)-dimensional multimodal vehicle features to a (1,768)-dimensional metric space, resulting in new multimodal vehicle features. On one hand, it uses triplet loss to constrain the metric space of the multimodal vehicle features. On the other hand, it inputs the multimodal vehicle features into a fully connected classification network for classification, obtaining classification vectors and outputting the vehicle classification results.

[0133] The multimodal vehicle features are vectors of dimension (1,768), and a triplet loss L is used. triplet The triplet loss is defined as follows:

[0134]

[0135] Where 'a' represents the anchor example, 'p' represents the positive sample, 'n' represents the negative sample, and ||1 represents the 1-norm; 'margin' is a hyperparameter that is adjusted according to the dataset and experimental requirements. A larger margin value results in a greater distance between positive and negative samples, leading to better classification performance, but makes network training more difficult and slows down convergence. Conversely, a smaller margin value results in a smaller distance between positive and negative samples, reducing classification performance, but makes network training easier and faster.

[0136] The classifier is a one-layer fully connected classification network with an input vector of (1, 768) dimensions and an output classification vector of (1, 700) dimensions. The second dimension (700) of the output vector represents the number of vehicle IDs in the MIFS vehicle re-identification dataset. The classification vector uses cross-entropy loss Li. id The constraint is that the cross-entropy loss is defined as:

[0137]

[0138] Where M represents the total number of samples in the same batch for calculating cross-entropy loss, C represents the total number of classes, and y i,c p represents the one-hot encoding (0 or 1) of the true label of the i-th sample in class c. i,c This represents the predicted probability that the i-th sample belongs to class c (the output value of the fully connected layer, ranging from 0 to 1).

[0139] The goal of network training is to reduce the value of the joint loss L to a set value. The joint loss L is defined as:

[0140]

[0141] Where α is the weight of the triplet loss, L id For cross-entropy loss, L triplet This is the loss of the triplet.

[0142] Preferably, in another embodiment, a training method for a multimodal spatiotemporal feature fusion network model is provided, wherein the multimodal spatiotemporal feature fusion network model is obtained through the following training method:

[0143] 1) Data loading and batch processing

[0144] The MIFS vehicle re-identification dataset is divided into training and testing sets according to a set ratio (e.g., 8:2), and a random sampling strategy is used to construct training batches. Each batch contains N vehicle IDs, and for each vehicle ID, K consecutive vehicle images and their corresponding attribute codes are randomly selected to form an image vector of (N, K, 224, 224, 3) and an attribute vector of (N, K×22).

[0145] 2) Phased training

[0146] First, freeze the feature extraction network module (the improved ResNet-50 network after freezing only includes the head network and the backbone network), and train the multimodal spatiotemporal feature fusion network model until the mAP index tends to stabilize (for example, the mAP index changes by no more than 1% in 10 consecutive training rounds); then unfreeze all network parameters and fine-tune them.

[0147] Using the joint loss function L=L id +αL triplet The cross-entropy loss L id Used for monitoring vehicle ID classification tasks; triplet loss L triplet The metric space of the multimodal fusion features is constrained; α is a hyperparameter representing the weight of the triplet loss. The optimizer is AdamW (Adaptive Moment Estimation with Weight Decay), with an initial learning rate set and dynamically adjusted using a cosine annealing strategy.

[0148] Preferably, in another embodiment, when calculating the triplet loss L triplet When calculating the distance between two vectors, the 1-norm is used; for each anchor sample, the sample with the same vehicle ID that is furthest away is selected as the most difficult positive sample, and the sample with different vehicle IDs that is closest away is selected as the most difficult negative sample, in order to improve feature discriminativeness.

[0149] Preferably, in another embodiment, label smoothing (coefficient 0.1) and random dropping (probability 0.3) are used to prevent overfitting.

[0150] 3) Training monitoring

[0151] After each training round, the mAP (mean Average Precision) and Rank-1 accuracy of the test set are calculated. If the performance does not improve after a set number of rounds (e.g., 10 rounds), the training is terminated early and the training result corresponding to the highest mAP value is saved as the optimal result of the model, i.e., the optimal model.

[0152] This invention discloses a vehicle re-identification method, system, and device based on spatiotemporal feature fusion. First, a vehicle re-identification dataset representing temporal relationships using multiple frames of images is constructed. A YOLOv11 object detection model and Deepsort tracking algorithm are employed to detect and track vehicles, thereby acquiring vehicle images. An innovative multimodal spatiotemporal feature fusion network is designed. Spatial features are extracted using a pre-trained feature extraction network model (e.g., an improved ResNet-50 network model), and spatiotemporal feature fusion is achieved by combining innovative sine and cosine positional encoding. An attention mechanism is introduced to optimize feature representation. The multimodal spatiotemporal feature fusion network integrates vehicle visual features and attribute features, employing a weighted fusion strategy to achieve multimodal feature complementarity, and combining triplet loss and cross-entropy loss for joint optimization. This method effectively solves the problem of vehicle appearance feature variations under multiple viewpoints, significantly improving the accuracy and robustness of vehicle re-identification.

[0153] To address the issue of changes in vehicle appearance features under multiple viewing angles and varying lighting conditions, this invention employs a positional encoding method and a dynamic feature map stitching method to efficiently integrate the spatial and temporal features of the vehicle's appearance during driving, thereby improving the network's robustness in responding to changes in viewing angle and lighting.

[0154] To balance performance and accuracy, the multimodal spatiotemporal feature fusion network of this invention can dynamically change the spatiotemporal fusion feature map splicing method extracted by the feature extraction network module to adapt to different number of input frames, thereby meeting the performance and accuracy requirements of different scenarios.

[0155] This invention relates to a vehicle re-identification method, system, and device based on spatiotemporal feature fusion. Combining the advantages of Transformer in capturing long-distance dependencies and convolutional networks in capturing local details, the multimodal spatiotemporal feature fusion network not only captures detailed information such as vehicle edge textures but also focuses on the positional distance relationships between vehicle components, thereby improving the accuracy of network recognition.

[0156] In some embodiments, certain aspects of the above-described techniques may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly implemented on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the above-described techniques. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, cache, random access memory (RAM), or other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats interpreted or otherwise executed by one or more processors.

[0157] Computer-readable storage media can include any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., optical discs (CDs), digital versatile optical discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard disks), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS) based storage media. Computer-readable storage media can be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard disk drive), removably attached to a computing system (e.g., an optical disc or universal serial bus-based (USB) flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0158] Please note that not all activities or elements described in the general description above are essential, a particular activity or part of the apparatus may not be essential, and one or more further activities or included elements may be performed in addition to those described. Furthermore, the order in which the activities are listed does not necessarily represent the order in which they are performed. Moreover, these concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of this disclosure as set forth in the following claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are included within the scope of this disclosure.

[0159] The benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, any benefits, advantages, solutions to problems, and any features that may lead to or make any benefit, advantage, or solution more apparent should not be construed as critical, essential, or essential features of any or all claims. Furthermore, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter can be modified and implemented in different but equivalent ways that would be apparent to those skilled in the art benefiting from the teachings herein. There is no intention to limit the details of the constructions or designs shown herein other than those described in the claims. Therefore, it is apparent that the specific embodiments disclosed above can be altered or modified, and all such changes are considered to be within the scope of the disclosed subject matter.

Claims

1. A vehicle re-identification method based on spatiotemporal feature fusion, characterized in that, include: Construct the MIFS vehicle re-identification dataset: acquire surveillance videos from adjacent cameras in the same time period, detect and track vehicles appearing in the surveillance videos in real time, and assign the same vehicle ID to the same vehicle in consecutive frames. Extract vehicle images and associated attribute data corresponding to each vehicle ID from the surveillance video. Save the vehicle images corresponding to each vehicle ID as image files and the associated attribute data in an XML file to form a structured dataset. Treat vehicles with matching vehicle appearance shapes in the surveillance videos of adjacent cameras as the same vehicle, assign them a new vehicle ID, and update the saved image file and XML file of the vehicle to obtain the MIFS vehicle re-identification dataset. The MIFS vehicle re-identification dataset is preprocessed as follows: the vehicle images are standardized in size; the vehicle images in the MIFS vehicle re-identification dataset are normalized by calculating the mean and variance of the image data; image enhancement operations are applied to the normalized vehicle images; the associated attribute data of each vehicle image are encoded separately and concatenated into a vehicle attribute vector; the vehicle attribute vectors corresponding to all vehicle images with the same vehicle ID are concatenated to obtain the concatenated vehicle attribute vector for that vehicle ID. Using a multimodal spatiotemporal feature fusion network model, the vehicle's global spatiotemporal features and vehicle attribute features are extracted from the vehicle image corresponding to the vehicle ID. Vector addition is performed on the two to obtain and save the multimodal vehicle features of the vehicle ID. The multimodal vehicle features of the vehicle ID are then compared with the multimodal vehicle features of all other vehicle IDs that have been saved but not matched by the surveillance cameras at adjacent locations using a 1-norm calculation. If the distance between the two multimodal vehicle features is less than a preset distance threshold, they are matched as the same vehicle.

2. The vehicle re-identification method based on spatiotemporal feature fusion according to claim 1, characterized in that, The step of encoding each associated attribute data of the vehicle image and concatenating them into a vehicle attribute vector includes: One-hot encoding is used to encode vehicle color and vehicle type; integer encoding is used to encode the lane where the vehicle is located; the vehicle's position in the image is floating-point data and is not encoded, so the original data is used.

3. The vehicle re-identification method based on spatiotemporal feature fusion according to claim 2, characterized in that, The multimodal spatiotemporal feature fusion network model includes a global spatiotemporal feature extraction module, an attribute feature extraction module, and a multimodal feature fusion module; The process of extracting vehicle global spatiotemporal features and vehicle attribute features from the vehicle image corresponding to the vehicle ID using a multimodal spatiotemporal feature fusion network model, and performing vector addition on the two to obtain the multimodal vehicle features of the vehicle ID specifically includes: The global spatiotemporal feature extraction module extracts high-level spatial semantic information of vehicle images from multiple vehicle images arranged in chronological order, and fuses it with the temporal features of the vehicle during its driving process to obtain the global spatiotemporal features of the vehicle. The attribute feature extraction module is used to extract vehicle attribute features from the concatenated vehicle attribute vector of all vehicle IDs. The multimodal feature fusion module is used to perform vector addition on the vehicle's global spatiotemporal features and vehicle attribute features to obtain the multimodal vehicle features.

4. The vehicle re-identification method based on spatiotemporal feature fusion according to claim 3, characterized in that, The global spatiotemporal feature extraction module includes a spatiotemporal feature fusion module and an attention mechanism module; the spatiotemporal feature fusion module includes a feature extraction network module and a dynamic duration spatiotemporal feature fusion module. The feature extraction network module is an improved ResNet-50 network, which includes a head network, a backbone network, and a classification network. The head network accepts an image with an input dimension of (224, 224, 3) and outputs a feature map with a dimension of (112, 112, 96). The backbone network takes the output of the head network as input and performs feature extraction according to the method of reducing the first two dimensions of the feature map by a factor of 16 and expanding the third dimension by a factor of 8, to obtain spatial features with an output dimension of (7, 7, 768). The classification network reconstructs the spatial features with a dimension of (7, 7, 768) into a one-dimensional vector of (1, 37632) and maps it to a vector of (1, N) dimensions for classification, where N is the total number of vehicle IDs.

5. The vehicle re-identification method based on spatiotemporal feature fusion according to claim 4, characterized in that, The global spatiotemporal feature extraction module extracts high-level spatial semantic information of vehicle images from multiple vehicle images arranged chronologically, and fuses it with the temporal features of the vehicle's movement to obtain the vehicle's global spatiotemporal features, specifically including: 3-1-1) The improved ResNet-50 network is pre-trained to extract high-level semantic information for each vehicle image. The features of the vehicle image with normalized size (224, 224, 3) are extracted and the spatial features (7, 7, 768) are obtained. The dynamic temporal-spatial feature fusion module described in 3-1-2) uses a location encoding method to encode the temporal sequence information and spatial information of each vehicle image, generating a location encoding vector with dimensions (7,7,768); the formula for location encoding is as follows: Where PE represents the position encoding function; frame index represents the sequence number of the vehicle image corresponding to each vehicle ID, counting from 0; i is the dimension index, with even-numbered dimension indices of each location encoding vector calculated using a sine function and odd-numbered dimension indices calculated using a cosine function; d model The hidden layer dimension of the attention mechanism module is represented by 768; c is the channel number of the feature map, starting from 0. The number of c is the same as the number of channels of the spatial features extracted by the improved ResNet-50, which is a total of 768 channels. 3-1-3) Add the location encoding vector to the spatial features to obtain the spatiotemporal fusion features of each vehicle image, with the dimensions still being (7,7,768). 3-1-4) The spatiotemporal fusion features of multiple vehicle images are stitched together to obtain a stitched spatiotemporal fusion feature map; 3-1-5) Using a single layer of residual convolutional network, the spliced ​​spatiotemporal fusion feature map is reduced in dimensionality to obtain a vector of (7,7,768) dimensions; 3-1-6) The (7,7,768) dimension vector obtained by the dimensionality reduction is transformed into a (49,768) dimension vector by the flattening operation, where 49 represents the first and second dimensions of the dimensionality reduction vector expanded row by row; The attention mechanism module described in 3-1-7) maps the (49,768)-dimensional vector obtained in 3-1-6) to a (1,768)-dimensional metric space to obtain the vehicle's global spatiotemporal features.

6. The vehicle re-identification method based on spatiotemporal feature fusion according to claim 3, characterized in that, The attribute feature extraction module includes a fully connected network layer and an attention mechanism module. Specifically, extracting vehicle attribute features from the concatenated vehicle attribute vectors corresponding to all vehicle IDs using the attribute feature extraction module includes: The attribute feature extraction module concatenates the vehicle attribute vectors corresponding to each vehicle image with the same vehicle ID to obtain the concatenated vehicle attribute vector for that vehicle ID; and extracts vehicle attribute features using the concatenated vehicle attribute vector. The fully connected layer maps the concatenated vehicle attribute vector of the vehicle ID to a 768-dimensional feature space to obtain intermediate vehicle attribute features. The attention mechanism module maps the intermediate state vehicle attribute features to a (1,768) dimensional metric space to obtain the vehicle attribute features.

7. The vehicle re-identification method based on spatiotemporal feature fusion according to claim 1, characterized in that, The multimodal spatiotemporal feature fusion network model is obtained through the following training method: 1) Data loading and batch processing The MIFS vehicle re-identification dataset is divided into a training set and a test set according to a set ratio. A random sampling strategy is used to construct training batches. Each batch contains N vehicle IDs. For each vehicle ID, K consecutive vehicle images and their corresponding attribute codes are selected to form an image vector of (N, K, 224, 224, 3) and an attribute vector of (N, K×22). 2) Phased training First, freeze the feature extraction network module and train the multimodal spatiotemporal feature fusion network model until the mAP metric stabilizes; then unfreeze all network parameters and fine-tune them. Using the joint loss L=L id +αL triplet The cross-entropy loss L id Used for monitoring vehicle ID classification tasks; triplet loss L triplet The metric space constrains the multimodal vehicle features; α is a hyperparameter representing the weight of the triplet loss; The optimizer used is AdamW, with an initial learning rate set and a cosine annealing strategy used for dynamic adjustment. 3) Training monitoring After each training round, the mAP and Rank-1 accuracy of the test set are calculated. If the performance does not improve after a set number of consecutive rounds, the training is terminated early and the training result corresponding to the highest mAP value is saved as the optimal result of the model.

8. The vehicle re-identification method based on spatiotemporal feature fusion according to claim 7, characterized in that, This also includes calculating the triplet loss L. triplet At that time, the distance between the multimodal vehicle features of every two vehicle IDs is calculated using the 1-norm; for each anchor sample, the sample of the same vehicle ID that is furthest away is selected as the most difficult positive sample, and the sample of different vehicle IDs that is closest away is selected as the most difficult negative sample.

9. A vehicle re-identification system based on spatiotemporal feature fusion, implementing the vehicle re-identification method based on spatiotemporal feature fusion as described in any one of claims 1-8, characterized in that, It includes a data acquisition module, a data preprocessing module, and a multimodal spatiotemporal feature fusion network module; The data acquisition module acquires the monitoring videos of adjacent monitoring cameras in the same time period, detects and tracks vehicles appearing in the monitoring videos in real time, and assigns the same vehicle ID to the same vehicle in consecutive frames. Extract vehicle images and associated attribute data corresponding to each vehicle ID from the surveillance video. Save the vehicle images corresponding to each vehicle ID as image files and the associated attribute data in an XML file to form a structured dataset. Treat vehicles with matching vehicle appearance shapes in the surveillance videos of adjacent cameras as the same vehicle, assign them a new vehicle ID, and update the saved image file and XML file of the vehicle to obtain the MIFS vehicle re-identification dataset. The data preprocessing module standardizes the size of the vehicle images; normalizes the vehicle images in the MIFS vehicle re-identification dataset by calculating the mean and variance of the image data in the MIFS vehicle re-identification dataset; applies image enhancement operations to the normalized vehicle images; encodes each associated attribute data of the vehicle images and concatenates them into a vehicle attribute vector; and concatenates the vehicle attribute vectors corresponding to all vehicle images with the same vehicle ID to obtain the concatenated vehicle attribute vector for that vehicle ID. The multimodal spatiotemporal feature fusion network module extracts spatial features of vehicle images through an improved ResNet-50 network, and combines them with a position encoding method that reflects the temporal order of vehicle movement to generate spatiotemporal fusion features. The spatiotemporal fusion features of multiple vehicle images with the same vehicle ID are dynamically stitched together, and after dimensionality reduction by residual convolution, they are mapped to vehicle global spatiotemporal features through an attention mechanism. The stitched vehicle attribute vectors with the same vehicle ID are extracted into vehicle attribute features by an attribute feature extraction module. The vehicle global spatiotemporal features and the vehicle attribute features are vector-added to obtain the multimodal vehicle features of the vehicle ID and saved. The multimodal vehicle features of the vehicle ID are compared with the multimodal vehicle features of all other vehicle IDs that have been saved but not matched by the monitoring cameras at adjacent points. The 1-norm is calculated. If the distance between the two multimodal vehicle features is less than a preset distance threshold, they are matched as the same vehicle.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the vehicle re-identification method based on spatiotemporal feature fusion as described in any one of claims 1-8.

Images