Vehicle re-identification method, system and device based on spatio-temporal feature fusion

Abstract

The present application belongs to the technical field of vehicle re-identification, and discloses a vehicle re-identification method, system and device based on spatio-temporal feature fusion. The present application comprises the following steps: acquiring monitoring videos of adjacent point locations in the same time period, respectively, and detecting and tracking vehicles in real time; extracting vehicle images corresponding to each vehicle ID and associated attribute data and saving them to form a structured data set; matching the vehicle appearance of the vehicles in the adjacent point location monitoring videos with the vehicles to assign new same vehicle IDs to the vehicles, update the structured data set, obtain an MIFS vehicle re-identification data set and perform preprocessing; extracting vehicle global spatio-temporal features and vehicle attribute features from all vehicle images through a multi-modal spatio-temporal feature fusion network model, performing vector addition operation on the two features, obtaining multi-modal vehicle features, and performing vehicle re-identification. The present application can optimize data utilization efficiency to efficiently model spatio-temporal fusion features, thereby improving the efficiency and accuracy of vehicle re-identification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of vehicle re-identification, and particularly relates to a vehicle re-identification method, system and device based on spatio-temporal feature fusion. BACKGROUND

[0002] Vehicle re-identification (ReID) is one of the core technologies in the fields of intelligent transportation and video surveillance, aiming to retrieve the same vehicle across cameras or time periods. In recent years, with the development of deep learning, vehicle re-identification technology has made significant progress, but still faces many challenges.

[0003] Existing vehicle re-identification methods are mainly based on spatial feature extraction of single-frame images, using convolutional neural networks (CNN) or Transformer architecture to learn the visual features of vehicles (such as color, texture, vehicle model, etc.). However, such methods have limitations such as insufficient use of temporal information, poor robustness to changes in viewing angle, and susceptibility to occlusion and lighting interference. The insufficient use of temporal information refers to the fact that single-frame images are difficult to capture the motion features (such as driving direction, speed change, etc.) and appearance feature changes (such as vehicle angle, light change, etc.) of vehicles, while the temporal information in consecutive frames helps to improve the re-identification accuracy, but most existing methods do not fully model the time dimension. The poor robustness to changes in viewing angle refers to the fact that the appearance of the same vehicle under different camera viewing angles differs significantly, and relying solely on static spatial features can easily lead to mismatching. The susceptibility to occlusion and lighting interference refers to the fact that feature extraction is unstable when single-frame images are occluded or lighting changes, while multi-frame information can alleviate such problems through temporal context.

[0004] To overcome the above-mentioned defects, some studies attempt to introduce temporal modeling, such as using three-dimensional (3D) convolution or recurrent neural networks (RNN) to process video sequences, but such methods have high computational complexity and low efficiency of spatio-temporal feature fusion. In addition, existing methods usually rely on cross-camera data association, while ignoring the enhancement effect of temporal consistency of consecutive frames within the same camera on feature learning. Therefore, efficient temporal modeling and data utilization optimization become the content of further research. SUMMARY

[0005] The purpose of the present application is to overcome the deficiencies of the prior art and provide a vehicle re-identification method, system and device based on spatio-temporal feature fusion, which optimizes data utilization efficiency to efficiently model spatio-temporal fusion features, thereby improving the efficiency and accuracy of vehicle re-identification.

[0006] Specifically, the present application is implemented by adopting the following technical scheme.

[0007] In one aspect, the present application provides a vehicle re-identification method based on spatio-temporal feature fusion, comprising:

[0008] Constructing a MIFS (Multi Interval Frame Sequence) vehicle re-identification dataset: acquiring monitoring videos of monitoring cameras of adjacent point locations in the same time period, respectively, detecting and tracking vehicles appearing in the monitoring videos in real time, and assigning the same vehicle ID to the same vehicle in the front and rear frames; extracting vehicle images corresponding to each vehicle ID and associated attribute data from the monitoring videos, saving the vehicle images corresponding to each vehicle ID in the form of image files, saving the associated attribute data in XML files, and constituting a structured dataset; matching the appearances of vehicles in the monitoring videos of the monitoring cameras of adjacent point locations as the same vehicle, assigning a new vehicle ID, and updating the saved image files and XML files of the vehicle, thereby obtaining a MIFS vehicle re-identification dataset;

[0009] Preprocessing the MIFS vehicle re-identification dataset: uniformly sizing the vehicle images; normalizing the vehicle images in the MIFS vehicle re-identification dataset by calculating the mean and variance of the image data; applying image enhancement operations to the normalized vehicle images; encoding each associated attribute data of the vehicle images and concatenating them into a vehicle attribute vector; concatenating the vehicle attribute vectors corresponding to all vehicle images of the same vehicle ID to obtain a concatenated vehicle attribute vector of the vehicle ID;

[0010] Extracting vehicle global spatio-temporal features and vehicle attribute features from the vehicle images corresponding to the vehicle ID through a multi-modal spatio-temporal feature fusion network model, performing vector addition operations on the two features, obtaining multi-modal vehicle features of the vehicle ID, and saving them; performing 1-norm calculation on the multi-modal vehicle features of the vehicle ID and all saved multi-modal vehicle features of other vehicle IDs that have not been matched by the monitoring cameras of adjacent point locations, and if the distance between the two multi-modal vehicle features is less than a pre-set distance threshold, matching them as the same vehicle.

[0011] Further, the encoding of each associated attribute data of the vehicle images and the concatenation into a vehicle attribute vector comprise:

[0012] Encoding the vehicle color and vehicle type using a one-hot encoding method; encoding the vehicle lane using an integer encoding method; the position of the vehicle in the image is a floating-point type data, which is not encoded and uses the original data.

[0013] Further, the multi-modal spatio-temporal feature fusion network model comprises a global spatio-temporal feature extraction module, an attribute feature extraction module and a multi-modal feature fusion module.

[0014] The multi-modal vehicle feature of the vehicle ID is obtained by performing vector addition operation on the global spatio-temporal feature of the vehicle and the attribute feature of the vehicle extracted from the vehicle image corresponding to the vehicle ID by the multi-modal spatio-temporal feature fusion network model, and the multi-modal vehicle feature of the vehicle ID comprises:

[0015] The global spatio-temporal feature extraction module extracts high-level spatial semantic information of the vehicle image from the plurality of vehicle images arranged in time sequence, and fuses the high-level spatial semantic information with the time feature in the vehicle driving process to obtain the global spatio-temporal feature of the vehicle.

[0016] The attribute feature extraction module is used to extract the vehicle attribute feature from the spliced vehicle attribute vector of all vehicle IDs.

[0017] The multi-modal feature fusion module is used to perform vector addition operation on the global spatio-temporal feature of the vehicle and the attribute feature of the vehicle to obtain the multi-modal vehicle feature.

[0018] Further, the global spatio-temporal feature extraction module comprises a spatio-temporal feature fusion module and an attention mechanism module; the spatio-temporal feature fusion module comprises a feature extraction network module and a dynamic time length spatio-temporal feature fusion module.

[0019] The feature extraction network module is an improved Resnet-50 network, and the improved Resnet-50 network comprises a head network, a backbone network and a classification network; the head network accepts an image with an input dimension of (224, 224, 3) and outputs a feature map with a dimension of (112, 112, 96); the backbone network takes the output of the head network as input, and performs feature extraction in a manner that the first two dimensions of the feature map are reduced by 16 times and the third dimension is expanded by 8 times, to obtain a spatial feature with an output dimension of (7, 7, 768); the classification network reconstructs the spatial feature with a dimension of (7, 7, 768) into a one-dimensional vector with a dimension of (1, 37632), and maps the one-dimensional vector into a vector with a dimension of (1, N) for classification, where N is the total number of vehicle IDs.

[0020] Further, the global spatio-temporal feature extraction module extracts high-level spatial semantic information of the vehicle image from the plurality of vehicle images arranged in time sequence, and fuses the high-level spatial semantic information with the time feature in the vehicle driving process to obtain the global spatio-temporal feature of the vehicle, and the global spatio-temporal feature extraction module comprises:

[0021] 3-1-1) using the pre-trained improved Resnet-50 network to extract high-level semantic information of each vehicle image, extracting the features of the vehicle image after normalization with a size of (224, 224, 3) and obtaining a spatial feature with a size of (7, 7, 768);

[0022] 3-1-2) the dynamic duration spatio-temporal feature fusion module, using a position encoding method to encode the time sequence information and spatial information of each vehicle image, generating a position encoding vector with a size of (7, 7, 768) after position encoding; wherein the formula of position encoding is as follows:

[0023]

[0024]

[0025] wherein PE represents the position encoding function; frame index represents the serial number of each vehicle ID corresponding vehicle image, the serial number is counted from 0; i is the dimension index, the even dimension index of each position encoding vector is calculated using a sine function, and the odd dimension index is calculated using a cosine function; d model represents the hidden layer dimension of the attention mechanism module, which is 768; c is the channel serial number of the feature map, which is counted from 0, and the count of c is the same as the channel number of the spatial feature extracted by the improved Resnet-50, which is 768 channels in total;

[0026] 3-1-3) adding the position encoding vector and the spatial feature to obtain the spatio-temporal fusion feature of each vehicle image, and the dimension is still (7, 7, 768);

[0027] 3-1-4) concatenating the spatio-temporal fusion features of multiple vehicle images to obtain a concatenated spatio-temporal fusion feature map;

[0028] 3-1-5) using a one-layer residual convolutional network to reduce the dimension of the concatenated spatio-temporal fusion feature map to obtain a vector with a dimension of (7, 7, 768);

[0029] 3-1-6) converting the (7, 7, 768) dimension vector obtained by dimension reduction into a (49, 768) dimension vector through convolution operation, wherein 49 represents the first and second dimensions of the reduced vector unfolded by row;

[0030] 3-1-7) the attention mechanism module maps the (49, 768) dimension vector obtained in 3-1-6) to a (1, 768) dimension metric space to obtain a vehicle global spatio-temporal feature.

[0031] Further, the attribute feature extraction module comprises a fully connected network and an attention mechanism module: the attribute feature extraction module extracts vehicle attribute features from the spliced vehicle attribute vectors corresponding to all vehicle IDs, which specifically comprises:

[0032] The attribute feature extraction module splices the vehicle attribute vectors corresponding to each vehicle image of the same vehicle ID to obtain a spliced vehicle attribute vector of the vehicle ID; and extracts vehicle attribute features from the spliced vehicle attribute vector;

[0033] The fully connected layer maps the spliced vehicle attribute vector of the vehicle ID to a 768-dimensional feature space to obtain intermediate vehicle attribute features;

[0034] The attention mechanism module maps the intermediate vehicle attribute features to a (1, 768)-dimensional metric space to obtain the vehicle attribute features.

[0035] Further, the multi-modal spatio-temporal feature fusion network model is obtained by the following training method:

[0036] 1) Data loading and batch processing

[0037] The MIFS vehicle re-identification dataset is divided into a training set and a test set according to a set proportion, and a random sampling strategy is used to construct a training batch; each batch contains N vehicle IDs, and K consecutive vehicle images and their corresponding attribute codes are selected for each vehicle ID to form an image vector of (N, K, 224, 224, 3) and an attribute vector of (N, Kx22);

[0038] 2) Training in stages

[0039] First, freeze the feature extraction network module, train the multi-modal spatio-temporal feature fusion network model until the mAP index tends to be stable; then unfreeze all network parameters and fine-tune them;

[0040] A joint loss L = L id + αL triplet is used, where the cross-entropy loss L id is used to supervise the vehicle ID classification task; the triplet loss L triplet is used to constrain the metric space of the multi-modal vehicle features; α is a hyperparameter representing the weight of the triplet loss; the optimizer is AdamW, the initial learning rate is set, and the cosine annealing strategy is used for dynamic adjustment;

[0041] 3) Training monitoring

[0042] After each round of training, the mAP and Rank-1 accuracy of the test set are calculated, and when the performance is not improved for a continuous set of rounds, the training is terminated in advance and the training result corresponding to the highest mAP value is saved as the optimal result of the model.

[0043] Further, the training method of the multi-modal spatio-temporal feature fusion network model further comprises, when calculating the triple loss L triplet Further, the training method of the multi-modal spatio-temporal feature fusion network model further comprises, when calculating the triple loss L

[0044] On the other hand, the present application also provides a vehicle re-identification system based on spatio-temporal feature fusion, which realizes the vehicle re-identification method based on spatio-temporal feature fusion, and comprises a data acquisition module, a data preprocessing module, and a multi-modal spatio-temporal feature fusion network module.

[0045] The data acquisition module acquires monitoring videos of monitoring cameras at adjacent points in the same time period, respectively, detects and tracks vehicles appearing in the monitoring videos in real time, assigns the same vehicle ID to the same vehicle in front and back frames, extracts vehicle images and associated attribute data corresponding to each vehicle ID from the monitoring videos, saves the vehicle images corresponding to each vehicle ID in the form of image files, saves the associated attribute data in XML files to form a structured data set, and matches the appearance shapes of vehicles in the monitoring videos of monitoring cameras at adjacent points to regard the vehicles as the same vehicle, assigns a new vehicle ID to the vehicle, and updates the saved image files and XML files of the vehicle to obtain a MIFS vehicle re-identification data set.

[0046] The data preprocessing module uniformly sizes the vehicle images, normalizes the vehicle images in the MIFS vehicle re-identification data set by calculating the mean and variance of the image data in the MIFS vehicle re-identification data set, applies image enhancement operations to the normalized vehicle images, encodes each associated attribute data of the vehicle images and splices them into a vehicle attribute vector, and splices the vehicle attribute vectors corresponding to all vehicle images of the same vehicle ID to obtain a spliced vehicle attribute vector of the vehicle ID.

[0047] The multi-modal spatio-temporal feature fusion network module extracts the spatial features of the vehicle image through an improved ResNet-50 network, generates spatio-temporal fusion features in combination with a position encoding method reflecting the time sequence of vehicle movement, splices the spatio-temporal fusion features of multiple frames of vehicle images of the same vehicle ID dynamically, maps the spatio-temporal fusion features of the vehicle into global spatio-temporal features of the vehicle through an attention mechanism after dimension reduction by residual convolution, extracts the spliced vehicle attribute vector of the same vehicle ID into vehicle attribute features through an attribute feature extraction module, and performs vector addition operation on the global spatio-temporal features of the vehicle and the vehicle attribute features to obtain the multi-modal vehicle features of the vehicle ID and save them; and the multi-modal vehicle features of the vehicle ID and all saved multi-modal vehicle features of other vehicle IDs that are not matched of the monitoring camera of the adjacent point are subjected to 1-norm calculation, and if the distance between the two multi-modal vehicle features is less than a pre-set distance threshold, the two are matched as the same vehicle.

[0048] In still another aspect, the application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the vehicle re-identification method based on spatio-temporal feature fusion.

[0049] The vehicle re-identification method, system and device based on spatio-temporal feature fusion of the application have the following advantages:

[0050] The vehicle re-identification method, system and device based on spatio-temporal feature fusion of the application first construct a vehicle re-identification data set representing the time sequence relationship of multiple images, realize vehicle detection and tracking by using a YOLOv11 target detection model and a Deepsort tracking algorithm to obtain vehicle images. A multi-modal spatio-temporal feature fusion network is designed innovatively, spatial features are extracted by a pre-trained feature extraction network model (such as an improved Resnet-50 network model), spatio-temporal feature fusion is realized in combination with an innovative sine-cosine position encoding, and an attention mechanism is introduced to optimize feature expression. The multi-modal spatio-temporal feature fusion network integrates vehicle visual features and attribute features, realizes multi-modal feature complementation by using a weighted fusion strategy, and is jointly optimized in combination with a triplet loss and a cross-entropy loss. This method effectively solves the problem of vehicle appearance feature changes under multiple perspectives, and significantly improves the accuracy and robustness of vehicle re-identification.

[0051] In order to solve the problem of vehicle appearance feature changes under multiple perspectives and light changes, the application uses a position encoding method and a dynamic feature map splicing method to efficiently fuse the appearance spatial features and time change features of a vehicle during driving, and improve the robustness of the network in response to perspective and light changes.

[0052] In order to balance performance and precision, the multi-modal spatio-temporal feature fusion network of the application can dynamically change the splicing mode of the spatio-temporal fusion feature map extracted by the feature extraction network module to adapt to the input of different frame numbers, so as to meet the requirements of performance and precision in different scenes.

[0053] The vehicle re-identification method, system and device based on spatio-temporal feature fusion of the application combine the characteristics that the Transformer is good at capturing long-distance dependency and the convolutional network is good at capturing local details, the multi-modal spatio-temporal feature fusion network not only captures the detail information such as the edge texture of the vehicle, but also pays attention to the position distance relationship between the vehicle components, so that the precision of network recognition can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a system schematic diagram of an embodiment of the application.

[0055] Figure 2 is a method flow schematic diagram of an embodiment of the application.

[0056] Figure 3 is a MIFS vehicle re-identification data set directory structure schematic diagram of an embodiment of the application.

[0057] Figure 4 is a detection frame vehicle image saving and lane line division schematic diagram of an embodiment of the application.

[0058] Figure 5 is an example schematic diagram of the same vehicle under different camera perspectives of an embodiment of the application.

[0059] Figure 6 is a multi-modal spatio-temporal feature fusion network structure schematic diagram of an embodiment of the application.

[0060] Figure 7 is a Resnet-50 network improvement mode schematic diagram of an embodiment of the application.

[0061] Figure 8 is a dynamic time length feature map splicing convolution schematic diagram of an embodiment of the application.

[0062] Figure 9 is a vehicle attribute encoding bit number and splicing schematic diagram of an embodiment of the application. DETAILED DESCRIPTION

[0063] The application will be further described in detail below in combination with embodiments and with reference to the accompanying drawings.

[0064] An embodiment of the application is a vehicle re-identification system based on spatio-temporal feature fusion, as shown in Figure 1 , comprising a data acquisition module, a data preprocessing module and a multi-modal spatio-temporal feature fusion network module.

[0065] The data acquisition module respectively acquires the monitoring videos of the monitoring cameras of adjacent point locations in the same time period, detects and tracks vehicles appearing in the monitoring videos in real time (for example, detection is performed based on a YOLOv11 detection model, and tracking is performed based on a Deepsort tracking algorithm), extracts vehicle images and associated attribute data (for example, vehicle color, vehicle type, vehicle lane number data, and vehicle position, that is, vehicle detection box bottom center coordinate data) of each vehicle ID from the monitoring videos, and stores the data as a structured data set. The vehicle images corresponding to each vehicle ID are saved in the form of image files, and the associated attribute data is saved in an XML file to form a structured data set. The data acquisition module matches the appearance shape of the vehicles in the monitoring videos of the monitoring cameras of adjacent point locations as the same vehicle, assigns a new vehicle ID, and updates the saved image file and XML file of the vehicle to obtain an MIFS vehicle re-identification data set.

[0066] The data preprocessing module uniformly sizes the extracted vehicle images (for example, to 224x224 pixels), calculates the mean and variance of the image data in the MIFS data set, and normalizes the vehicle images in the MIFS vehicle re-identification data set according to the mean and variance; random cropping, rotation, flipping, and other image enhancement operations are applied to the normalized vehicle images. Each associated attribute data of the vehicle image is encoded and spliced into a vehicle attribute vector. For example, 11 one-hot encodings are used for vehicle color, 8 one-hot encodings are used for vehicle type, 1 integer encoding is used for the lane, and 2 floating-point values are used as vehicle position information, which are spliced into a 22-dimensional vehicle attribute vector. The vehicle attribute vectors corresponding to multiple frames of vehicle images of the same vehicle are spliced to obtain a spliced vehicle attribute vector, and the dimension is the number of vehicle images of the vehicle ID multiplied by 22, which is spliced according to the method shown in the formula. Figure 9

[0067] ​The multi-modal spatio-temporal feature fusion network module extracts the spatial features (for example, 7x7x768 dimensions) of a single vehicle image in the monitoring video through an improved ResNet-50 network, generates spatio-temporal fusion features in combination with a position encoding method reflecting the time sequence of vehicle motion, and dynamically splices the spatio-temporal fusion features of multiple frames of vehicle images of the same vehicle ID, which are mapped into vehicle global spatio-temporal features (1x768 dimensions) through an attention mechanism after residual convolution dimension reduction. The spliced vehicle attribute vector of the same vehicle ID is refined into vehicle attribute features (1x768) by the attribute feature extraction module. The vehicle global spatio-temporal features and the vehicle attribute features are subjected to vector addition operation, and the multi-modal vehicle features of the vehicle ID are saved. The multi-modal vehicle features of the vehicle ID and the multi-modal vehicle features of other vehicle IDs of all saved unmatched vehicles of the monitoring camera at the adjacent point are subjected to 1-norm calculation, and if the distance between the two multi-modal vehicle features is less than a pre-set distance threshold, they are matched as the same vehicle.

[0068] The vehicle re-identification method based on spatio-temporal feature fusion of the application, as shown in Figure 2 , includes the following steps:

[0069] I. Constructing a MIFS vehicle re-identification dataset.

[0070] Obtain the monitoring videos of the monitoring cameras at adjacent points in the same time period, respectively, detect and track the vehicles appearing in the monitoring videos in real time, and assign the same vehicle ID to the same vehicle in the front and rear frames; extract the vehicle images and associated attribute data corresponding to each vehicle ID from the monitoring videos, save the vehicle images corresponding to each vehicle ID in the form of image files, and save the associated attribute data in XML files to form a structured dataset; match the vehicles with the same appearance shape (for example, vehicle side window shape, rear window shape, vehicle tail shape, and tail lamp shape) in the monitoring videos of the monitoring cameras at adjacent points as the same vehicle, assign a new vehicle ID, and update the saved image files and XML files of the vehicle to obtain a MIFS vehicle re-identification dataset.

[0071] The MIFS vehicle re-identification dataset directory is as shown in Figure 3 , the vehicle images and associated attribute data are extracted from the monitoring videos of each monitoring camera, respectively, and are stored as image files (.jpg files) and XML files (.xml files). The construction of the MIFS vehicle re-identification dataset specifically includes:

[0072] 1-1) Obtain vehicle images

[0073] Two monitoring videos of the adjacent point's monitoring cameras in the same time period are obtained respectively. The Deepsort tracking algorithm based on the target detection model (such as YOLOv11 target detection model) is used to detect the vehicles appearing in the two monitoring videos of the adjacent point's monitoring cameras in the same time period, and the vehicle ID (Identification, identifier) is assigned to different vehicles by using the data such as target appearance, detection box size and position, vehicle speed, etc.

[0074] Preferably, the YOLOv11 target detection model detects the vehicles appearing in each frame image of the monitoring video, and passes the detected vehicles to the Deepsort tracking algorithm in the form of a detection box (such as a rectangle represented by the upper left corner coordinates and the lower right corner coordinates). The Deepsort tracking algorithm matches the vehicles detected by the YOLOv11 target detection model in the front and rear frame images based on the detection box, target appearance in the detection box, target moving speed, etc., and assigns the same vehicle ID to the same vehicle and different vehicle IDs to different vehicles, with the vehicle ID increasing in sequence from 0.

[0075] 1-2) Obtain associated attribute data

[0076] The time interval for saving the image of the vehicle in the detection box (referred to as vehicle image) is set, and the total number of vehicle images required to be saved is set; each vehicle ID corresponding vehicle image is saved in the form of an image file; the vehicle ID, vehicle detection box bottom center coordinate data and vehicle lane number data in the tracked vehicle image are saved in an XML file, such as in the form of a json object in an XML file. The vehicle lane number data is obtained according to the detection box bottom center coordinate data and the pre-divided lane line, as shown in the following formula: Figure 4 The XML file corresponds to the vehicle image file one by one.

[0077] 1-3) Vehicle color classification and vehicle type classification

[0078] The saved each image file is classified by vehicle color and vehicle type, and the classification result data is saved in the XML file corresponding to the vehicle image according to the vehicle ID. The open source vehicle type classification model and vehicle color classification model can be used to classify the saved each vehicle image file by vehicle color and vehicle type.

[0079] 1-4) Match the same vehicle in the monitoring video frames of two cameras by using the vehicle appearance shape

[0080] Since the same vehicle appears in the monitoring video screen of different point camera at different times, the Deepsort tracking algorithm may assign different IDs to the same vehicle. Therefore, the vehicle appearance shape (such as the side window shape, rear window shape, tail shape, and tail lamp shape of the vehicle, etc.) is matched to match the vehicle images in the monitoring video of the monitoring cameras of adjacent point sites, and the vehicle with the same appearance shape in the monitoring video of the monitoring cameras of adjacent point sites is regarded as the same vehicle, and a new vehicle ID is assigned, as shown in Figure 5 The new vehicle ID starts from 0 and increases in sequence. The following takes the 700 vehicle IDs in the MIFS vehicle re-identification dataset as an example to illustrate the scheme of the application.

[0081] II. Preprocessing of the MIFS vehicle re-identification dataset

[0082] The image files and XML files of the MIFS vehicle re-identification dataset are preprocessed, including uniformizing the vehicle images; the mean and variance of the image data in the MIFS vehicle re-identification dataset are calculated, the vehicle images in the MIFS vehicle re-identification dataset are normalized according to the mean and variance, and image enhancement operations are applied to the normalized vehicle images; the associated attribute data of the vehicle images are encoded respectively and spliced into vehicle attribute vectors. The vehicle attribute vectors corresponding to all vehicle images of the same vehicle ID are spliced to obtain the spliced vehicle attribute vector of the vehicle ID. The specific steps are as follows:

[0083] 2-1) Calculate the mean and standard value of the image data in the MIFS vehicle re-identification dataset.

[0084] According to the image files of the MIFS vehicle re-identification dataset, the mean mean c and variance std c of the image data are calculated, which facilitates subsequent normalization processing of the image data. The calculation formula of the mean mean c of the image data is as follows:

[0085]

[0086] where c represents the R, G, or B color channel, n is the total number of images, w i is the width of the i-th image, h i is the height of the i-th image, I i,c (x, y) is the pixel value of the i-th image at position (x, y) in channel c, and N is the total number of pixels:

[0087]

[0088] After obtaining the mean value mean c of the image data c , the variance std c of the image data c is calculated as follows:

[0089] .

[0090] 2-2) Uniform the size of the vehicle image, normalize the vehicle image in the MIFS vehicle re-identification dataset by the mean and variance of the image data in the MIFS vehicle re-identification dataset.

[0091] Adjust the image size in the image file of the MIFS vehicle re-identification dataset to a set value, for example (224, 224, 3), which respectively represent the width, height and color channel number of the image. The Reize function in the torchvision (torch vision function toolkit) package can be used to adjust the image size. Normalize the image data by using the mean value mean c and the variance std c of the image data out , to obtain normalized image data. The pixel value pixel in of the normalized image data is:

[0092]

[0093] wherein pixel in is the original pixel value of the image.

[0094] 2-3) Apply image enhancement operation to the normalized vehicle image.

[0095] Perform random cropping, random rotation and random horizontal flipping image enhancement operations on the normalized image data.

[0096] Random cropping, random rotation and random horizontal flipping operations can be performed using the image processing functions included in the open source torchvision toolkit.

[0097] Specifically, the random cropping operation can be implemented using the random cropping function of the torchvision toolkit, which includes setting the cropping region size parameter, randomly selecting a region with the same size as the cropping region size from the original input standard size image for cropping, and then resizing the cropped image to the standard size. This operation effectively enhances the model's ability to recognize local features of vehicles.

[0098] The random rotation operation can be implemented by using the random rotation function of the torchvision toolkit: taking the center of the image as the rotation base point, setting the rotation angle range (RotationDegree), and randomly changing the rotation angle of the image within the range of [-RotationDegree, RotationDegree] when the random rotation operation is performed. The random rotation operation moderately increases the change in the viewing angle while keeping the vehicle body structure unchanged.

[0099] The random horizontal flip operation can be implemented by using the random horizontal flip function of the torchvision toolkit: setting a flip probability, and performing a horizontal mirror flip operation when the number generated by the random number generator is less than the flip probability. This operation simulates the scene characteristics of bidirectional driving of vehicles in real road environments.

[0100] 2-4) Encode each associated attribute data of the vehicle image respectively, and splice it into a vehicle attribute vector.

[0101] In the XML file of the MIFS vehicle re-identification dataset, the vehicle ID, vehicle color, vehicle type, vehicle lane, and vehicle position in the image are recorded. Since the vehicle color, vehicle type, vehicle lane, and vehicle position in the image play a large role in vehicle re-identification, these vehicle attributes are encoded. Preferably, the vehicle color and vehicle type are encoded using the one-hot encoding method, the vehicle lane is encoded using the integer encoding method, and the vehicle position in the image is a floating-point type data and is not encoded, but the original data is used. Finally, 11 bits of vehicle color one-hot encoding, 8 bits of vehicle type one-hot encoding, 1 bit of lane number, and 2 bits of position encoding are obtained, totaling 22 bits.

[0102] III. Extracting vehicle global spatio-temporal features and vehicle attribute features from the vehicle images corresponding to the vehicle ID by using the multi-modal spatio-temporal feature fusion network model, performing vector addition operation on the two features, obtaining the multi-modal vehicle features of the vehicle ID, and saving them; performing 1-norm calculation on the multi-modal vehicle features of the vehicle ID corresponding to the vehicle ID and all saved multi-modal vehicle features of other vehicle IDs that have not been matched by the adjacent point monitoring camera, and based on a pre-set threshold, determining whether the two vehicles match as the same vehicle. If the distance between the two multi-modal vehicle features is less than the pre-set distance threshold, it is determined that the two vehicles match and are the same vehicle.

[0103] As Figure 6As shown, the multi-modal spatio-temporal feature fusion network model includes a global spatio-temporal feature extraction module, an attribute feature extraction module, and a multi-modal feature fusion module. Among them, x is the vehicle ID, which uniquely identifies a certain vehicle in the data set; id x_0 represents the first vehicle image of the vehicle with vehicle ID x; id x_1 represents the second vehicle image of the vehicle with vehicle ID x; and so on. The specific steps are as follows:

[0104] 3-1) The global spatio-temporal feature extraction module extracts high-level spatial semantic information (including vehicle texture, color, direction, etc.) of the vehicle image from the multiple vehicle images arranged in chronological order, and fuses the extracted high-level spatial semantic information with the time features in the vehicle driving process in an efficient manner to obtain the global spatio-temporal features of the vehicle.

[0105] Preferably, in another embodiment, the global spatio-temporal feature extraction module includes a spatio-temporal feature fusion module and an attention mechanism module. The spatio-temporal feature fusion module includes a feature extraction network module and a dynamic duration spatio-temporal feature fusion module.

[0106] The feature extraction network module is an improved Resnet-50 network, which is obtained by improving the Resnet-50 network, and the improvement method is as shown in Figure 7

[0107] The Resnet-50 network includes a head network (Head), a backbone network (Backbone), and a classifier network (Classifier). The head network receives an image with an input dimension of (224, 224, 3) and outputs a feature map with a dimension of (112, 112, 64). The backbone network takes the output of the head network as input and reduces the first two dimensions of the feature map by 16 times and expands the third dimension by 8 times to obtain a feature map with an output dimension of (7, 7, 512). The classifier network reconstructs the feature map with a dimension of (7, 7, 512) into a one-dimensional vector with a dimension of (1, 25088) (25088 is obtained from 7x7x512), and maps it into a classification vector with a dimension of (1, N) (N represents the number of vehicle IDs).

[0108] ​In the improved Resnet-50 network, a head network, a backbone network and a classification network are included; the output dimension of the head network is changed to (112, 112, 96); the backbone network receives the output of the head network as input and outputs a feature map with a dimension of (7, 7, 768) according to the feature extraction mode; the classification network reconstructs the feature map with a dimension of (7, 7, 768) into a one-dimensional vector with a dimension of (1, 37632) (37632 is obtained from 7*7*768), which is still mapped into a vector with a dimension of (1, 700) for classification. That is, in the improved Resnet-50 network, the head network accepts an input image with a dimension of (224, 224, 3) and outputs a feature map with a dimension of (112, 112, 96); the backbone network takes the output of the head network as input and, according to the feature extraction mode that the first two dimensions of the feature map are reduced by 16 times and the third dimension is expanded by 8 times, obtains a spatial feature with an output dimension of (7, 7, 768); the classification network reconstructs the spatial feature with a dimension of (7, 7, 768) into a one-dimensional vector with a dimension of (1, 37632), and maps it into a vector with a dimension of (1, N) for classification, N being the total number of vehicle IDs, for example, N being 700.

[0109] Preferably, in another embodiment, the improved Resnet-50 network is obtained by the following steps:

[0110] The image data of the MIFS vehicle re-identification dataset is randomly divided into a training set and a test set in a ratio of 8:2, with the vehicle ID as the category, a total of 700 categories. Cross-entropy loss is used as the loss function for training the improved Resnet-50 network.

[0111] The global spatio-temporal feature extraction module extracts high-level spatial semantic information of the vehicle image from a plurality of vehicle images arranged in time sequence, and fuses it with the time feature in the vehicle driving process, to obtain the specific steps of the vehicle global spatio-temporal feature as follows:

[0112] 3-1-1) Utilize the pre-trained improved Resnet-50 network to extract high-level semantic information of each vehicle image, including vehicle texture, color, direction, etc., extract the features of the normalized vehicle image with a size of (224, 224, 3) and obtain spatial features with a size of (7, 7, 768) (representing the width, height and channel number of the feature map, respectively). In the feature extraction network module, the pre-trained improved Resnet-50 network is removed from the classification network, and only the head network and the backbone network are used.

[0113] 3-1-2) Dynamic duration spatio-temporal feature fusion module, using position encoding method to encode the time sequence information and spatial information of each vehicle image; the position vector with a dimension of (7, 7, 768) is generated by position encoding. The formula of position encoding is as follows:

[0114]

[0115]

[0116] Where PE (Position Encoding) represents the position encoding function; frame index represents the sequence number of the vehicle image for each vehicle ID, counting from 0; i is the dimension index, with even-numbered dimension indices of each location encoding vector using a sine function and odd-numbered dimension indices using a cosine function; d model represents the hidden layer dimension of the attention mechanism module, with a value of 768; c is the channel number of the feature map, starting from 0. The count of c is the same as the number of channels of the spatial features extracted by the improved ResNet-50, a total of 768 channels.

[0117] 3-1-3) Add the location encoding vector to the spatial features to obtain the spatiotemporal fusion features of each vehicle image, with the dimensions still being (7,7,768).

[0118] 3-1-4) The spatiotemporal fusion features of multiple vehicle images are stitched together to obtain a stitched spatiotemporal fusion feature map, as shown in the stitching method. Figure 8 As shown, different stitching methods are used depending on the number of vehicle images captured. Figure 8 (a) in the image indicates the stitching method when the number of vehicle image cropped is 4. Figure 8 (b) in the figure represents the stitching method when the number of vehicle image cropped is 9. Figure 8 (b) in comparison to Figure 8 (a) in the example stitches together more features (when the vehicle image cropping interval is fixed, the more vehicle images are cropped and the longer the time span), the richer the temporal features, but the computational cost is greater.

[0119] 3-1-5) Use a single residual convolutional network to reduce the dimensionality of the spatiotemporal fusion feature map to (7, 7, 768). Adjust the stride of the residual convolutional network according to the number of vehicle image segments. For example... Figure 8 As shown in (a) above, the spatiotemporal fusion feature map obtained by concatenation has dimensions (14, 14, 768). Setting the stride of the residual convolutional network to 2 reduces the dimension to (7, 7, 768). Figure 8 As shown in (b), the spatiotemporal fusion feature map obtained by splicing has a dimension of (21,21,768). The stride of the residual convolutional network is set to 3, which reduces the dimension to (7,7,768).

[0120] 3-1-6) Flatten operation is performed on the reduced dimension vector (7, 7, 768) to convert it into a vector of dimension (49, 768), wherein 49 represents the first and second dimensions of the reduced dimension vector expanded by row.

[0121] 3-1-7) The attention mechanism module maps the vector of dimension (49, 768) obtained in 3-1-6) to a metric space of dimension (1, 768) to obtain the vehicle global spatio-temporal feature.

[0122] 3-2) The attribute feature extraction module is used to extract the vehicle attribute feature from the spliced vehicle attribute vector of all vehicle IDs.

[0123] The attribute feature extraction module includes a fully connected network and an attention mechanism module:

[0124] As shown in Figure 9 , the attribute feature extraction module splices the vehicle attribute vector (including 11 bits of vehicle color one-hot encoding, 8 bits of vehicle type one-hot encoding, 2 bits of vehicle position encoding in the image, and 1 bit of vehicle lane integer encoding) corresponding to each vehicle image of the same vehicle ID to obtain a spliced vehicle attribute vector of the vehicle ID, with a dimension of the number of vehicle images of the vehicle ID multiplied by 22.

[0125] The fully connected layer maps the spliced vehicle attribute vector of the vehicle ID to a feature space of 768 dimensions to obtain an intermediate state vehicle attribute feature.

[0126] The attention mechanism module maps the intermediate state vehicle attribute feature to a metric space of dimension (1, 768) to obtain the vehicle attribute feature.

[0127] 3-3) The multi-modal feature fusion module is used to perform vector addition operation on the vehicle global spatio-temporal feature and the vehicle attribute feature to obtain the multi-modal vehicle feature of the vehicle ID and save it. The corresponding multi-modal vehicle feature of the vehicle ID and all saved multi-modal vehicle features of other vehicle IDs of the adjacent monitoring camera that have not been matched are calculated in the 1-norm, and if the distance between the two multi-modal vehicle features is less than a pre-set distance threshold, they are matched as the same vehicle.

[0128] The multi-modal feature fusion module includes a vector addition module of the vehicle global spatio-temporal feature (F global_spatial_temporal ) and the vehicle attribute feature (F attribute ), an attention mechanism module, and a classifier:

[0129] The vehicle global spatio-temporal feature and the vehicle attribute feature are both vectors of dimension (1, 768), and vector addition operation is performed to obtain the multi-modal vehicle feature (F multimodal), the multi-modal vehicle feature dimension is still (1, 768), see the following formula.

[0130]

[0131] wherein β is a hyperparameter, representing the weight of the vehicle attribute feature.

[0132] The attention mechanism module maps the multi-modal vehicle feature of (1, 768) dimension to the metric space of (1, 768) dimension, to obtain a new multi-modal vehicle feature. On the one hand, the metric space of the multi-modal vehicle feature is constrained by the triplet loss. On the other hand, the multi-modal vehicle feature is input into the fully connected classification network for classification, to obtain a classification vector and output a vehicle classification result.

[0133] The multi-modal vehicle feature is a vector of (1, 768) dimension, and the triplet loss L triplet is constrained, and the triplet loss is defined as:

[0134]

[0135] wherein a is an anchor, p is a positive sample, n is a negative sample, ||1 represents the 1-norm; margin is a hyperparameter, which is adjusted according to the data set and the experiment needs. The greater the margin value is, the greater the metric distance between the positive and negative samples is, and the better the classification effect is, but the network training is more difficult, and the convergence speed of the network is slower. The smaller the margin value is, the smaller the metric distance between the positive and negative samples is, and the classification effect will be reduced, but the network training is relatively simple, and the convergence speed of the network is faster.

[0136] The classifier is a fully connected classification network, the input vector is of (1, 768) dimension, and the output is a classification vector of (1, 700) dimension. The second dimension (700) of the output vector represents the number of vehicle IDs in the MIFS vehicle re-identification data set. The classification vector adopts the cross-entropy loss L id constraint, and the cross-entropy loss is defined as:

[0137]

[0138] wherein M represents the total number of samples in the same batch for calculating the cross-entropy loss, C represents the total number of classes, y i,c represents the one-hot encoding (0 or 1) of the true label of the i-th sample in the class c, p i,c represents the predicted probability (the output value of the fully connected layer, ranging from 0 to 1) of the i-th sample belonging to the class c.

[0139] The target of the network training is that the value of the joint loss L is less than a set value, and the joint loss L is defined as:

[0140]

[0141] wherein a is the weight of the triplet loss, L id is the cross-entropy loss, L triplet is the triplet loss.

[0142] Preferably, in another embodiment, the training method of the multi-modal spatio-temporal feature fusion network model is obtained by the following training method:

[0143] 1) Data loading and batch processing

[0144] The MIFS vehicle re-identification dataset is divided into a training set and a test set according to a set proportion (for example, 8:2), and a random sampling strategy is used to construct a training batch. Each batch contains N vehicle IDs, and K consecutive vehicle images and their corresponding attribute encodings are randomly selected for each vehicle ID, forming an image vector of (N, K, 224, 224, 3) and an attribute vector of (N, K x 22).

[0145] 2) Training in stages

[0146] First, freeze the feature extraction network module (the improved ResNet-50 network after freezing only contains the head network and the backbone network), train the multi-modal spatio-temporal feature fusion network model until the mAP index tends to be stable (for example, the difference of the mAP index in continuous 10 rounds of training does not change by more than 1%), and then unfreeze all network parameters for fine-tuning.

[0147] A joint loss function L = L id + aL triplet is used, wherein the cross-entropy loss L id is used for supervised vehicle ID classification tasks; the triplet loss L triplet constrains the metric space of multi-modal fusion features; a is a hyperparameter representing the weight of the triplet loss. The optimizer is AdamW (Adaptive Moment Estimation with Weight Decay, adaptive moment estimation with weight decay), the initial learning rate is set, and the cosine annealing strategy is used for dynamic adjustment.

[0148] Preferably, in another embodiment, when calculating the triplet loss L triplet , the distance between two vectors is calculated using the 1-norm; for each anchor sample, the most difficult positive sample is selected as the sample with the farthest distance of the same vehicle ID, and the most difficult negative sample is selected as the sample with the closest distance of different vehicle IDs to improve the discriminability of the features.

[0149] Preferably, in another embodiment, label smoothing (coefficient 0.1) and random dropout (probability 0.3) are used to prevent overfitting.

[0150] 3) Training monitoring

[0151] After each round of training, the mAP (mean Average Precision) and Rank-1 accuracy of the test set are calculated, and when the performance does not improve for a continuous set of rounds (for example, 10 rounds), the training is terminated in advance and the training result corresponding to the highest mAP value is saved as the optimal result of the model, that is, the optimal model.

[0152] The vehicle re-identification method, system and device based on spatio-temporal feature fusion of the application first construct a vehicle re-identification data set representing the time sequence relationship of multiple images, use a YOLOv11 target detection model and a Deepsort tracking algorithm to realize vehicle detection and tracking to obtain vehicle images. A multi-modal spatio-temporal feature fusion network is innovatively designed, spatial features are extracted through a pre-trained feature extraction network model (such as an improved Resnet-50 network model), spatio-temporal feature fusion is realized in combination with innovative sine-cosine position encoding, and an attention mechanism is introduced to optimize feature expression. The multi-modal spatio-temporal feature fusion network integrates vehicle visual features and attribute features, uses a weighted fusion strategy to realize complementary multi-modal features, and combines triple loss and cross-entropy loss for joint optimization. This method effectively solves the problem of vehicle appearance feature changes under multiple perspectives, and significantly improves the accuracy and robustness of vehicle re-identification.

[0153] In order to solve the problem of vehicle appearance feature changes under multiple perspectives and light changes, the application uses a position encoding method and a dynamic feature map splicing method to efficiently fuse the appearance spatial features and time change features of vehicles during driving, thereby improving the robustness of the network in response to perspective and light changes.

[0154] In order to balance performance and accuracy, the multi-modal spatio-temporal feature fusion network of the application can dynamically change the splicing method of the spatio-temporal fusion feature map extracted by the feature extraction network module to adapt to different frame numbers of input, thereby meeting the performance and accuracy requirements in different scenarios.

[0155] The vehicle re-identification method, system and device based on spatio-temporal feature fusion of the application combine the characteristics that Transformer is good at capturing long-distance dependency and convolutional network is good at capturing local details, the multi-modal spatio-temporal feature fusion network not only captures detailed information such as vehicle edge texture, but also pays attention to the position distance relationship between vehicle components, which can improve the accuracy of network recognition.

[0156] In some embodiments, certain aspects of the techniques described above can implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer- readable storage medium. The software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium can include, for example, a magnetic or optical disk storage such as a compact disk (CD) or DVD, a solid state memory like flash, cache or random access memory (RAM) or other non-volatile memory device or a combination of magnetic, optical, and / or solid state memory. The executable instructions can be any set of instructions executed by one or more processors to cause the processor to perform a set of operations, whether those operations are related or unrelated.

[0157] A computer readable storage medium can include any storage medium, or combination of storage media, accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer readable storage medium can be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

[0158] Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device can not be required, and that one or more further activities or devices can be performed or included in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to particular embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0159] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that can cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, essential, or a key element or to the patency of any or all the claims. Furthermore, the particular embodiments disclosed above are illustrative only as the disclosed subject matter can be modified and practiced in different but equivalent manners that are apparent from the teachings embodied in this specification and the appended claims. No terminology employed herein is intended to be particularly indicative or descriptive acertain design or method disclosed herein. Accordingly, all such changes are considered to be within the scope of the disclosed subject matter.

Claims

1. A vehicle re-identification method based on spatio-temporal feature fusion, characterized in that, The method comprises the following steps: Constructing a multi-interval frame sequence (MIFS) vehicle re-identification dataset: acquiring monitoring videos of monitoring cameras at adjacent locations in the same time period, respectively, detecting and tracking vehicles appearing in the monitoring videos in real time, and assigning the same vehicle ID to the same vehicle in the front and rear frames; Extracting vehicle images corresponding to each vehicle ID and associated attribute data from the monitoring videos, saving the vehicle images corresponding to each vehicle ID in the form of image files, saving the associated attribute data in XML files, and constructing a structured dataset; matching the appearance shapes of vehicles in the monitoring videos of monitoring cameras at adjacent locations as the same vehicle, assigning a new vehicle ID, and updating the saved image files and XML files of the vehicle, thereby obtaining a MIFS vehicle re-identification dataset; Preprocessing the MIFS vehicle re-identification dataset: uniformly sizing the vehicle images; normalizing the vehicle images in the MIFS vehicle re-identification dataset by calculating the mean and variance of the image data; applying image enhancement operations to the normalized vehicle images; encoding each associated attribute data of the vehicle images and concatenating them into a vehicle attribute vector; concatenating the vehicle attribute vectors corresponding to all vehicle images of the same vehicle ID to obtain a concatenated vehicle attribute vector of the vehicle ID; Extracting vehicle global spatio-temporal features and vehicle attribute features from the vehicle images corresponding to the vehicle ID by using a multi-modal spatio-temporal feature fusion network model, performing vector addition operations on the two features, and obtaining multi-modal vehicle features of the vehicle ID and saving them; performing 1-norm calculation on the multi-modal vehicle features of the vehicle ID and all saved other vehicle IDs that have not been matched by monitoring cameras at adjacent locations, and if the distance between the two multi-modal vehicle features is less than a pre-set distance threshold, then the two are matched as the same vehicle; the multi-modal spatio-temporal feature fusion network model comprises a global spatio-temporal feature extraction module, an attribute feature extraction module, and a multi-modal feature fusion module; The method of extracting vehicle global spatio-temporal features and vehicle attribute features from the vehicle images corresponding to the vehicle ID by using a multi-modal spatio-temporal feature fusion network model, performing vector addition operations on the two features, and obtaining multi-modal vehicle features of the vehicle ID specifically comprises: The global spatio-temporal feature extraction module extracts high-level spatial semantic information of vehicle images from a plurality of vehicle images arranged in chronological order, and fuses the information with time features in the vehicle driving process to obtain vehicle global spatio-temporal features; The attribute feature extraction module extracts vehicle attribute features from the concatenated vehicle attribute vectors of all vehicle IDs; The multi-modal feature fusion module performs vector addition operations on the vehicle global spatio-temporal features and vehicle attribute features to obtain the multi-modal vehicle features. 2.The vehicle re-identification method based on spatio-temporal feature fusion according to claim 1, characterized in that, The method of encoding each associated attribute data of the vehicle images and concatenating them into a vehicle attribute vector comprises: The vehicle color and the vehicle type are encoded by using a one-hot encoding method, and the lane where the vehicle is located is encoded by using an integer encoding method; the position of the vehicle in the image is a floating point type data, and is not encoded, and the original data is adopted. 3.The vehicle re-identification method based on spatio-temporal feature fusion according to claim 1, characterized in that, The global spatio-temporal feature extraction module comprises a spatio-temporal feature fusion module and an attention mechanism module; the spatio-temporal feature fusion module comprises a feature extraction network module and a dynamic time length spatio-temporal feature fusion module; The feature extraction network module is an improved Resnet-50 network, and the improved Resnet-50 network comprises a head network, a backbone network and a classification network; the head network accepts an image with an input dimension of (224, 224, 3) and outputs a feature map with a dimension of (112, 112, 96); the backbone network takes the output of the head network as input and adopts a feature extraction mode of reducing the first two dimensions of the feature map by 16 times and expanding the third dimension by 8 times to obtain a spatial feature with an output dimension of (7, 7, 768); and the classification network reconstructs the spatial feature with a dimension of (7, 7, 768) into a one-dimensional vector with a dimension of (1, 37632) and maps it into a vector with a dimension of (1, N) for classification, wherein N is the total number of vehicle IDs. 4.The vehicle re-identification method based on spatio-temporal feature fusion according to claim 3, characterized in that, The global spatio-temporal feature extraction module extracts high-level spatial semantic information of vehicle images from a plurality of vehicle images arranged in time sequence and fuses the high-level spatial semantic information with time features in the driving process of the vehicle to obtain vehicle global spatio-temporal features, which specifically include: 3-1-1) high-level semantic information of each vehicle image is extracted by using a pre-trained improved Resnet-50 network, features of a vehicle image with a normalized size of (224, 224, 3) are extracted, and a spatial feature with a size of (7, 7, 768) is obtained; 3-1-2) the dynamic time length spatio-temporal feature fusion module encodes time sequence information and spatial information of each vehicle image by using a position encoding method, and generates a position encoding vector with a dimension of (7, 7, 768); wherein the formula of the position encoding is as follows: ； ； wherein PE represents a position encoding function; frame index denotes the sequence number of the vehicle image corresponding to each vehicle ID, the sequence number is counted from 0; i is a dimension index, even dimension indexes of each position encoding vector are calculated using a sine function, and odd dimension indexes are calculated using a cosine function; d model denotes the hidden layer dimension of the attention mechanism module, and takes a value of 768; c is a channel sequence number of a feature map, counted from 0, and the count of c is the same as the channel number of the spatial feature extracted by the improved Resnet-50, which is 768 channels in total; 3-1-3) the position encoding vector and the spatial feature are added to obtain spatio-temporal fusion features of each vehicle image, and the dimension is still (7, 7, 768); 3-1-4) spatio-temporal fusion feature maps of a plurality of vehicle images are spliced to obtain spliced spatio-temporal fusion feature maps; 3-1-5) a one-layer residual convolutional network is used to reduce the dimension of the spliced spatio-temporal fusion feature maps to obtain a vector with a dimension of (7, 7, 768); 3-1-6) the vector with a dimension of (7, 7, 768) obtained by reducing the dimension is converted into a vector with a dimension of (49, 768) through a convolution operation, wherein 49 represents that the first and second dimensions of the vector obtained by reducing the dimension are unfolded by rows; 3-1-7) the attention mechanism module maps the vector with a dimension of (49, 768) obtained in 3-1-6) to a metric space with a dimension of (1, 768) to obtain vehicle global spatio-temporal features. 5.The vehicle re-identification method based on spatio-temporal feature fusion according to claim 1, characterized in that, The attribute feature extraction module comprises a full connection network and an attention mechanism module: the attribute feature extraction module extracts vehicle attribute features from the spliced vehicle attribute vectors corresponding to all vehicle IDs, and the extraction specifically comprises: The attribute feature extraction module splices the vehicle attribute vectors corresponding to each vehicle image of the same vehicle ID to obtain a spliced vehicle attribute vector of the vehicle ID; and extracts vehicle attribute features from the spliced vehicle attribute vector; The full connection network maps the spliced vehicle attribute vector of the vehicle ID to a 768-dimensional feature space to obtain intermediate vehicle attribute features; The attention mechanism module maps the intermediate vehicle attribute features to a (1, 768)-dimensional metric space to obtain the vehicle attribute features. 6.The vehicle re-identification method based on spatio-temporal feature fusion according to claim 3, characterized in that, The multi-modal spatio-temporal feature fusion network model is obtained by the following training method: 1) Data loading and batch processing The MIFS vehicle re-identification dataset is divided into a training set and a test set according to a set proportion, and a random sampling strategy is used to construct a training batch; each batch contains N vehicle IDs, and K consecutive vehicle images and their corresponding attribute codes are selected for each vehicle ID to form an image vector of (N, K, 224, 224, 3) and an attribute vector of (N, K x 22); 2) Training in stages The feature extraction network module is first frozen, and the multi-modal spatio-temporal feature fusion network model is trained until the mAP index tends to be stable; then all network parameters are unfrozen for fine-tuning; Adopt joint loss L = L id + aL triplet , where cross-entropy loss L id is used for supervised vehicle ID classification task; triplet loss L triplet constrains the metric space of the multi-modal vehicle features; a is a hyper-parameter, representing the weight of the triplet loss; The optimizer is AdamW, the initial learning rate is set, and the cosine annealing strategy is used for dynamic adjustment; 3) Training monitoring After each round of training, the mAP and Rank-1 accuracy of the test set are calculated, and when the performance has not improved for a continuous set number of rounds, the training is terminated in advance and the training result corresponding to the highest mAP index is saved as the optimal model result.

7. The vehicle re-identification method based on spatio-temporal feature fusion according to claim 6, characterized in that, Also included in calculating the triple loss L triplet When calculating the triple loss L , the distance between the multi-modal vehicle features of each two vehicle IDs is calculated using the 1-norm; for each anchor sample, the sample of the same vehicle ID with the farthest distance is selected as the hardest positive sample, and the sample of the different vehicle ID with the closest distance is selected as the hardest negative sample.

8. A vehicle re-identification system based on spatio-temporal feature fusion, which implements the vehicle re-identification method based on spatio-temporal feature fusion according to any one of claims 1-7. The data acquisition module, the data preprocessing module, and the multi-modal spatio-temporal feature fusion network model are included. The data acquisition module acquires monitoring videos of monitoring cameras at adjacent points in the same time period, detects and tracks vehicles appearing in the monitoring videos in real time, and assigns the same vehicle ID to the same vehicle in front and back frames; Vehicle images and associated attribute data corresponding to each vehicle ID are extracted from the monitoring videos, the vehicle images corresponding to each vehicle ID are saved in the form of image files, the associated attribute data are saved in XML files, and a structured dataset is formed; vehicles with matching appearances in the monitoring videos of monitoring cameras at adjacent points are matched as the same vehicle, a new vehicle ID is assigned, and the image file and the XML file of the vehicle are updated to obtain a MIFS vehicle re-identification dataset; The data preprocessing module unifies the sizes of the vehicle images, normalizes vehicle images in the MIFS vehicle re-identification data set by calculating the mean and variance of the image data in the MIFS vehicle re-identification data set, applies an image enhancement operation to the normalized vehicle images, respectively encodes each associated attribute data of the vehicle images and splices the attribute data into a vehicle attribute vector, splices vehicle attribute vectors corresponding to all vehicle images of the same vehicle ID to obtain a spliced vehicle attribute vector of the vehicle ID, and splices the vehicle attribute vectors corresponding to all vehicle images of the same vehicle ID to obtain a spliced vehicle attribute vector of the vehicle ID. The multi-modal spatio-temporal feature fusion network model extracts spatial features of vehicle images through an improved ResNet-50 network, generates spatio-temporal fusion features in combination with a position encoding method reflecting the order of vehicle movement time, and extracts vehicle attribute features through an attribute feature extraction module. The spatio-temporal fusion features of multiple frames of vehicle images of the same vehicle ID are dynamically spliced, mapped into vehicle global spatio-temporal features through an attention mechanism after residual convolution dimension reduction, the spliced vehicle attribute vector of the same vehicle ID is refined into vehicle attribute features through the attribute feature extraction module, the vehicle global spatio-temporal features and the vehicle attribute features are subjected to vector addition operation, the multi-modal vehicle features of the vehicle ID are obtained and saved, and the multi-modal vehicle features of the vehicle ID and all saved multi-modal vehicle features of other vehicle IDs that are not matched of the monitoring camera of the adjacent point are subjected to 1-norm calculation, and if the distance between the two multi-modal vehicle features is less than a pre-set distance threshold, the two are matched as the same vehicle.

9. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by a processor to realize the steps of the vehicle re-identification method based on spatio-temporal feature fusion according to any one of claims 1-7.

Images