Vehicle re-identification optimization method based on spatio-temporal clustering reordering

Abstract

Provided is vehicle re-identification optimization based on spatio-temporal clustering reordering. The provided vehicle re-identification optimization method based on spatio-temporal clustering reordering comprises: acquiring an image to be identified and time attributes and space attributes of the image to be identified; sorting similarity of an appearance feature vector of each image of a candidate data set and an appearance feature vector of the image to be identified based on a difference between the appearance feature vector of the image to be identified and the appearance feature vector of each image of the candidate data set, to obtain a matching image retrieval sequence; calculating time data and space data of each image in the matching image retrieval sequence; clustering according to respective spatio-temporal vectors of the first N images to obtain a plurality of retrieval image clusters; sorting each retrieval image cluster according to a difference between the time attributes and / or space attributes of the image to be identified and each retrieval image cluster; and outputting images of each retrieval image cluster in a sorting order of each retrieval image cluster as a re-identification result of the image to be identified.

Description

Technical Field

[0001] This application relates to vehicle re-identification technology, specifically, to an optimized method for re-identifying vehicles in images based on spatiotemporal clustering reordering. Background Technology

[0002] Re-identification is a subtask of computer vision. Its purpose is to extract image sequences that match a specified target image from a given candidate image gallery using appearance features extracted from the image and some auxiliary features such as license plates and colors.

[0003] Gallery (candidate image library): In re-identification tasks, "Gallery" refers to a collection of images containing known identities or targets. These images are typically used to build a database for the re-identification system to compare and match during queries.

[0004] Images in the "Gallery" library can come from various sources, such as surveillance cameras, image databases, or data collected from specific projects. Images in the Gallery (candidate image library) are labeled with various attributes, such as temporal attributes (time stamp of the image being captured), spatial attributes (camera location where the image was captured), and target attributes (such as vehicle license plate numbers or other unique vehicle identifiers, vehicle pose (position, orientation, etc.)). Optionally, image attributes may also include feature vectors of the image or targets within it.

[0005] Query (Image to be identified): In re-identification tasks, "Query" refers to the image to be identified or retrieved, typically used for comparison and matching with images in the "Gallery". "Query" can be a single image or a group of images. The goal of the re-identification task is to find one or more images in the "Gallery" that are similar to or match the "Query".

[0006] Anchor Image: An anchor image is used as a reference image in a re-identification task. It is typically one of a set of images used during the training phase. The anchor image is compared and matched with other images in the training data (Gallery) to learn feature representations and calculate similarity metrics. During training, the anchor image serves as a reference point for calculating the similarity or distance between images and is used to train the re-identification neural network model.

[0007] The difference between Query (image to be recognized) and Anchor (anchor image) is that Anchor image is mainly used in the network model training stage of the re-recognition process, while Query image is mainly used in the testing and application stage of the re-recognition process.

[0008] The target of the re-identification task can be an object such as a person or a vehicle. The method of this application is mainly applied in the field of vehicle re-identification. There are significant differences in appearance features between vehicle targets and pedestrian targets: (1) Vehicle targets have a high degree of appearance similarity. Due to standardized production, a large number of vehicles of the same color and model appear on the road. These vehicles often need to be distinguished based on very subtle differences in appearance features (such as interior decorations, license plates, etc.), which greatly increases the difficulty of vehicle re-identification. (2) Because the movement of vehicle targets is restricted by the road, their movement direction is relatively fixed compared to pedestrian targets. Moreover, the shooting direction of each road camera is also fixed. Therefore, the differences in the appearance features of vehicles collected by cameras facing different sides of the vehicle are large.

[0009] Figure 1 This diagram illustrates how different cameras in a lane photograph the same vehicle. (See attached image.) Figure 1 The road direction is defined as the direction in which vehicles travel within a lane. Camera 1 is oriented opposite to the road direction, thus primarily capturing images of features in front of the vehicle. Camera 2 is oriented roughly in the same direction as the road, therefore primarily capturing images of features behind the vehicle. This increases the difficulty of matching different images belonging to the same vehicle in the matching image gallery. Therefore, vehicle re-identification algorithms based solely on appearance features are constrained by these appearance feature limitations, thus limiting the accuracy of re-identification.

[0010] The accuracy of re-identification tasks can be described by metrics such as the CMC (Cumulative Match Characteristic) curve, Rank, and mAP (mean Average Precision).

[0011] CMC (Cumulative Match Characteristic) Curve: The CMC curve is an evaluation method used to measure the matching rate among the top N results in an image retrieval task. On the CMC curve, the horizontal axis represents the rank N, and the vertical axis represents the matching rate. For example, point CMC(1) on the CMC curve represents the matching rate of the 1st ranked result, CMC(5) represents the cumulative matching rate of the top 5 results, and so on. Generally, a higher CMC curve (in the positive direction of the vertical axis) indicates better performance.

[0012] Rank: In image retrieval tasks, Rank represents the position of the query result image in the ranking results. For example, if a query result image has a Rank of 3, it means that the image ranks 3rd in the ranking results. A lower Rank value indicates that the query result image and the image to be identified (Query) are more similar under the specified evaluation criteria (such as appearance features). Therefore, the higher the correct image ranks in the query results, the more accurately the retrieval system has matched it.

[0013] mAP (mean Average Precision): mAP is a commonly used metric for evaluating the performance of image retrieval systems. It combines the accuracy of all query result images with their ranking information in the sorted results. For each query result image, mAP calculates the area under the Precision-Recall curve (Average Precision), and then averages this across all query result images. A higher mAP value indicates better performance because it represents the system's average accuracy across all query result images.

[0014] Existing network models for vehicle re-identification include HRCN (Zhao, JJ, et al. (2021). "Heterogeneous Relational Complement for Vehicle Re-identification." 2021 IEEE / Cvf International Conference on Computer Vision (Iccv 2021): 205-214.), ASSEN (Li, H., et al. (2022). "Attribute and State Guided Structural Embedding Network for Vehicle Re-Identification." IEEE Trans Image Process 31: 5949-5962.), and UFDN (Qian, W., et al. (2022). "Unstructured Feature Decoupling for Vehicle Re-identification." Computer Vision-Eccv2022, Pt Xiv 13674: 336-353.), etc. Existing technologies also include the VeRi-776 dataset for training vehicle re-identification network models (Liu, XC, et al. (2016). "A DeepLearning-Based Approach to Progressive Vehicle Re-identification for Urban Surveillance." Computer Vision-Eccv 2016, Pt Ii 9906:869-884.). For simplicity, such network models for re-identification are referred to as ReID network models.

[0015] Min-Max normalization and Z-Score normalization are normalization algorithms in existing technologies.

[0016] Min-Max normalization scales the original data to a specific range (usually [0,1] or [-1,1]) so that all data points fall within this range. Specifically, Min-Max normalization maps each original data point x to a normalized value x' using the following formula:

[0017]

[0018] Here, min(x) is the minimum value in the dataset, and max(x) is the maximum value in the dataset. Through this normalization process, the minimum value of the original data will be mapped to the normalized minimum value (usually 0), the maximum value will be mapped to the normalized maximum value (usually 1), and other values ​​will be scaled proportionally.

[0019] Z-score normalization is also a common normalization method. It normalizes data by subtracting the mean from each data point and then dividing by the standard deviation. This results in a mean of 0 and a standard deviation of 1. The specific formula is as follows:

[0020]

[0021] Where z is the normalized value, x is the original value, μ is the mean, and σ is the standard deviation.

[0022] The purpose of Z-score normalization is to eliminate scale differences in data, allowing data to be compared and analyzed on the same scale. It does not change the original distribution shape of the data; it merely performs a linear transformation, mapping the data to a distribution centered at the mean and with standard deviations as units. Therefore, Z-score normalization preserves the relative positions and distribution shape of sample points, only changing the scale and units of the data. Summary of the Invention

[0023] Network models such as HRCN, ASSEN, and UFDN rely solely on vehicle appearance image features as the judgment criterion. In some closed road conditions, they fail to consider temporal and spatial information as judgment criteria. This leads to some misidentified images (similar in appearance, such as color and model) being selected as correct images despite temporal and spatial discrepancies. Consequently, key re-identification metrics, such as CMC and mAP, which describe the accuracy of re-identification, decrease, thus affecting the model's re-identification accuracy.

[0024] Figure 2 This diagram illustrates misidentified images that have similar appearance features but conflicting timelines.

[0025] For example, the image to be identified (Query) is an image of vehicle id=1 taken by camera 1 at 18:00. If the result of retrieving a similar image from the candidate image library based on the image to be identified (id=1) is image id=2 (also taken by camera 1, but at 19:00), since, under normal circumstances, over a period of up to one hour (from 18:00 to 19:00), vehicles have traveled a considerable distance on the road and would not remain in the same camera's field of view for an extended period, they should not be captured by the same camera. By 19:00, the vehicle in the image to be identified should have already moved far away from the capture range of camera 1. Therefore, the image (id=2) captured by camera 1 at 19:00 that is visually similar to the image to be identified (Query) should be a misidentified image and should not be considered a result of vehicle re-identification.

[0026] Figure 3 This diagram illustrates misidentified images that have similar appearance features but conflicting spatial relationships.

[0027] Figure 3 In the example, the image to be identified (Query) is an image of vehicle id=1 taken by camera 1 at 18:00. The candidate image database retrieves an image of vehicle id=2, taken by camera 5 at 18:01, which is similar to the image to be identified. Furthermore, the spatial distance between camera 1 and camera 5 is 5 kilometers. Since the vehicle's speed should objectively be within a reasonable range, and the road has traffic regulations limiting speed, the range a vehicle can travel in a short time is limited. Therefore, when camera 1 acquires the image to be identified (Query) (id=1), the spatial location that vehicle id=1 can reach within a very short time (1 minute) will not be far from camera 1 (e.g., within 1 or 2 kilometers). Therefore, Figure 3 The vehicle image with id=2 captured by camera 5, although similar in appearance to the vehicle image with id=1 captured by camera 1, is also a misidentified image and should not be used as the result of vehicle re-identification.

[0028] Figure 4 This presents yet another example of an image to be identified.

[0029] Figure 5 This demonstrates, in yet another example, the image retrieval results obtained by re-identifying the image based on the image to be identified using the HRCN model.

[0030] Figure 4 and Figure 5 The images in the dataset are from the Veri776 dataset.

[0031] Compare Figure 4 and Figure 5 , Figure 5 In the displayed image retrieval results, each image is visually similar to... Figure 4 The images to be identified have a high degree of similarity.

[0032] Figure 5 The system also uses dark and light borders around the images to distinguish between correct and incorrectly identified images. Images marked with dark borders are correct (e.g., ...). Figure 5 Images from the left in the first row (images 2, 4-6), images from the left in the second row (images 1, 4, 5, and 8), and image 4 in the third row (images 4 in the third row) are marked with light-colored borders as incorrectly identified images (e.g., Figure 5 Images from left to right in the first row (images 1, 3, 7, and 8); images from left to right in the second row (images 2, 3, 6, and 7); and images from left to right in the third row (images 1-3). Correctly identified images show vehicles that are the same as those in the image to be identified, while incorrectly identified images show vehicles that are similar in appearance to those in the image to be identified but are different.

[0033] From this, we can discover some and Figure 4 Misidentified images that are very similar to the images to be identified, such as Figure 5 The first image in the first row (corresponding to vehicle ID: 300) is related to the vehicle in time and space. Figure 4 The images to be identified conflict—the cameras for these two images are located at the same or adjacent intersections, but the acquisition times differ significantly. However, due to... Figure 5 The first image in the first row (corresponding to vehicle ID: 300) and Figure 4 The images to be identified are very similar in appearance features, so the feature vectors calculated by the HRCN network model have very small differences, thus placing it first in the vehicle re-identification results. The more erroneous identification results there are in the top-ranked re-identification results, the worse the accuracy metrics (e.g., CMC, Rank, mAP) will be.

[0034] For the vehicle re-identification problem in road traffic in the short term, two characteristics related to vehicle motion (called vehicle motion characteristics) can be modeled: (1) The vehicle target has a relatively stable motion pattern. Since the vehicle's motion on the road needs to comply with traffic regulations, the vehicle needs to appear in the field of view of different road cameras along a specific route; (2) The vehicle target's motion speed has a certain range. Therefore, in a short period of time (e.g., within a few hours), matching images of the same vehicle taken by multiple cameras deployed at different locations on the road can be retrieved in the candidate image library. At the same time, the same vehicle will not stay in the acquisition range of several cameras that are in the same place or spatially adjacent for a long time.

[0035] Based on the characteristics of vehicle target motion described above, we can model scenarios of incorrectly identified images that are similar in appearance to the image to be identified (Query) but conflict in time and space. This allows us to determine the specific types of temporal and spatial conflicts and utilize these conflict types to filter and remove incorrectly identified images from the image re-identification results, thereby improving the accuracy of vehicle re-identification in the image retrieval stage. Specifically, the accuracy metrics of vehicle re-identification in the image retrieval stage (e.g., CMC, Rank, mAP) are related to the order of images in the image retrieval results. Inevitably, incorrectly identified images will exist in the image retrieval results of vehicle re-identification. The more correct images are ranked above incorrectly identified images in the image retrieval results, the better the accuracy metrics will be, and the faster or earlier users will obtain the correct image retrieval results.

[0036] According to embodiments of this application, leveraging the temporal and spatial conflicts between the image to be identified and the incorrectly identified image in the vehicle re-identification scenario of short-term road traffic, a clusterer based on temporal and spatial information (STcluster) is designed. Building upon the re-ranking results of mainstream network models, STcluster utilizes temporal and spatial continuity to cluster the image re-ranking results based on temporal and spatial distances, and then performs a secondary re-ranking of the clustering results using their temporal and spatial distances to the image to be identified. By reducing the ranking of images that conflict with the image to be identified in the temporal and spatial dimensions in the image retrieval results, the ranking of correct images in the image retrieval results is improved, thereby enhancing the accuracy of the retrieval results.

[0037] According to a first aspect of this application, a first vehicle re-identification optimization method based on spatiotemporal clustering re-ranking is provided, comprising: acquiring an image to be identified and its temporal and spatial attributes; extracting appearance feature vectors for the image to be identified and images in a candidate image library; ranking the similarity between the appearance feature vectors of each image in the candidate image library and the appearance feature vectors of the image to be identified based on the differences between the appearance feature vectors of the image to be identified and the images in the candidate image library, thereby obtaining a matching image retrieval sequence; and calculating the differences in temporal and spatial attributes between each image in the matching image retrieval sequence and the image to be identified, as used for matching. The image retrieval sequence includes temporal and spatial data for each image. The temporal and spatial data for each image in the matching image retrieval sequence are normalized to obtain a spatiotemporal vector for each image in the matching image retrieval sequence, where the spatiotemporal vector includes both temporal and spatial dimensions. The first N images in the matching image retrieval sequence are clustered to obtain multiple retrieval image clusters, where clustering is based on the spatiotemporal vectors of the first N images, and N is a positive integer. Each retrieval image cluster is sorted according to the differences in temporal and / or spatial attributes between the image to be identified and each retrieval image cluster. The images of each retrieval image cluster are output as the re-identification result for the image to be identified, according to the sorting order of the retrieval image clusters.

[0038] A second vehicle re-identification optimization method based on spatiotemporal clustering reordering according to the first aspect of this application is provided, wherein appearance feature vectors are extracted from the image to be identified and the images in the candidate image library by a feature extraction neural network.

[0039] A third vehicle re-identification optimization method based on spatiotemporal clustering reordering is provided according to the first aspect of this application, wherein the temporal attribute of the image is the time when the image was acquired; and the spatial attribute of the image is the pose of the vehicle in the image.

[0040] A fourth vehicle re-identification optimization method based on spatiotemporal clustering reordering according to the first aspect of this application is provided, wherein the normalization of the temporal data and spatial data of each image in the image retrieval sequence for matching includes: normalizing the temporal data and spatial data of each image in the image retrieval sequence for matching using the Min-Max method or the Z-Score method respectively.

[0041] A fifth vehicle re-identification optimization method based on spatiotemporal clustering reordering according to the first aspect of this application is provided, wherein the clustering of the first N images of the matching image retrieval sequence is performed according to the spatiotemporal vector and appearance feature vector of each of the first N images, and the clustering is performed in three dimensions: time, space and appearance feature vector.

[0042] A sixth vehicle re-identification optimization method based on spatiotemporal clustering reordering according to the first aspect of this application is provided, wherein the first N images of the matching image retrieval sequence are clustered by setting different weights for the time dimension and spatial dimension in the spatiotemporal vector of each of the first N images, and clustering is performed according to the weighted time dimension and spatial dimension, wherein the weight of the time dimension is greater than the weight of the spatial dimension.

[0043] A seventh vehicle re-identification optimization method based on spatiotemporal clustering re-ranking according to the first aspect of this application is provided, wherein ranking the retrieved image clusters according to the differences in temporal and / or spatial attributes between the image to be identified and each retrieved image cluster includes: taking the maximum value of the temporal data of the images within each retrieved image cluster as the temporal attribute of the cluster, and ranking the retrieved image clusters according to the differences in the temporal attributes between the image to be identified and each retrieved image cluster, wherein the retrieved image clusters with smaller differences in the temporal attributes of the image to be identified are ranked first; taking the maximum value of the spatial data of the images within each retrieved image cluster as the spatial attribute of the cluster, and ranking the retrieved image clusters according to the differences in the spatial attributes between the image to be identified and each retrieved image cluster, wherein the retrieved image clusters with smaller differences in the spatial attributes of the image to be identified are ranked first; or dividing the combination of temporal and spatial dimensions into multiple specified regions according to the motion characteristics of vehicles in the image, wherein the specified regions represent the specified spaces in which the estimated vehicles in the image should be located at a specified time; and ranking the retrieved image clusters according to the distances between each retrieved image cluster and the multiple specified regions, wherein the retrieved image clusters closest to the multiple specified regions are ranked first.

[0044] An eighth vehicle re-identification optimization method based on spatiotemporal clustering reordering according to the first aspect of this application is provided, wherein in the re-identification results of the image to be identified, the images within each retrieved image cluster are ordered according to their order in the matching image retrieval sequence.

[0045] A first vehicle re-identification optimization method based on spatiotemporal clustering re-ranking according to a second aspect of this application is provided, comprising: acquiring an image to be identified and its temporal and spatial attributes; extracting appearance feature vectors for the image to be identified and images in a candidate image library; ranking the appearance feature vectors of each image in the candidate image library and the image to be identified based on the differences between their appearance feature vectors, thereby obtaining a matching image retrieval sequence; and calculating the spatiotemporal similarity between each image in the matching image retrieval sequence and the image to be identified, wherein the spatiotemporal similarity represents the matching image retrieval sequence. The differences between each image in the image to be identified and the estimated vehicle trajectories of each vehicle in the image to be identified are analyzed. The first N images of the matching image retrieval sequence are clustered to obtain multiple retrieval image clusters, where clustering is based on the spatiotemporal similarity of the first N images and / or the unique identifier of the target vehicle included in each of the first N images, where N is a positive integer. Each retrieval image cluster is ordered based on the distance between the image to be identified and the trajectories represented by each retrieval image cluster, or the probability that the image to be identified belongs to the trajectories represented by each retrieval image cluster. The images of each retrieval image cluster are output in the sorted order as the re-identification results of the image to be identified.

[0046] A second vehicle re-identification optimization method based on spatiotemporal clustering re-ranking according to a second aspect of this application is provided, comprising: acquiring an image to be identified and its temporal and spatial attributes; extracting appearance feature vectors for the image to be identified and images in a candidate image library; ranking the appearance feature vectors of each image in the candidate image library and the image to be identified based on the differences in appearance feature vectors between the image to be identified and each image in the candidate image library, thereby obtaining a matching image retrieval sequence; clustering the first N images of the matching image retrieval sequence to obtain multiple retrieval image clusters, wherein the clustering is based on the spatiotemporal similarity of the first N images and / or the unique identifier of the target vehicle included in each of the first N images, where N is a positive integer; ranking each retrieval image cluster based on the distance between the image to be identified and the driving trajectory represented by each retrieval image cluster or the probability that the image to be identified belongs to the driving trajectory represented by each retrieval image cluster; and outputting the images of each retrieval image cluster in the sorted order as the re-identification result of the image to be identified.

[0047] An information processing apparatus according to a third aspect of this application is provided, including a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements one of the vehicle re-identification optimization methods based on spatiotemporal clustering reordering according to the first and second aspects of this application. Attached Figure Description

[0048] When read in conjunction with the accompanying drawings, and by referring to the following detailed description of illustrative embodiments, this application, its preferred modes of use, and further objects and advantages will be best understood, wherein the drawings include:

[0049] Figure 1 The diagram illustrates how different cameras on a lane photograph the same vehicle.

[0050] Figure 2 This diagram illustrates misidentified images that have similar appearance features but conflicting timelines.

[0051] Figure 3 This diagram illustrates misidentified images that have similar appearance features but conflicting spatial relationships.

[0052] Figure 4 This presents yet another example of an image to be identified.

[0053] Figure 5 This demonstrates, in yet another example, the image retrieval results obtained by re-identifying the image based on the image to be identified using the HRCN model.

[0054] Figure 6 Showing Figure 5 Scatter plot of the temporal-spatial distribution of image retrieval results.

[0055] Figure 7 A block diagram is shown of a vehicle re-identification optimization method based on spatiotemporal clustering reordering according to an embodiment of this application.

[0056] Figures 8A-8C The trajectories of vehicles with the same appearance from a candidate image library of various embodiments are shown.

[0057] Figure 9 The distribution of images in the matching image retrieval sequence under extreme conditions is shown in the dimensions of temporal and spatial feature gaps.

[0058] Figure 10 Showing Figure 9 The distribution of images in the matching image retrieval sequence across appearance features, temporal data, and spatial data dimensions.

[0059] Figure 11 The illustration shows a scenario where correctly recognized images are generated despite significant time-dimension differences.

[0060] Figure 12 The diagram illustrates a scenario where recognition images are generated with significant differences in both the temporal and spatial dimensions. Detailed Implementation

[0061] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0062] Figure 6 Showing Figure 5 Scatter plot of the temporal-spatial distribution of image retrieval results.

[0063] Figure 6 In the image search results, the horizontal axis represents the image and the image in the search results. Figure 4 The spatial distance between the images to be identified and the images in the image retrieval results is represented by the vertical axis. Figure 4 The difference in the time the images to be identified were captured is shown in the figure. The marked points in the figure represent images in the image retrieval results, and the numbers next to the marked points represent the image IDs. See also Figure 5 , Figure 5 Each image in the image is also labeled with an image ID, and the same image ID indicates that the vehicle in these images is the same vehicle.

[0064] Figure 6 In the data, the image with id=295 is located below the region forming the coordinate axes, while the image with id=300 is located above the region forming the coordinate axes. Through comprehensive analysis... Figure 5 and Figure 6 The results show that the distribution of some incorrectly identified images that are similar in appearance features to the image to be identified is significantly different from the distribution of the image to be identified and its corresponding correctly identified image in the time and space dimensions. Therefore, according to the embodiments of this application, this difference in time and space distribution can be used to optimize the results of vehicle re-identification.

[0065] Figure 7 A block diagram is shown of a vehicle re-identification optimization method based on spatiotemporal clustering reordering according to an embodiment of this application.

[0066] See Figure 7 The proposed vehicle re-identification optimization method based on spatiotemporal clustering reordering consists of three stages: (1) data extraction stage, (2) network model processing stage, and (3) image retrieval stage.

[0067] In (1) the data extraction stage, temporal and spatial attributes are extracted from the image to be identified and the images in the candidate image library. The temporal attribute of the image is, for example, the timestamp data of the image being acquired, while the spatial attribute of the image is the pose of the vehicle in the image. Optionally, the position of the camera that acquired the image is used as the pose of the vehicle in the image (e.g., using the horizontal coordinates of the camera position as the spatial attribute of the image, ignoring the orientation and height of the vehicle pose). Still optional, the image is acquired by a wide-angle camera located at a high altitude, or by satellite, in which case the image covers a large spatial range, and the spatial attribute of the image is further determined based on the spatial range covered by the image and the position / orientation of the vehicle in the image. And the differences in temporal and spatial attributes between the image to be identified and multiple or all images in the candidate image library are calculated as the temporal and spatial data of each image in the candidate image library.

[0068] (2) In the network model processing stage, the image to be identified and the images in the candidate image library are input into the network model (ReID network model) used for re-identification (e.g., HRCN / ASSEN / UFDN). The network model outputs the appearance feature vectors of the image to be identified and the images in the candidate image library. Based on the similarity between the appearance feature vectors of each image in the candidate image library and the appearance feature vector of the image to be identified, the images in the candidate image library are first reordered (called the first reordering) to obtain a matching image retrieval sequence based on the appearance feature vectors.

[0069] This involves using a pre-trained network model for vehicle image re-identification. Optionally, the network model is also trained using a dataset to output appearance feature vectors for input images.

[0070] Optionally, in application, the ReID network model can pre-extract appearance feature vectors for each image in the candidate image gallery, thus eliminating the need to obtain the image to be identified (Query) and then wait for the appearance feature vector extraction results. Still optionally, depending on application requirements, the candidate image gallery images compared with the image to be identified (Query) can be filtered, for example, images within a specified range and / or region can be selected, thereby reducing the scale of the appearance feature vector comparison and ranking process.

[0071] (3) In the image retrieval stage, the matching images (obtained from stage (2)) are clustered or classified according to their attribute information to form multiple clusters. Each cluster is then reordered based on the differences between each cluster and the image to be identified (called secondary reordering), while the relative order of the matching images within each cluster remains unchanged. Based on the sorting results of the clusters, matching images are obtained from each cluster, and finally, a matching image retrieval sequence based on features and spatiotemporal information of the image to be identified is obtained. The images ranked higher in the final matching image retrieval sequence come from the clusters ranked higher in the secondary reordering, and the order of the images within each cluster remains the same in the final matching image retrieval sequence.

[0072] In one example, a normalization algorithm is used to process the temporal and spatial data (obtained from stage (1)) of the matching images (obtained from stage (2)) to obtain a normalized spatiotemporal vector, which includes, for example, a temporal dimension and a spatial dimension. Then, a clustering operation is performed on the spatiotemporal vectors of the first N images of the appearance feature-based matching image retrieval sequence (obtained from stage (2)) to cluster images with similar spatiotemporal vectors into multiple clusters.

[0073] In one example, matching images are clustered or classified according to the unique identifier of the target vehicle in the image (e.g., license plate number), with images containing vehicles with the same identifier grouped together to form multiple clusters. In another example, matching images are clustered according to the unique identifier of the target vehicle within them, as well as multi-dimensional attributes such as the spatiotemporal vectors of each image.

[0074] In one example, the maximum value of the temporal data of the images within a cluster is used as the cluster's temporal data. The clusters are then sorted by their temporal distance from the images to be identified in the temporal dimension.

[0075] In another example, the maximum value of the spatial data of the images within a cluster is used as the spatial data of the cluster. The clusters are then sorted by the spatial distance between their spatial data and the images to be identified in the spatial dimension.

[0076] Figure 7 The network model used in the network model processing stage is an existing or potentially future network model. The technical improvement of this application embodiment does not lie in the network model (not in the structure of the network model and its training process). Furthermore, the input image provided to the network model does not include the temporal / spatial attributes of the image to be identified, nor does it include the temporal / spatial data of the images in the candidate image library. Instead, based on the matching image retrieval sequence obtained from the network model's output, clustering is further performed using the temporal / spatial data of each image in the matching image retrieval sequence, and a secondary reordering is performed based on the clustering results. This optimizes the matching image retrieval sequence by adjusting the order of the images within the sequence.

[0077] According to embodiments of this application, the matching image retrieval sequence obtained from the output of the network model is clustered according to spatiotemporal vectors, dividing the matching image retrieval sequence into multiple clusters. Multiple images within each cluster have a smaller distance in the temporal / spatial dimensions compared to images in other clusters.

[0078] When multiple images within a cluster include the correctly identified image, images that are similar in appearance to the correct image in the temporal / spatial dimensions are also likely to be correctly identified. Therefore, even if some images in the cluster have low similarity in appearance features to the image to be identified due to reasons such as camera perspective, road and vehicle occlusion, etc., they are still assigned a higher ranking in the final optimized matching image retrieval sequence (e.g., with a lower Rank value) because they belong to the cluster, thus improving the accuracy of the re-identification results.

[0079] When multiple images within a cluster include incorrectly identified images that conflict with the image to be identified in the temporal or spatial dimensions, the cluster is relatively far from the correct image in the temporal / spatial dimensions (due to the conflict). Therefore, the cluster can filter and exclude incorrectly identified images within a certain spatial / temporal range. Even if some images within the cluster have high similarity in appearance features to the image to be identified (e.g., the same color and the same car model), the accuracy of the re-identification result can be improved by assigning these images a lower ranking in the final optimized matching image retrieval sequence.

[0080] It is understood that, according to the embodiments of this application, it cannot be guaranteed that there are no misidentified images in the top-ranked clusters. For example, for vehicles that have similar appearance features to the vehicle in the image to be identified and are traveling in the same direction, their images are misidentified images, but it is difficult to filter and exclude them using the methods of the embodiments of this application.

[0081] Furthermore, it is possible to predict the regions in the image to be identified where vehicles may or are highly likely to appear in the spatial / temporal dimensions, and assign higher rankings to clusters of neighboring such regions during the secondary re-ranking. For example, based on the road, direction of travel, speed range, and time of image acquisition of the vehicle in the image to be identified, the position of the vehicle in the image at different times can be estimated, thereby estimating the regions in the spatial / temporal dimensions where it may or is highly likely to appear.

[0082] Return to view Figure 7 The technical improvement portion embodying the embodiments of this application is referred to as the STcluster component. The processing performed by the STcluster component is divided into three stages: the input processing stage, the clustering processing stage, and the secondary reordering stage.

[0083] In the input processing stage, in addition to the input image to be identified (Query) and the candidate image library (Gallery), the temporal and spatial attributes of the images in the Query and the candidate image library (Gallery) are added as input. For example, the temporal attribute is the timestamp recorded when each image was acquired, and the spatial attribute of the image is the location of the camera that captured the image, recorded when each image was acquired.

[0084] Optionally, a camera distance matrix representing the distance between each camera is pre-established, with both dimensions being camera identifiers, and the elements of the distance matrix representing the distance between two cameras. Thus, based on the cameras that acquired the two images, the distance between the cameras that acquired the two images is retrieved from the camera distance matrix, serving as the spatial data for the two images.

[0085] For images (including the image to be identified (Query) and images from the candidate image library (Gallery)) still serving as input to the network model, appearance feature extraction is performed, which is a prior art technique. The newly added temporal and spatial attributes need to be processed to transform them into suitable data formats in preparation for subsequent clustering and secondary re-ranking. This application provides various implementation methods for temporal and spatial data processing.

[0086] As one implementation method, based on the image acquisition time and intersection information, the feature differences in the temporal and spatial dimensions between the image to be identified (Query) and each image in the candidate image library (Gallery) are calculated as the temporal and spatial data of the images in the candidate image library (Gallery). Then, the temporal and spatial data of the images in the candidate image library (Gallery) are normalized to remove the dimensions of the temporal and spatial feature differences. The normalization method can use algorithms such as Min-Max normalization or Z-Score normalization. Optionally, the calculation of the temporal and spatial data of the images in the candidate image library (Gallery) can be performed on all images in the candidate image library (Gallery), or it can be performed only on images in the matching image retrieval sequence output from the network model.

[0087] As another implementation method, the temporal and spatial data of each image in the candidate image library (Gallery) are obtained based on the difference in the temporal / spatial dimensions between the image to be identified (Query) and each image in the candidate image library (Gallery). Then, based on additional conditions such as road speed limits and driving trajectories, a specific function is used to calculate the spatiotemporal similarity between the image to be identified (Query) and each image in the candidate image library (Gallery).

[0088] As an example, given that the driving trajectory of a vehicle is expressed as a function f, to cluster images in a candidate image library (Gallery) whose positional attributes are within or near the driving trajectory to filter out vehicles with driving trajectories f, the distance from each image in the query and candidate image library (Gallery) to the driving trajectory f is used as the spatiotemporal similarity between each image in the query and candidate image library (Gallery). Thus, images whose positional attributes are close to driving trajectory f and images whose positional attributes are far from driving trajectory f can be further distinguished through subsequent clustering processing. Optionally, possible driving trajectories can be obtained from road topology, traffic flow routes in the road, and historical driving data of a specified area. Again, optionally, vehicles in each image of the candidate image library (Gallery) are identified, distinguished, and tracked separately to obtain multiple estimated driving trajectories for multiple vehicles in each image of the candidate image library (Gallery). Based on the distances from each image in the candidate image library (Gallery) to each estimated driving trajectory, and the distances from the image to be identified (Query) to each driving trajectory, the spatiotemporal similarity between the image to be identified (Query) and each image in the candidate image library (Gallery) is obtained. Optionally, based on the probability that each image in the candidate image library (Gallery) belongs to each estimated driving trajectory, and the probability that the image to be identified (Query) belongs to each driving trajectory, the spatiotemporal similarity between the image to be identified (Query) and each image in the candidate image library (Gallery) is obtained. Optionally, the driving trajectory of the vehicle to be identified (Query) is also related to factors such as time, vehicle speed, and road speed limits. Therefore, the distance between the image's positional attribute and the driving trajectory f, or the probability that the image's positional attribute belongs to the driving trajectory f, can be jointly expressed by attributes such as the vehicle's pose (position and orientation), time, driving speed, and road speed limits in the image, thus obtaining the spatiotemporal similarity between the image to be identified (Query) and each image in the candidate image library (Gallery). Optionally, the spatiotemporal similarity is expressed in multiple dimensions.

[0089] See also Figure 8A This displays the trajectories of vehicles with the same appearance within a candidate image gallery. Typically, a vehicle is recorded multiple times within the gallery. For example, multiple cameras along a route may capture multiple images of the same vehicle sequentially. These images represent the vehicle's trajectory. Figure 8AIn this process, the pose and temporal attributes of images in the candidate image library (Gallery) are used to fit trajectory 1 and trajectory 2. The images represented by trajectory 1 and trajectory 2 respectively exhibit a clear linear distribution in the time-pose coordinate system. Therefore, trajectory 1 likely represents multiple images taken of the same vehicle while it is traveling at a constant speed, while trajectory 2 likely represents multiple images taken of another vehicle while it is traveling. The slope of trajectory 1 / trajectory 2 also reflects the speed of the vehicles they represent. The closer the image to be identified (Query) is to a particular trajectory, the greater the probability that it belongs to the same vehicle as the corresponding trajectory. It can be understood that the images represented by trajectory 1 and trajectory 2, as well as the image to be identified (Query), have similar appearances. Therefore, trajectory 1 and trajectory 2 come from different vehicles with similar appearances. Accordingly, in the secondary re-ranking, the image represented by trajectory 1 is ranked before the image represented by trajectory 2 in the image similarity-based retrieval results in the candidate image library (Gallery).

[0090] Figure 8B The trajectory of a vehicle with the same appearance in a candidate image gallery is shown in yet another embodiment. Figure 8B In the example, the candidate image library (Gallery) contains few images with a similar appearance to the image to be identified (Query), in which case the trajectory may not be linear. For this situation, a simple clustering method can be used ( Figure 8B In the process, clustering is used to obtain cluster 1 and cluster 2 to distinguish trajectories. Then, the distance between the image to be identified (Query) and each cluster is used as the basis for sorting the images in the search results. For example, the Euclidean distance from the image to be identified (Query) to the cluster center, or the Euclidean distance from the image to be identified (Query) to the nearest point in the cluster, is used as the distance from the image to be identified (Query) to the cluster.

[0091] Figure 8C The trajectory of a vehicle with the same appearance in a candidate image gallery is shown in yet another embodiment.

[0092] In some cases, the image to be identified (Query) and vehicle images with the same appearance in the candidate image library (Gallery) may be far apart in the time-pose coordinate system. In such cases, the probability of the image to be identified (Query) falling into the trajectory can be determined by observing the "trend" of the predicted trajectory. See also Figure 8C The system fits trajectory 1 and trajectory 2 to vehicle images from a candidate image library (Gallery), and also predicts the "trend" based on trajectory 1 and trajectory 2. The "trend" represents, for example, the estimated spatial position / attitude of the vehicle within a specified time range. Figure 8CIn this model, the image to be identified (Query) lies within the "trend" range of trajectory 1, thus the distance between the image to be identified (Query) and trajectory 1 is closer. Optionally, the probability of the image to be identified (Query) belonging to each trajectory can be used as the basis for re-ranking. Predicting the trajectory trend can be achieved using methods such as AMCL (adaptive Monte Carlo Localization), interpolation, fitting, etc., or future location prediction techniques can be utilized. Conditions such as road speed limits can be used to narrow down the range of ambiguous regions of trajectories far from sample points.

[0093] In the clustering stage, after obtaining the temporal and spatial differences in appearance features between each image in the matching image sequence and the image to be identified (Query image) from the output of the network model, the temporal and spatial feature differences of each image in the matching image sequence are used as a pair (including temporal and spatial data). Clustering is then performed on the pairs of the first n images in the matching image retrieval sequence, aggregating images with similar pairs among these n images (e.g., n=20) into a single cluster, thus obtaining multiple clusters. Optionally or further, clustering is performed based on the spatiotemporal similarity of each image in the matching image sequence. Optionally or further, clustering is performed based on the unique identifier of the target in each image in the matching image sequence. Several clustering schemes are available for the clustering stage:

[0094] Option 1 uses K-Means or DBSCAN clustering methods, taking the temporal and spatial data of the first n images of the matching image sequence as input coordinates, the spatiotemporal phase velocity of the first n images of the matching image sequence as input coordinates, and / or the unique identifier of the target included in the first n images of the matching image sequence as input coordinates to cluster the first n images into multiple clusters.

[0095] Option 2 employs an unequal-weight clustering method. Based on the aforementioned characteristic of high vehicle speed, different weights are assigned to temporal and spatial data to cluster the first n images into multiple clusters. Alternatively, different weights are assigned to different dimensions of temporal data, spatial data, spatiotemporal similarity, and / or the unique identifier of the target included in the image to cluster the first n images into multiple clusters.

[0096] The clustering algorithms of Scheme 1 and Scheme 2 have different inputs. In Scheme 1, the attributes of the matched image sequence are weighted equally, while in Scheme 2, the attributes of the matched image sequence are assigned different weights (for example, because vehicles move at high speeds, the range they can reach in a short time is relatively large, so time data is given greater weight, and images with small time data but suitable spatial data are clustered into a cluster).

[0097] In the secondary reordering stage, after simultaneously obtaining the matching image sequences based on normalized temporal and spatial data, spatiotemporal similarity, and / or unique identifiers of targets in the images, as well as the matching image sequences based on appearance features, the appearance feature-based matching image sequences are reordered secondaryly according to the clustering results using the temporal and spatial data (or based on spatiotemporal similarity) of the clusters. This yields the final secondary reordered sequence of matching image sequences based on temporal and spatial data (or based on spatiotemporal similarity). Several schemes exist for performing secondary reordering on matching image sequences:

[0098] Option 1, Time-weighted Priority Strategy: Time weight is used as the standard for secondary re-ranking. The maximum time value of all images within each image cluster is used as the cluster's time data (because the secondary re-ranking stage aims to filter out incorrectly identified images with temporal and spatial conflicts from previous modeling; therefore, the maximum time value of all images within a cluster is chosen as the time data for each cluster to reflect temporal conflicts). Image clusters that are closer to the image to be identified in the time dimension are prioritized and ranked higher, while the relative order of matching images within the cluster remains unchanged.

[0099] Option 2, Distance Weight Priority Strategy: Spatial weight is used as the standard for secondary re-ranking. The spatial data of each image cluster is the maximum value of the spatial data of the images within the cluster. Image clusters that are spatially closer to the image to be identified are prioritized and ranked higher, while the relative order of matching images within the cluster remains unchanged.

[0100] Option 3, a combined time and distance weight strategy: This approach uses both time and spatial weights as the criteria for secondary re-ranking. Based on the characteristics of vehicle motion, reasonable regions are divided in both time and space dimensions. The shortest distance from each cluster to this reasonable region is calculated as the re-ranking standard. Image clusters that are closer to the reasonable region in both time and space dimensions are prioritized and ranked higher, while the relative order of matching images within the same cluster remains unchanged.

[0101] Option 4, a combined strategy of time and distance conflicts: For multiple clusters formed by clustering, calculate the time and distance data for each cluster. Based on the time and distance data, identify clusters with time and distance conflicts, and rank these clusters lower among the multiple clusters. For example, use the ratio of a cluster's distance data to its time data as the speed of that cluster relative to the target in the image to be identified. Clusters whose speed relative to the target in the image deviates significantly from (greater than or less than) the expected speed range of the vehicle target are identified as having conflicts.

[0102] Option 5: Sort the clusters based on the distance from the query image to the driving trajectory represented by each cluster or the probability of belonging to the driving trajectory represented by each cluster. Prioritize clusters with smaller distances or higher probabilities and sort them first, while the relative order of matching images within each cluster remains unchanged.

[0103] Experiment 1

[0104] We added the spatiotemporal clustering reordering method (denoted as STcluster) according to the embodiments of this application to the network models (HRCN and ASSEN models) that currently perform well in vehicle re-identification domain classification experiments, and conducted tests. The test dataset was the standard VeRi-776 dataset, and the test results are summarized in Table 1. The clustering algorithm used was the DBSCAN algorithm with a cluster radius of 0.35; the normalization algorithm used was the Min-Max normalization algorithm.

[0105] Table 1 Statistical table of Experiment 1 results

[0106] HRCN 96.84％ 98.75％ 99.46％ 82.78％ HRCN+STcluster(Ours) 98.27％ 99.28％ 99.70％ 83.08％ HRCN (best) 97.32％ 98.87％ 99.17％ 82.83％ HRCN (best) + STcluster (Ours) 98.09％ 99.40％ 99.52％ 83.47％ ASSEN 96.48％ 98.63％ 99.17％ 81.46％ ASSEN+STcluster(Ours) 97.02％ 98.93％ 99.40％ 81.84％

[0107] Interpretation of the experimental results in Table 1:

[0108] The first column of Table 1 lists the model / method names, where HRCN, HRCN(best), and ASSEN represent the experimental results (control group) without using the STcluster method of this application to optimize vehicle retrieval results. HRCN(best) represents the best-performing model obtained after multiple training iterations of the HRCN model. HRCN+STcluster, HRCN(best)+STcluster, and ASSEN+STcluster represent the experimental results (experimental group) after incorporating the STcluster method of this application to perform secondary re-ranking optimization of vehicle image retrieval results based on the HRCN and ASSEN models, respectively. The second, third, and fourth columns of Table 1 show the results under different accuracy metrics. According to the previous definitions of accuracy metrics Rank and mAP, Rank-1 represents the frequency that the first image in the vehicle image retrieval result is the correct image (equivalent to CMC@1 in the CMC curve), and Rank-5 represents the frequency that the correct image appears in the first 5 images of the re-ranked sequence (equivalent to CMC@5 in the CMC curve). The last column of Table 1 shows mAP, which represents the average accuracy on each image to be identified. The higher the values ​​of Rank and mAP, the higher the accuracy of the current vehicle image re-identification and the better the vehicle re-identification effect.

[0109] Experiment 2

[0110] Since Z-score normalization does not change the relative position and shape of the sample points, but only the scale of the data, it will not interfere with the clustering results. Therefore, in the embodiments of this application, the effect of using Z-score normalization should be better than that of using Min-Max normalization for vehicle spatiotemporal information data. Therefore, in the data processing stage, the Z-score normalization algorithm was used instead of the Min-Max normalization algorithm, and an advanced experiment (Experiment 2) was conducted. The results of Experiment 2 are shown in Table 2.

[0111] Table 2. Statistical table of Experiment 2 results (using Z-score normalization)

[0112] HRCN 96.84％ 98.75％ 99.46％ 82.78％ HRCN+STcluster(Ours) 98.93％ 99.70％ 99.76％ 83.20％ HRCN (best) 97.32％ 98.87％ 99.17％ 82.83％ HRCN (best) + STcluster (Ours) 98.63％ 99.40％ 99.52％ 83.61％ ASSEN 96.48％ 98.63％ 99.17％ 81.46％ ASSEN+STcluster(Ours) 97.97％ 99.23％ 99.40％ 81.99％

[0113] After adopting the secondary reordering according to the embodiments of this application, it was further discovered that some incorrectly identified images that are similar to some correctly identified images in the time and space dimensions but have large differences in appearance features are aggregated together in an extreme case.

[0114] Figure 9 The distribution of images in the matching image retrieval sequence under extreme conditions is shown in the dimensions of temporal and spatial feature gaps. Figure 9 In the figure, the temporal and spatial feature differences of the incorrectly identified image with id 35 are similar to those of the partially correct identified images (as shown by the dashed circles in Figure 8). Therefore, during spatiotemporal feature-based clustering, the identified images within the dashed circles are clustered into the same cluster. Such clustering results reduce the accuracy of the re-identification task.

[0115] According to another embodiment of this application, in order to handle such extreme cases and further improve the accuracy of image re-identification, it is proposed to further introduce the appearance feature dimension output by the network model for the input image into the STcluster component, and combine the appearance feature dimension, time data and spatial data into triples for three-dimensional spatial clustering. In this way, the appearance feature difference can be used to further increase the feature difference between the misidentified image and the correctly identified image in the clustering space, thereby improving the accuracy of the clustering algorithm.

[0116] Figure 10 Showing Figure 9 The distribution of images in the matching image retrieval sequence across appearance features, temporal data, and spatial data dimensions.

[0117] Figure 10The vertical axis represents the appearance feature dimension. The difference between the misidentified image with id 35 and the correctly identified image with id 38 in the appearance feature dimension is further increased. At this time, clustering algorithm can easily exclude the misidentified image with id 35 from the cluster of correctly identified images (clustering them into different clusters), thereby further using the time difference to filter out the cluster composed of the misidentified image with id 35.

[0118] After performing three-dimensional spatial clustering, the multiple clusters obtained from the clustering are then reordered.

[0119] As an example, since the appearance feature dimension has already been utilized in 3D spatial clustering, the resulting clusters possess appearance feature attributes. For instance, the mean or extreme value of the appearance feature vectors of each image within a cluster can be used as the cluster's appearance feature vector. When performing secondary re-ranking on the clusters formed by clustering, the cluster's appearance feature vector, the cluster's temporal data, and / or the cluster's spatial data can all be used as the basis for ranking. Optionally, during ranking, corresponding weights can be assigned to the cluster's appearance feature vector, the cluster's temporal data, and / or the cluster's spatial data. For example, the cluster's appearance feature vector has the highest weight, thus having the greatest influence in the secondary re-ranking; the cluster's temporal data is assigned a moderate weight; and the cluster's spatial data is assigned a relatively minimum weight.

[0120] Alternatively, other strategies can be used for secondary re-sorting. For example, clusters that conflict with the image to be identified in time / space can be filtered out first based on the cluster's temporal / spatial data, and these clusters can be sorted last. Then, the remaining clusters can be sorted based on their appearance feature vectors, temporal data, and / or spatial data.

[0121] Experiment 3

[0122] Based on Experiment 2, the experiment was repeated using an example of a clustering method employing three dimensions—appearance features, temporal data, and spatial data—during the clustering phase. The experimental results are shown in Table 3.

[0123] Table 3. Statistical table of Experiment 3 results (using Z-score normalization + adding appearance feature dimension)

[0124] HRCN (best) 97.32％ 98.87％ 99.17％ 82.83％ HRCN (best) + STcluster (Ours) 98.87％ 99.52％ 99.52％ 83.66％ ASSEN 96.48％ 98.63％ 99.17％ 81.46％ ASSEN+STcluster(Ours) 98.39％ 99.28％ 99.52％ 82.03％ UFDN 96.20％ 98.70％ 99.20％ 81.50％ UFDN+STcluster(Ours) 98.60％ 99.20％ 99.40％ 82.30％

[0125] Experimental data demonstrates that the vehicle re-identification optimization method based on spatiotemporal clustering re-ranking proposed in this application can solve the problem that some vehicle re-identification network models based on appearance feature differences cannot screen out erroneous images with different temporal and spatial distributions of the images to be identified (Query) during the image retrieval and re-ranking stages, and can further improve the accuracy of vehicle re-identification tasks in the image retrieval stage.

[0126] The vehicle re-identification optimization method based on spatiotemporal clustering reordering according to the embodiments of this application is particularly suitable for vehicle re-identification in short-term road traffic. For example, the scenario represented by the VeRi-776 dataset belongs to such a short-term scenario. When the acquisition time span of the candidate dataset (Gallery) images is large (e.g., there are images spanning multiple days), due to the difficulty in effectively identifying temporal / spatial conflicts, the accuracy index of the image retrieval results using this method is relatively lower than the accuracy index in short-term scenarios. The specific reasons are as follows: (1) In long-term road traffic environments, the time span of the same camera acquiring the same vehicle target may be too large (e.g., the same vehicle is acquired within several days in the field of view of the same camera or adjacent cameras); (2) In long-term road traffic environments, the orientation of the vehicle target movement in the image to be identified (Query) will be very diverse, and there will be many other cameras far away from the camera that acquired the image to be identified (Query) that have also acquired candidate image library (Gallery) images of the vehicle target. Therefore, in this case, the two characteristics of vehicle movement mentioned above will no longer hold, and therefore the scenario of temporal and spatial conflicts of incorrectly identified images proposed based on these characteristics will no longer exist. However, it is understandable that even in long-term road traffic scenarios, since the candidate image gallery necessarily includes images of short-term road traffic scenarios, and the embodiments of this application can effectively filter and exclude incorrectly identified images in short-term road traffic scenarios, applying the embodiments of this application also helps to improve the accuracy of retrieval results for long-term road traffic scenarios.

[0127] Figure 11 The illustration shows a scenario where correctly recognized images are generated despite significant time-dimension differences.

[0128] Figure 11 This demonstrates a scenario where temporal conflicts are difficult to determine in long-term road re-identification datasets. Camera 1 acquires an image to be identified (Query, id=1), and then after a long period of time (e.g., 24 hours), camera 1 acquires another image (as an image in the Gallery) that is similar in appearance to the image to be identified. However, due to the large time difference between the two images, based on the vehicle motion characteristics proposed earlier (2), the vehicle object corresponding to the image to be identified (Query) may move back into the field of view of camera 1, making it impossible to determine whether there is a temporal conflict between the image to be identified (Query) and the similar-looking retrieved image. Such scenarios may occur during periodic activities such as commuting to work.

[0129] Figure 12 The diagram illustrates a scenario where recognition images are generated with significant differences in both the temporal and spatial dimensions.

[0130] Figure 12 This demonstrates the difficulty in determining spatial conflicts in long-term road re-identification scenarios. Camera 1 acquires an image to be identified (Query), and then after a long period of time (e.g., 12 hours), camera 5, located far from camera 1 (e.g., 5km), acquires another image that is similar in appearance to the image to be identified (Query) (as an image in the Gallery). At this point, due to the large time difference between the two images, based on the vehicle motion characteristics (1) mentioned earlier, the vehicle object corresponding to the image to be identified (Query) may have moved into the field of view of camera 5. Therefore, it is impossible to determine whether there is a spatial conflict between the image to be identified (Query) and the similar-looking retrieved image.

[0131] However, even in Figure 11 and Figure 12 In the scenario shown, according to the embodiments of this application, it is impossible to determine whether the image to be identified (Query) and the searched image with a similar appearance conflict. However, when clustering images in the matching image search sequence, such searched images that cannot be identified as having conflicts will be retained in one or more clusters (without being discarded or excluded from the results), and after secondary reordering, they will be retained in the optimized matching image search sequence (their ranking may change compared to the original matching image search sequence), so that users can still obtain such searched images.

[0132] Although the examples referred to in the present application are described for illustrative purposes only and not for limiting the scope of the application, changes, additions and / or deletions to the implementation may be made without departing from the scope of the application.

[0133] Those skilled in the art who benefit from the teachings presented in the above description and associated drawings will recognize many modifications and other embodiments of the present application described herein. Therefore, it should be understood that the present application is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terminology is used herein, it is used only in its general and descriptive sense and not for limiting purposes.

Claims

1. A vehicle re-identification optimization method based on spatiotemporal clustering re-ranking, comprising: Obtain the image to be identified, as well as the temporal and spatial attributes of the image to be identified; Extract appearance feature vectors from the image to be identified and the images in the candidate image library, respectively. Based on the difference between the appearance feature vectors of the image to be identified and each image in the candidate image library, the similarity between the appearance feature vectors of each image in the candidate dataset and the image to be identified is ranked to obtain a matching image retrieval sequence; The differences in temporal and spatial attributes between each image in the matching image retrieval sequence and the image to be identified are calculated and used as temporal and spatial data for each image in the matching image retrieval sequence. The temporal and spatial data of each image in the image retrieval sequence are normalized to obtain the spatiotemporal vector of each image in the image retrieval sequence, which includes the temporal dimension and the spatial dimension. Clustering is performed on the first N images of the matching image retrieval sequence to obtain multiple retrieval image clusters, wherein the clustering is performed based on the spatiotemporal vectors of the first N images, where N is a positive integer; The search image clusters are sorted according to the differences in the temporal and / or spatial attributes between the image to be identified and each search image cluster. The images of each retrieved image cluster are output as the re-identification results of the image to be identified, according to the sorting order of each retrieved image cluster; wherein, the sorting of each retrieved image cluster based on the differences in the temporal and / or spatial attributes between the image to be identified and each retrieved image cluster includes: The maximum value of the time data of the images within each searched image cluster is used as the time attribute of the cluster. The searched image clusters are sorted according to the difference between the time attributes of the image to be identified and each searched image cluster, with the searched image clusters with smaller differences in time attributes from the image to be identified being sorted first. The maximum spatial data value of the images within each retrieved image cluster is used as the spatial attribute of the cluster. The retrieved image clusters are then sorted according to the differences in spatial attributes between the image to be identified and each retrieved image cluster, with the retrieved image clusters having smaller differences in spatial attributes from the image to be identified ranked first; or Based on the motion characteristics of vehicles in the image, the combination of time and space dimensions is divided into multiple designated regions, each representing a space in which the estimated vehicle in the image should be located at a specified time. Each retrieved image cluster is sorted according to its distance from the multiple designated regions, with the retrieved image cluster closest to the multiple designated regions being sorted first.

2. The method according to claim 1, wherein The appearance feature vectors of the image to be identified and the images in the candidate image library are extracted by a feature extraction neural network.

3. The method according to claim 1 or 2, wherein The temporal attribute of an image is the time when the image was captured; The spatial attribute of an image is the pose of the vehicles in the image.

4. The method according to claim 1, wherein The normalization of the temporal and spatial data of each image in the image retrieval sequence for matching includes: The temporal and spatial data of each image in the image retrieval sequence are normalized using the Min-Max method or the Z-Score method, respectively.

5. The method according to claim 1, wherein The clustering of the first N images of the matching image retrieval sequence is performed based on the spatiotemporal vectors and appearance feature vectors of the first N images, and clustered according to the three dimensions of time, space and appearance feature vectors.

6. The method according to claim 1, wherein The clustering of the first N images of the matching image retrieval sequence involves setting different weights for the time and space dimensions in the spatiotemporal vectors of the first N images, and then clustering them according to the weighted time and space dimensions, where the weight of the time dimension is greater than the weight of the space dimension.

7. The method according to claim 1, wherein In the re-identification results of the image to be identified, the images within each retrieved image cluster are sorted according to their order in the matching image retrieval sequence.

8. A vehicle re-identification optimization method based on spatiotemporal clustering re-ranking, comprising: Obtain the image to be identified, as well as the temporal and spatial attributes of the image to be identified; Extract appearance feature vectors from the image to be identified and the images in the candidate image library, respectively. Based on the difference between the appearance feature vectors of the image to be identified and each image in the candidate image library, the similarity between the appearance feature vectors of each image in the candidate dataset and the image to be identified is ranked to obtain a matching image retrieval sequence; Clustering is performed on the first N images of the matching image retrieval sequence to obtain multiple retrieval image clusters, wherein the clustering is based on the spatiotemporal similarity of each of the first N images and / or the unique identifier of the target vehicle included in each of the first N images, where N is a positive integer; and wherein the spatiotemporal similarity between the image to be identified and each image in the candidate image library is obtained by using the distance to the driving trajectory or the probability of belonging to the driving trajectory of each of the first N images of the candidate image library; The search image clusters are sorted according to the distance between the image to be identified and the driving trajectory represented by each search image cluster, or the probability that the image to be identified belongs to the driving trajectory represented by each search image cluster. The images of each retrieved image cluster are output in the sorted order as the re-identification results of the image to be identified.

9. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1-8.

Images