Vehicle cross-mirror tracking method and device, perception device, medium and product

Abstract

The application relates to the technical field of intelligent transportation, in particular to a vehicle cross-mirror tracking method and device, a perception device, a medium and a product, which comprise the following steps: acquiring front and back view images of at least one target vehicle; inputting the front view image into a first feature extraction model, the first feature extraction model extracting a front view feature vector and generating an initial back view feature vector, and merging to obtain a candidate feature vector; inputting the back view image into a second feature extraction model, the second feature extraction model outputting a back view feature vector of each target vehicle to obtain a target feature vector; storing the identification and the candidate view feature vector of each target vehicle into a candidate list, and performing similarity matching on the target feature vector and the candidate feature vector, so that the target vehicle of cross-mirror tracking is given. Thus, the problem of "large intra-class gap and small inter-class gap of vehicle appearance features" in the vehicle re-identification of the cross-mirror tracking task is solved.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a method, device, sensing equipment, medium and product for cross-view vehicle tracking. Background Technology

[0002] Tracking algorithms are an important research direction in the field of computer vision, playing a fundamental and crucial role in various perception tasks. Cross-camera tracking algorithms, as an important extension of traditional tracking algorithms, focus on achieving target re-identification across multiple camera viewpoints. This ensures that the same target has a unified ID (Identifier) ​​under different lenses and achieves continuous and seamless tracking when the target traverses the field of view of different cameras.

[0003] In related technologies, cross-camera tracking algorithms mainly include the following methods: While target feature matching can achieve accurate target recognition, it heavily relies on high-resolution images and substantial computing resources. Its effectiveness is limited, especially when vehicle features differ significantly across multiple viewpoints. Spatial trajectory relationship matching is relatively simple and resource-efficient, but its accuracy heavily depends on the precision of multi-target tracking trajectories and traffic speed measurement. Therefore, it is prone to ID swapping, missed detections, and ID loss when targets are occluded or numerous. Combining feature matching and trajectory matching aims to compensate for the shortcomings of the previous two methods by fusing target features and trajectory information to enhance matching reliability. However, it still suffers from the limitations of the aforementioned methods, and its robustness and applicability need improvement. Matching methods incorporating temporal relationships further enhance the robustness and applicability of the algorithm based on the previous three methods. However, this method requires advance measurement and planning of specific road scenarios, thus lacking universality and primarily applicable to offline scenarios to improve re-identification accuracy. Summary of the Invention

[0004] This application provides a method, apparatus, sensing device, medium, and product for cross-camera vehicle tracking, in order to alleviate problems such as real-time issues, weak robustness, and low matching accuracy in cross-camera tracking.

[0005] The first aspect of this application provides a method for cross-camera vehicle tracking. The method is applied to a roadside intelligent sensing device, which is equipped with a first camera and a second camera. The first camera captures a frontal view image of the vehicle, and the second camera captures a rear view image of the vehicle. The method includes the following steps: acquiring the frontal view image and the rear view image of at least one target vehicle; inputting the frontal view image into a first feature extraction model, which outputs a frontal view feature vector for each target vehicle, wherein the first feature extraction model extracts an initial frontal view feature vector from the frontal view image and generates an initial rear view feature vector based on the frontal view image, and merges the initial frontal view feature vector and the initial rear view feature vector to obtain a candidate feature vector; inputting the rear view image into a second feature extraction model, which outputs a target rear view feature vector for each target vehicle, and copies and merges the target rear view feature vectors to obtain a target feature vector; storing the identifier and candidate feature vector of each target vehicle in a candidate list; performing similarity matching between the target feature vector and the candidate feature vector; selecting the identifier corresponding to the candidate feature vector with the highest similarity and assigning it to the target vehicle for cross-camera tracking.

[0006] Optionally, the first feature extraction model includes a first convolutional neural network model, a second convolutional neural network model, and a conditional generative adversarial network (GAN). The frontal view image is input into both the first convolutional neural network model and the GAN. The first convolutional neural network model outputs the initial frontal view feature vector, and the GAN outputs the initial back view image corresponding to the frontal view image. The back view image is then input into the first convolutional neural network model, which outputs the initial back view feature vector. The initial frontal view feature vector and the initial back view feature vector are merged to obtain the candidate feature vector. The initial back view image is then input into the second convolutional neural network model, which outputs the initial back view feature vector. The target back view feature vector is copied and merged to obtain the target feature vector.

[0007] Optionally, before inputting the frontal view image into the first feature extraction model, the method further includes: acquiring an image dataset of front and rear feature pairs of the vehicle, wherein the image dataset contains frontal view images and rear view images of the vehicle; and training the conditional generative adversarial network using the front and rear feature datasets.

[0008] Optionally, when acquiring the image dataset of front and rear feature pairs of the vehicle, the acquisition parameters of the first camera and the second camera are fixed.

[0009] Optionally, acquiring the frontal view image and the rear view image of at least one target vehicle includes: setting the respective acquisition areas of the first camera and the second camera; acquiring a first acquisition image of the first camera within its acquisition area and a second acquisition image of the second camera within its acquisition area; identifying the first acquisition image to obtain the frontal view image of the at least one target vehicle, and identifying the second acquisition image to obtain the rear view image of the at least one target vehicle.

[0010] Optionally, there is a blind spot between the first camera and the second camera. Acquiring the frontal view image and the rear view image of at least one target vehicle includes: determining the traffic flow density based on the number of vehicles entering the blind spot, and determining the spatial search range of each target vehicle based on the traffic flow density; determining the traffic flow speed based on the image acquired by the first camera, and determining the temporal search range of each target vehicle based on the traffic flow speed; and controlling the first camera and the second camera to acquire images within the spatial search range and the temporal search range.

[0011] A second aspect of this application provides a vehicle cross-view tracking device. The device is applied to a roadside intelligent sensing device, which includes a first camera and a second camera. The first camera captures a frontal view image of the vehicle, and the second camera captures a rear view image of the vehicle. The device comprises: an acquisition module for acquiring a frontal view image and a rear view image of at least one target vehicle; and a first processing module for inputting the frontal view image into a first feature extraction model, which outputs a frontal view feature vector for each target vehicle. The first feature extraction model extracts an initial frontal view feature vector from the frontal view image and a feature vector based on the frontal view image. An initial rear-view feature vector is generated from the corner image. The initial front-view feature vector and the initial rear-view feature vector are merged to obtain a candidate feature vector. A second processing module is used to input the rear-view image into a second feature extraction model. The second feature extraction model outputs the target rear-view feature vector for each target vehicle and copies the target rear-view feature vector for merging to obtain a target feature vector. A target matching module is used to store the identifier and candidate feature vector of each target vehicle into a candidate list. Based on the similarity matching between the target feature vector and the candidate feature vector, the identifier corresponding to the candidate feature vector with the highest similarity is selected and assigned to the target vehicle for cross-camera tracking.

[0012] A third aspect of this application provides a roadside intelligent sensing device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the vehicle cross-mirror tracking method as described in the above embodiments.

[0013] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the vehicle cross-mirror tracking method as described above.

[0014] The fifth aspect of this application provides a computer program product, which, when executed, is used to implement the vehicle cross-mirror tracking method as described in the above embodiments.

[0015] Therefore, this application has the following beneficial effects:

[0016] This application introduces an adaptive spatiotemporal dynamic search mechanism that intelligently adjusts the search range based on traffic flow density. It utilizes a conditional generation algorithm module to generate vehicle appearance features from different angles, alleviating the feature matching challenge. Simultaneously, it constructs a dedicated dataset of vehicle front and rear features. Through a candidate target information storage list and its dynamic management mechanism, candidate vehicle information is maintained in real time, thereby improving the accuracy and real-time performance of vehicle re-identification. This alleviates the technical problems of weak robustness and low matching accuracy in cross-camera tracking.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0019] Figure 1 An architecture diagram of the embedded condition generation algorithm provided for related technologies;

[0020] Figure 2 This is a flowchart of a vehicle cross-mirror tracking method provided according to an embodiment of this application;

[0021] Figure 3 This is a flowchart of a vehicle cross-mirror tracking method according to an embodiment of this application;

[0022] Figure 4 A flowchart of an algorithm submodule according to an embodiment of this application;

[0023] Figure 5 This is a block diagram of a vehicle cross-mirror tracking device according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] Tracking algorithms play a crucial role in computer vision, serving as the foundation for various perception tasks. Among them, cross-camera tracking algorithms, as an important extension of traditional tracking algorithms, focus on re-identifying targets across multiple camera views. This involves assigning the target ID from one camera view to the same target in other views. The goal is to identify and track the same target across multiple cameras, ensuring that the same vehicle target has the same ID in different shots and enabling continuous tracking as the target passes through the fields of view of different cameras.

[0027] Cross-camera tracking algorithms mainly include the following methods: one is based on target average feature matching, or through multi-granularity, multi-level feature fusion and matching to achieve target re-identification. The disadvantages are that it requires very high-resolution original image data to complete multi-level fine-grained matching, and it consumes high computational resources. Furthermore, in the field of vehicle re-identification, the differences in features within the vehicle class are much greater than those for pedestrians, making feature matching based on multi-level fine-grained or average features impractical.

[0028] The second method is a matching method based on spatial trajectory relationships. This involves drawing multi-target tracking trajectories between two shots and matching these trajectories based on the spatial relationship between the two shots to achieve target ID matching. This method is relatively simple and can complete target re-identification with very low computational resource consumption. Its drawback is that it relies heavily on the accuracy of multi-target tracking trajectories within the field of view of each shot. When there are many targets and they are mutually occluded, problems such as ID swapping, missed detections, and lost IDs can easily occur. Cross-shot re-identification based on tracking trajectories with too many errors generally has poor performance and also relies on the spatial relationship and deployment alignment between shots. It cannot reliably complete multi-target re-identification in the absence of overlapping regions.

[0029] Third, this method combines the previous two approaches. While matching multi-target trajectories, it also calculates the similarity of features of multiple targets across multiple mirrors for feature matching, and finally fuses the two for matching. This method can overcome the shortcomings of the first two methods by incorporating target features of each trajectory for matching, thus significantly enhancing the reliability of trajectory matching. However, it still suffers from all the drawbacks of the second method and lacks strong robustness and applicability.

[0030] Fourth, based on the third approach, temporal relationship calculation is added, i.e., target features + spatial relationship + temporal relationship. Before matching, the temporal and spatial relationship information is used to filter out some targets, improving the matching success rate. Then, the spatial relationship, temporal relationship, and feature similarity are fused to complete the target re-identification and tracking across cameras. This method has improved robustness and applicability compared to the third approach. This scheme is the mainstream approach in international MTMCT (Multi-Target Multi-Camera Tracking) cross-camera tracking competition scenarios. However, it requires advance measurement and planning for each specific road scene and lacks universality. It can only improve the accuracy of re-identification by performing offline priors.

[0031] The following description, with reference to the accompanying drawings, outlines a vehicle cross-camera tracking method, apparatus, sensing device, medium, and product according to embodiments of this application. Addressing the issues mentioned in the background section regarding existing vehicle re-identification algorithms' reliance on prior methods, large workload for offline datasets, and difficulty in achieving real-time performance, this application provides a vehicle cross-camera tracking method. This method introduces an adaptive spatiotemporal dynamic search mechanism to intelligently adjust the search range based on traffic flow density; utilizes a conditional generation algorithm module to directionally generate vehicle appearance features from different angles, alleviating the feature matching challenge; simultaneously, it constructs a dedicated vehicle front and rear feature dataset; and maintains candidate vehicle information in real-time through a candidate target information storage list and its dynamic management mechanism, thereby improving the accuracy and real-time performance of vehicle re-identification. This alleviates the real-time issues, weak robustness, and low matching accuracy problems associated with cross-camera tracking.

[0032] Before detailing the vehicle cross-mirror tracking method in this application, it is necessary to understand Conditional Generative Adversarial Networks (cGANs). Specifically, cGANs introduce conditional inputs into the generator and discriminator, and use the Pix2PixHD algorithm to enable the generator to generate more expected output results based on specific conditions. Figure 1As shown, a Pix2PixHD generative algorithm employing a coarse-to-fine strategy is presented, where a global generator G1 and a local enhancement network G2 work collaboratively. In the residual set of G1, the input information is composed of the outputs of the G1 convolutional front end and the G2 deconvolutional back end, cleverly fusing global information with local detail information to generate a more refined image. Specifically, this conditional generative adversarial network effectively guides the generator's output by introducing conditions, such as RGB images, as input. During the training phase, a multi-scale discriminator is employed, utilizing three structurally similar discriminators {D1, D2, D3} to distinguish images at different scales. This design not only saves storage space and alleviates overfitting but also improves discriminative ability by expanding the receptive field. Simultaneously, the finest-scale discriminator promotes the generator to produce finer details, effectively eliminating repetitive details commonly found in generated images, thus achieving accurate differentiation between real and synthetic images at high resolution.

[0033] In summary, by embedding a conditional generative adversarial network into a ReID neural network model, it can generate vehicle image features according to corresponding angles. By training the conditional generative adversarial network with a self-made dataset, the network model can generate a feature view image of the vehicle's rear from a view image of the vehicle's head features as input. This overcomes the difficulty of re-identifying vehicles based on their appearance features in cross-camera tracking, which has a low accuracy rate.

[0034] Specifically, Figure 2 This is a flowchart illustrating a vehicle cross-mirror tracking method provided in an embodiment of this application.

[0035] like Figure 2 As shown, this vehicle cross-view tracking method is applied to a roadside intelligent sensing device. The roadside intelligent sensing device is equipped with a first camera and a second camera. The first camera captures images from the front view of the vehicle, and the second camera captures images from the rear view of the vehicle. The method includes the following steps:

[0036] In step S101, at least one frontal view image and a rear view image of the target vehicle are acquired.

[0037] Among them, the frontal view image can be an image of the scene in front of the vehicle captured by a camera at the front of the vehicle, and the rear view image can be an image of the scene behind the vehicle captured by a camera at the rear of the vehicle.

[0038] It is understood that the embodiments of this application acquire frontal and rear view images of the target vehicle, providing an important foundation and conditions for subsequent vehicle tracking. The frontal view image can reflect information such as road conditions, traffic signs, other vehicles and pedestrians in front of the vehicle, which helps to accurately determine the vehicle's driving status and surrounding environment. The rear view image can show the traffic conditions behind the vehicle, ensuring driving safety.

[0039] In this embodiment of the application, acquiring at least one frontal view image and rear view image of a target vehicle includes: setting the respective acquisition areas of a first camera and a second camera; acquiring a first acquisition image of the first camera within the acquisition area of ​​the first camera and a second acquisition image of the second camera within the acquisition area of ​​the second camera; identifying the first acquisition image to obtain a frontal view image of at least one target vehicle and identifying the second acquisition image to obtain a rear view image of at least one target vehicle.

[0040] The first camera can capture images from the front of the vehicle, and the second camera can capture images from the rear of the vehicle. The capture area can be the spatial range in which the camera can clearly capture and identify the target object. The first captured image can be an image captured by the first camera within its capture area, which may contain various elements such as vehicles, pedestrians, and road signs. The second captured image can be an image captured by the second camera within its capture area, which may mainly show the rear view of the vehicle. The target vehicle can be a specific vehicle that needs to be tracked or identified.

[0041] It is understood that by setting the acquisition areas of the first camera and the second camera, and acquiring and recognizing the corresponding front and rear view images, the embodiments of this application can ensure a more comprehensive monitoring range, which can capture the traffic conditions in front of the vehicle and observe the dynamics behind the vehicle, thereby achieving all-round monitoring of the vehicle's driving status.

[0042] Specifically, such as Figure 3As shown, an advanced tracking scheme combining Yolov8 and DeepOcsort is employed in each of the two single-view fields of view to mitigate issues such as missed detections / false detections leading to ID loss and updates, and target proximity causing ID swapping during single-view tracking. To allow sufficient computation time for the cross-view re-identification task, a "collection zone A" is set up in the road area of ​​the frontal video, near the blind spot of the sensing device. When the center of the target vehicle's detection frame passes through this "collection zone A," the corresponding vehicle target image and vehicle ID are captured and sent to the cross-view re-identification task. Similarly, another "collection zone B" is set up in the rear-view tracking task. When the vehicle exits the blind spot and enters the rear-view field of view in "collection zone B," the corresponding target-detected vehicle image is captured and provided to the cross-view re-identification task for corresponding calculations.

[0043] In this embodiment, there is a blind zone between the first camera and the second camera. Acquiring frontal and rear view images of at least one target vehicle includes: determining traffic flow density based on the number of vehicles entering the blind zone, and determining the spatial search range of each target vehicle based on the traffic flow density; determining traffic flow speed based on the images acquired by the first camera, and determining the temporal search range of each target vehicle based on the traffic flow speed; and controlling the first camera and the second camera to acquire images within the spatial and temporal search ranges.

[0044] Traffic flow density can be the number of vehicles in a lane or direction at a certain instant on a unit length of road segment. Spatial search range can be the spatial area where the target vehicle may appear, determined by traffic flow density. Temporal search range can be the time period during which the target vehicle may pass through a certain area, determined by traffic flow speed.

[0045] It is understood that this application embodiment determines traffic flow density by comprehensively analyzing the number of vehicles entering the blind zone, and sets the spatial search range for the target vehicle accordingly; simultaneously, it calculates traffic flow speed using images captured by the first camera, thereby dynamically adjusting the matching search range in space and time. Within a limited spatial and temporal range, it precisely controls the image acquisition of the first and second cameras, thereby improving acquisition efficiency and accuracy.

[0046] For example, in a four-lane scenario, when four or more vehicles are in the blind spot, the blind spot is considered to be in a congested traffic state. Drivers tend to stay in their lane, so the search for candidate targets can be limited to a pixel coordinate range within the width of a single lane. When fewer than four vehicles are in the blind spot, the blind spot is considered to be in a free-flowing traffic state. Drivers may change lanes, so the search for candidate targets can be limited to a pixel coordinate range within the width of one and a half lanes. Further filtering of the time range is needed. First, based on the vehicle's position within the forward-facing camera's field of view as it approaches the blind spot, the speed of each vehicle is roughly calculated using visual methods. Assuming vehicles move at approximately a constant speed within a blind spot no longer than 15 meters, the estimated passage time can be calculated from each vehicle's speed. Based on this estimated passage time, the search range in time can be dynamically adjusted, significantly improving the accuracy, robustness, and efficiency of vehicle ID matching.

[0047] In step S102, the frontal view image is input into the first feature extraction model, and the first feature extraction model outputs the frontal view feature vector of each target vehicle. The first feature extraction model extracts the initial frontal view feature vector of the frontal view image and generates an initial rear view feature vector based on the frontal view image. The initial frontal view feature vector and the initial rear view feature vector are merged to obtain the candidate feature vector.

[0048] The first feature extraction model can be a deep learning model specifically designed to extract features from a frontal view image. The initial frontal view feature vector can be an unprocessed feature vector directly extracted from the frontal view image by the first feature extraction model. The initial back view feature vector can be generated from the frontal view image, simulating or predicting the features of the back view. The target frontal view feature vector can be a feature vector obtained by feature fusion after the first feature extraction model performs feature fusion.

[0049] It is understood that in this embodiment of the application, the frontal view image is input into the first feature extraction model. By extracting and fusing feature vectors from both the front and rear directions, the comprehensiveness and representativeness of feature extraction are improved, which helps to improve the accuracy and stability of cross-camera tracking and adapt to complex traffic environments.

[0050] It should be noted that the Pix2PixHD algorithm is used to generate the corresponding initial back view image from the front view image, and the first feature extraction model is used to extract the feature vector of the generated back view image.

[0051] In this embodiment, the first feature extraction model includes a first convolutional neural network model, a second convolutional neural network model, a conditional generative adversarial network (GAN), and a fusion network. The frontal view image is input into both the first convolutional neural network model and the GAN. The first convolutional neural network model outputs an initial frontal view feature vector, and the GAN outputs an initial back view image corresponding to the frontal view image. The back view image is then input into the first convolutional neural network model, which outputs an initial back view feature vector. The initial frontal view feature vector and the initial back view feature vector are merged to obtain a candidate feature vector. The initial back view image is then input into the second convolutional neural network model, which outputs an initial back view feature vector. The target back view feature vector is copied and merged to obtain the target feature vector.

[0052] The first convolutional neural network model can be a special type of neural network, mainly used to process image data. The second convolutional neural network model can process the initial back view image generated by the conditional generative adversarial network and output the initial back view feature vector. The fusion network can be used to integrate information from different sources or different processing stages.

[0053] It is understood that the first feature extraction model in this embodiment of the application achieves deep processing of frontal view images by comprehensively utilizing a first convolutional neural network model, a second convolutional neural network model, a conditional generative adversarial network, and a fusion network. It not only extracts the initial frontal view feature vector from the frontal view image but also generates an initial rear view image through the conditional generative adversarial network and extracts the initial rear view feature vector from it. The fusion network merges these two feature vectors to obtain candidate feature vectors. Simultaneously, the generated rear view image is input into the second convolutional neural network to obtain an enhanced target feature vector, effectively combining image information from different perspectives and improving the accuracy and robustness of vehicle cross-view tracking.

[0054] Specifically, such as Figure 4As shown, within the field of view of Camera_A, the image of the target vehicle is obtained according to the size of the target detection box within a set area. Then, a target image of size 128*128 is obtained by unifying the size, and finally, it is expanded into a four-dimensional tensor of size (1, 3, 128, 128). The processed tensor is fed into a pre-trained convolutional neural network model for feature extraction. Simultaneously, the tensor is fed into a generator network to generate the rear angle image of the vehicle, and then the rear image is fed into the same convolutional neural network model to extract features. Finally, a 2048*1 dimensional original forward-looking feature vector a1 and a 2048*1 dimensional rear-looking feature vector a2 are obtained. The a1 and a2 feature vectors are then concatenated along with their first dimension to obtain a 2048*1 dimensional forward-looking feature vector A. Finally, the target's feature vector A and its corresponding ID are added to the candidate target information storage list.

[0055] Within the field of view of Camera_B, vehicle target images are acquired in a designated area near the bottom of the field of view. These images are then preprocessed to obtain a four-dimensional tensor (1, 3, 128, 128), which is fed into a pre-trained convolutional neural network model to obtain a 2048*1-dimensional feature vector B. The cosine similarity of the back-view feature vector B with multiple candidate feature vectors Ai (filtered by an adaptive spatiotemporal dynamic range search mechanism) from the candidate list is calculated. The pair of feature vectors Ai-B with the highest cosine similarity is selected, and the ID corresponding to Ai is assigned to the tracking target corresponding to the back-view feature vector B, thus completing the cross-camera tracking task. Similarly, if there are j vehicle targets in a specified area in a frame of the forward-looking data, there are i*j cosine distance calculation tasks.

[0056] Therefore, the system receives the target image and corresponding tracking ID acquired in the forward-looking tracking algorithm, preprocesses the target image to a standard size and format, and feeds it into the conditional generation network to obtain the vehicle feature image from the rear view. Then, the original front view image and the generated rear view image are simultaneously fed into a pre-trained convolutional neural network model to extract the feature vectors of the front and rear views, and perform weighted merging to obtain the final feature vector. Finally, this final feature vector and the corresponding ID are fed into the second module.

[0057] In this embodiment of the application, before inputting the frontal view image into the first feature extraction model, the method further includes: acquiring an image dataset of front and rear feature pairs of the vehicle, wherein the image dataset contains frontal view images and rear view images of the vehicle; and training a conditional generative adversarial network using the front and rear feature datasets.

[0058] It is understood that the embodiments of this application collect and organize image datasets of vehicle front and rear feature pairs, and use the dataset to train a conditional generative adversarial network, which effectively enriches vehicle feature information and improves the quality of generated images. Through in-depth training, the conditional generative adversarial network initially grasps the feature change rules and directional mapping ability of the vehicle under different perspectives, thereby demonstrating its adaptability in complex environments and effectively improving robustness.

[0059] Specifically, to enable the generation module in the algorithm to generate corresponding front and rear features of vehicles, a large-scale dataset of front and rear feature pairs of images was established. Data engineering steps, including data acquisition, data preprocessing, organization, and storage, were completed.

[0060] By referencing national traffic and road infrastructure standards and conducting extensive field experiments, and adhering to the principles of practicality, applicability, coverage performance, and cost balance, relevant parameters for data acquisition were determined, such as the position, height, and pitch angle of the acquisition camera, as well as the trigger position for acquiring vehicle image information.

[0061] The data acquisition location was set directly above the center of the road at a height of 7 meters and a pitch angle of 23 degrees. By fixing the acquisition parameters in advance, fixed-angle front and rear feature image pairs of vehicles can be obtained. This is done to improve the training convergence speed and inference accuracy of the generative network. To accurately and stably achieve the above experimental parameters, two drones were used to collect data in real traffic scenarios. The collected video data underwent time alignment, frame splitting, object detection and cropping, data pairing, numbering, quality inspection, and packaging to complete the construction of the vehicle front and rear feature generation dataset. This dataset covers the vast majority of mainstream vehicle models on Beijing's traffic roads to support the accuracy and generalization of the generative model in real-world application scenarios.

[0062] It should be noted that when acquiring the image dataset of front and rear feature pairs of the vehicle, the acquisition parameters of the first and second cameras were fixed. This ensured that the acquired frontal and rear view images maintained consistency in key parameters such as resolution, color, and brightness, thereby improving the quality and reliability of the dataset. By fixing the acquisition parameters, image deviations caused by differences in camera settings were reduced, enabling the trained conditional generative adversarial network to learn the vehicle's features from different viewpoints more accurately.

[0063] In step S103, the rear view image is input into the second feature extraction model, the second feature extraction model outputs the target rear view feature vector of each target vehicle, and the target rear view feature vector is copied and merged to obtain the target feature vector.

[0064] The second feature extraction model can be used to extract features from the rear view image and generate a target rear view feature vector.

[0065] It is understood that the embodiments of this application significantly enhance the completeness of vehicle feature extraction by inputting the rear view image into the second feature extraction model to extract the target rear view feature vector of the target vehicle. By merging the rear view feature vectors, the vehicle feature representation is further enriched, and the accuracy and efficiency of vehicle matching in cross-camera tracking are significantly improved, thereby greatly optimizing the overall tracking performance.

[0066] During vehicle movement, especially when the vehicle's direction changes or encounters obstructions, the rear-view feature vector provides valuable auxiliary information, effectively improving tracking robustness. Furthermore, the introduction of the rear-view feature vector not only enriches the feature representation of the target vehicle but also significantly improves the accuracy and efficiency of vehicle matching from different camera perspectives during cross-camera tracking, thereby greatly optimizing the overall performance of cross-camera tracking.

[0067] Specifically, this application embodiment designs a candidate target information storage list and its dynamic management mechanism. This mechanism is responsible for receiving the second feature vector and storing the second feature vectors, corresponding numbers, and pixel coordinates of multiple target vehicles in chronological order. The candidate target information storage list is used to store and record the tracking target information of vehicles that are about to leave the field of view and enter the blind spot in the second field of view. Each target is recorded using the data structure [ID, X, Y, Feature_Vector, Timestamp, Velocity] and written to a JSON file to support the re-identification task of the second shot. The dynamic management mechanism of the list is to maintain the update of the list. When the tracking target appearing in the second shot extracts the ID with the highest matching similarity from the list, or when the ID that failed to be extracted from the list and has been lingering for too long, the corresponding target information in the list will be automatically deleted, thereby reducing the error rate of re-identification matching.

[0068] In step S104, the identifier and candidate feature vector of each target vehicle are stored in the candidate list. The similarity between the target feature vector and the candidate feature vector is matched, and the identifier corresponding to the candidate feature vector with the highest similarity is selected and assigned to the target vehicle for cross-camera tracking.

[0069] The candidate list can be used to store the identifier of each target vehicle and its target frontal view feature vector. Cross-camera tracking is a technology that can continuously track the same target vehicle across multiple camera views.

[0070] It is understood that this application embodiment stores the identifier and candidate feature vector of each target vehicle in a candidate list, uses a similarity matching algorithm to compare the target feature vector with the candidate feature vector, and finally selects the vehicle identifier corresponding to the candidate feature vector with the highest similarity for cross-camera tracking. This achieves efficient and accurate cross-camera tracking, significantly enhancing the continuity and robustness of tracking, and enabling effective tracking even when the vehicle's direction changes or encounters occlusion. Furthermore, by directly assigning identifiers from the candidate list of tracked vehicles, the tracking process is simplified, and system efficiency is improved.

[0071] Specifically, the target vehicle image acquired in Camera_B using the "trigger-based acquisition method" is preprocessed into a standardized 128*128 size and format, and then fed into a pre-trained image feature extraction module to obtain the rear image feature vector. The upstream adaptive spatiotemporal dynamic search mechanism is used to filter out most of the candidate targets that are unsuitable in terms of time and space distance. Then, the feature vector B of the rear view is compared with the feature vectors Ai of the multiple filtered Camera_A to calculate cosine similarity. The vector pair with the highest similarity will be assigned the ID corresponding to feature vector Ai to the target to which vector B belongs, and a multi-target tracking task will be carried out in the rear view field of Camera_B to complete the cross-camera tracking task.

[0072] The vehicle cross-camera tracking method proposed in this application introduces an adaptive spatiotemporal dynamic search mechanism to intelligently adjust the search range based on traffic flow density. It utilizes a conditional generation algorithm module to generate vehicle appearance features from different angles, alleviating the feature matching challenge. Simultaneously, a dedicated dataset of vehicle front and rear features is constructed. Through a candidate target information storage list and its dynamic management mechanism, candidate vehicle information is maintained in real time, thereby improving the accuracy and real-time performance of vehicle re-identification. This alleviates the real-time issues, weak robustness, and low matching accuracy problems inherent in cross-camera tracking.

[0073] To facilitate a clearer understanding of the cross-mirror tracking method for vehicles according to the embodiments of this application by those skilled in the art, the following is combined with... Figure 4 A detailed explanation is provided below:

[0074] First, a section of the road near the blind spot within the field of view of Camera_A is designated as "Snapshot Zone A," displayed as area A. When a target vehicle enters Snapshot Zone A, the system will capture the target's ID, the pixel coordinates of the center point at the bottom of the tracking bounding box, and a 128*128 pixel target image. The target image is first sent to a feature extraction module to obtain a fixed-dimensional feature vector, which, along with the ID and pixel coordinates, is recorded in a structured data format in a JSON file. When the vehicle leaves the blind spot under the roadside sensing device, "Snapshot Zone B" is triggered, displayed as area B. Camera_B captures a target-level image of the vehicle. Through feature extraction, dynamic spatiotemporal filtering, and cosine similarity calculation, the ID with the highest similarity is assigned to the vehicle target that just appeared in Snapshot Zone B, and the target information corresponding to that ID is removed from the list. Additionally, IDs that remain in the list for a set time threshold will also be deleted.

[0075] In summary, this embodiment achieves cross-camera vehicle tracking without overlapping regions on a single-node roadside device, enhancing the perception range through cameras in both the front and rear directions. Feature similarity calculation is performed using the vehicle's front and rear angles, mitigating the feature spacing problem caused by differences in vehicle angles. A two-stage tracking algorithm is employed to process video data, and a conditional generation algorithm is introduced to generate vehicle rear features, effectively alleviating the challenge of vehicle re-identification. Simultaneously, an adaptive dynamic search mechanism improves the accuracy of target filtering and ensures the algorithm's real-time performance. Compared to existing technologies, this embodiment does not require large amounts of offline datasets and spatiotemporal prior work, making it more suitable for real-time roadside perception applications.

[0076] Next, the vehicle cross-mirror tracking device according to an embodiment of this application is described with reference to the accompanying drawings.

[0077] Figure 5 This is a block diagram of a vehicle cross-mirror tracking device according to an embodiment of this application.

[0078] like Figure 5 As shown, the vehicle cross-view tracking device 10 is applied to a roadside intelligent sensing device. The roadside intelligent sensing device is equipped with a first camera and a second camera. The first camera captures images from the front view of the vehicle, and the second camera captures images from the rear view of the vehicle. The vehicle cross-view tracking device 10 includes: an acquisition module 100, a first processing module 200, a second processing module 300, and a target matching module 400.

[0079] The system comprises the following components: an acquisition module 100 for acquiring frontal and rear view images of at least one target vehicle; a first processing module 200 for inputting the frontal view image into a first feature extraction model, which outputs a frontal view feature vector for each target vehicle, wherein the first feature extraction model extracts an initial frontal view feature vector from the frontal view image and generates an initial rear view feature vector based on the frontal view image, and merges the initial frontal view feature vector and the initial rear view feature vector to obtain a candidate feature vector; a second processing module 300 for inputting the rear view image into a second feature extraction model, which outputs a target rear view feature vector for each target vehicle, and copies and merges the target rear view feature vector to obtain a target feature vector; and a target matching module 400 for storing the identifier and candidate feature vector of each target vehicle in a candidate list, performing similarity matching between the target feature vector and the candidate feature vector, filtering the identifier corresponding to the candidate feature vector with the highest similarity between the target feature vector and assigning it to the target vehicle for cross-camera tracking.

[0080] It should be noted that the foregoing explanation of the vehicle cross-camera tracking method embodiment also applies to the vehicle cross-camera tracking device of this embodiment, and will not be repeated here.

[0081] The vehicle cross-camera tracking device proposed in this application introduces an adaptive spatiotemporal dynamic search mechanism to intelligently adjust the search range based on traffic flow density. It utilizes a conditional generation algorithm module to generate vehicle appearance features from different angles, alleviating the feature matching challenge. Simultaneously, it constructs a dedicated dataset of vehicle front and rear features. Through a candidate target information storage list and its dynamic management mechanism, it maintains candidate vehicle information in real time, thereby improving the accuracy and real-time performance of vehicle re-identification. This alleviates the problems of real-time performance, weak robustness, and low matching accuracy in cross-camera tracking.

[0082] Figure 6 This is a schematic diagram of the roadside intelligent sensing device provided in an embodiment of this application. The roadside intelligent sensing device may include:

[0083] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0084] When the processor 602 executes the program, it implements the vehicle cross-mirror tracking method provided in the above embodiments.

[0085] Furthermore, the roadside intelligent sensing equipment also includes:

[0086] Communication interface 603 is used for communication between memory 601 and processor 602.

[0087] The memory 601 is used to store computer programs that can run on the processor 602.

[0088] The memory 601 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0089] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0090] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0091] The processor 602 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0092] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described vehicle cross-mirror tracking method.

[0093] This application also provides a computer program product, which, when executed, is used to implement the vehicle cross-mirror tracking method as described in the above embodiments.

[0094] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0095] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0096] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0097] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0098] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0099] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for cross-view vehicle tracking, characterized in that, The method is applied to a roadside intelligent sensing device, which is equipped with a first camera and a second camera. The first camera captures a frontal view image of the vehicle, and the second camera captures a rearal view image of the vehicle. The method includes the following steps: Acquire at least one frontal view image and one rear view image of a target vehicle; The frontal view image is input into a first feature extraction model, and the first feature extraction model outputs a frontal view feature vector for each target vehicle. The first feature extraction model extracts an initial frontal view feature vector from the frontal view image and generates an initial rear view feature vector based on the frontal view image. The initial frontal view feature vector and the initial rear view feature vector are then merged to obtain a candidate feature vector. The rear view image is input into the second feature extraction model, the second feature extraction model outputs the target rear view feature vector of each target vehicle, and the target rear view feature vector is copied and merged to obtain the target feature vector; The identifier and candidate feature vector of each target vehicle are stored in a candidate list. The similarity between the target feature vector and the candidate feature vector is matched. The identifier corresponding to the candidate feature vector with the highest similarity is selected and assigned to the target vehicle for cross-camera tracking. The first feature extraction model includes a first convolutional neural network model, a second convolutional neural network model, and a conditional generative adversarial network, wherein, The frontal view image is input into the first convolutional neural network model and the conditional generative adversarial network respectively. The first convolutional neural network model outputs the initial frontal view feature vector, and the conditional generative adversarial network outputs the initial back view image corresponding to the frontal view image. The back view image is then input into the first convolutional neural network to output the initial back view feature vector. The initial frontal view feature vector and the initial back view feature vector are then merged to obtain the candidate feature vector. The initial back view image is input into the second convolutional neural network model, the second convolutional neural network model outputs the initial back view feature vector, the target back view feature vector is copied and merged to obtain the target feature vector; The acquisition of the frontal view image and the rear view image of at least one target vehicle includes: Set the respective acquisition areas of the first camera and the second camera; Acquire a first image captured by the first camera within its acquisition area, and a second image captured by the second camera within its acquisition area; The first acquired image is used to obtain a frontal view image of the at least one target vehicle, and the second acquired image is used to obtain a rear view image of the at least one target vehicle. There is a blind spot between the first camera and the second camera. Acquiring the frontal view image and the rear view image of at least one target vehicle includes: The traffic flow density is determined based on the number of vehicles entering the blind zone, and the spatial search range of each target vehicle is determined based on the traffic flow density. The traffic flow speed is determined based on the image captured by the first camera, and the time search range for each target vehicle is determined based on the traffic flow speed. The first camera and the second camera are controlled to acquire images within the spatial search range and the temporal search range.

2. The vehicle cross-mirror tracking method according to claim 1, characterized in that, Before inputting the frontal view image into the first feature extraction model, the method further includes: Obtain an image dataset of front and rear feature pairs of a vehicle, wherein the image dataset comprises a frontal view image and a rear view image of the vehicle; The conditional generative adversarial network is trained using the aforementioned before-and-after feature datasets.

3. The vehicle cross-mirror tracking method according to claim 2, characterized in that, When acquiring image datasets of front and rear feature pairs of a vehicle, the acquisition parameters of the first camera and the second camera are fixed.

4. A vehicle cross-mirror tracking device, characterized in that, The device is applied to a roadside intelligent sensing device, which is equipped with a first camera and a second camera. The first camera captures a frontal view image of the vehicle, and the second camera captures a rearal view image of the vehicle. It is used to implement the method described in any one of claims 1-3, wherein the device comprises: An acquisition module is used to acquire at least one frontal view image and a rear view image of a target vehicle; The first processing module is used to input the frontal view image into the first feature extraction model, and the first feature extraction model outputs the frontal view feature vector of each target vehicle. The first feature extraction model extracts the initial frontal view feature vector of the frontal view image and generates an initial rear view feature vector based on the frontal view image. The initial frontal view feature vector and the initial rear view feature vector are merged to obtain a candidate feature vector. The second processing module is used to input the rear view image into the second feature extraction model, the second feature extraction model outputs the target rear view feature vector of each target vehicle, and copies the target rear view feature vector and merges them to obtain the target feature vector; The target matching module is used to store the identifier and candidate feature vector of each target vehicle into a candidate list, perform similarity matching between the target feature vector and the candidate feature vector, filter the identifier corresponding to the candidate feature vector with the highest similarity and assign it to the target vehicle for cross-camera tracking.

5. A roadside intelligent sensing device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the vehicle cross-mirror tracking method according to any one of claims 1-3.

6. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the vehicle cross-mirror tracking method according to any one of claims 1-3.

7. A computer program product having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the vehicle cross-mirror tracking method according to any one of claims 1-3.

Images