Vision-based large-scene vehicle continuous tracking method and system

Abstract

The invention discloses a vision-based large-scene vehicle continuous tracking method and system. The method comprises the following steps: extracting multi-dimensional features of a target vehicle by using a feature extraction model; when the target vehicle is observed by a plurality of cameras in the monitoring camera network at the same time, generating cooperative state estimation of the target vehicle under a global coordinate system by integrating observation data of each camera and the network topology structure; on the basis of collaborative state estimation, predicting to obtain an ordered subsequent candidate camera set and respective corresponding prediction space-time windows; according to the sequence of the candidate cameras, re-identification matching is carried out based on the multi-dimensional features in the corresponding prediction space-time windows, and cross-camera relay tracking of the target vehicle is realized; and the world coordinate points of the target vehicle which is continuously and successfully tracked are spliced according to a time sequence to form a complete space-time trajectory, and the space-time trajectory and the high-precision GIS map of the wharf are displayed in an overlapped mode. According to the invention, the robustness and accuracy of vehicle tracking in a bulk cargo wharf scene can be improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent transportation technology, specifically to a vision-based method and system for continuous vehicle tracking in large scenes. Background Technology

[0002] Bulk cargo terminals are key hubs for cargo collection, distribution, and transportation, with complex internal road networks, high traffic volume, and diverse vehicle types. To improve terminal operation efficiency, optimize traffic scheduling, and strengthen safety supervision, it is essential to conduct continuous, uninterrupted tracking of vehicles traveling within the terminal throughout their entire operation.

[0003] Existing vision-based continuous vehicle tracking methods face the following main technical challenges when applied to large-scale scenarios such as bulk cargo terminals:

[0004] Traditional vehicle re-identification methods lack discriminative power and robustness. Existing methods mainly rely on the static appearance features of vehicles (such as color and model). However, vehicles of the same model in a dock often have highly similar appearances, and their dynamic features, such as covering status, cargo shape, and empty / full load status, frequently change due to loading and unloading operations. Traditional methods fail to effectively model these dynamic features that are strongly correlated with operations, resulting in weak feature representation capabilities and low accuracy in cross-camera matching.

[0005] Existing tracking architectures cannot effectively utilize networked information for collaborative perception and prediction. Current mainstream methods are essentially camera-to-camera local relays: when a vehicle disappears from one camera, the system typically searches the next adjacent camera based on a single appearance match. This architecture has the following drawbacks: when the target vehicle is simultaneously in the field of view of multiple cameras (such as at road intersections), it cannot fuse these multi-view observations to obtain a more accurate and stable global estimate of the vehicle's motion state; after the vehicle leaves the field of view, predictions of its subsequent trajectory are often based on local observations from a single camera, failing to incorporate the topology of the entire monitoring network (such as camera layout and road connectivity) for intelligent reasoning. This results in the system being unable to accurately predict an ordered sequence of subsequent cameras and its spatiotemporal window of appearance, leading to blind or overly broad searches, inefficiency, and a high risk of losing track of the vehicle.

[0006] In conclusion, existing technologies are insufficient for achieving efficient and robust continuous vehicle tracking in large-scale scenarios such as bulk cargo terminals, and improvements are needed. Summary of the Invention

[0007] The purpose of this invention is to provide a vision-based method, system, electronic device, readable storage medium, and computer program product for continuous vehicle tracking in large scenes, in order to solve the technical problems mentioned in the background art.

[0008] This invention provides a vision-based method for continuous vehicle tracking in large scenes, comprising the following steps:

[0009] Step S10: Use a feature extraction model to extract multi-dimensional features of the target vehicle from images captured by the surveillance camera network; the multi-dimensional features include the vehicle's basic appearance features and dynamic appearance features strongly related to dock operations.

[0010] Step S20: When the target vehicle is simultaneously observed by multiple cameras in the monitoring camera network, the cooperative state estimate of the target vehicle in the global coordinate system is generated by combining the observation data of each camera with the network topology.

[0011] Step S30: Based on the cooperative state estimation, an ordered set of subsequent candidate cameras and their corresponding prediction spatiotemporal windows are predicted; according to the order of each candidate camera, within its corresponding prediction spatiotemporal window, re-identification and matching are performed based on the multi-dimensional features to achieve relay tracking of the target vehicle across cameras.

[0012] Step S40: The world coordinates of the target vehicles that have been successfully tracked are stitched together in time sequence to form a complete spatiotemporal trajectory, and the spatiotemporal trajectory is overlaid and displayed on the high-precision GIS map of the dock.

[0013] The present invention also provides a vision-based large-scene continuous vehicle tracking system, the system comprising:

[0014] The feature extraction unit is used to: extract multi-dimensional features of the target vehicle from images captured by the surveillance camera network using a feature extraction model; the multi-dimensional features include the vehicle's basic appearance features and dynamic appearance features strongly related to the dock operations.

[0015] The cooperative state estimation unit is used to: when the target vehicle is observed simultaneously by multiple cameras in the monitoring camera network, integrate the observation data of each camera and the network topology to generate a cooperative state estimate of the target vehicle in the global coordinate system.

[0016] The tracking unit is used to: predict an ordered set of subsequent candidate cameras and their corresponding prediction spatiotemporal windows based on the cooperative state estimation; and perform re-identification matching based on the multi-dimensional features within their corresponding prediction spatiotemporal windows according to the order of each candidate camera, so as to realize relay tracking of the target vehicle across cameras.

[0017] The trajectory management unit is used to: stitch together the world coordinates of the target vehicles that have been successfully tracked in a time sequence to form a complete spatiotemporal trajectory, and overlay the spatiotemporal trajectory with a high-precision GIS map of the dock for display.

[0018] The present invention also provides an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in any of the preceding claims.

[0019] The present invention also provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in any of the preceding claims.

[0020] The present invention also provides a computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps of the method as described in any of the preceding claims.

[0021] This invention effectively improves vehicle identification under complex appearance changes by constructing a multi-dimensional re-identification system that integrates dynamic features of terminal operations. Furthermore, it utilizes multi-camera collaborative observation to generate a global state estimate, thereby predicting ordered candidate camera sequences and their spatiotemporal windows, avoiding the inefficiency caused by blind searches. This invention significantly overcomes the challenges of tracking interruptions caused by license plate recognition failure, similar appearances, and blind spots in monitoring. Ultimately, in large-scale scenarios at bulk cargo terminals, it generates continuous, complete vehicle movement trajectories that can be displayed in real-time on GIS maps, improving the robustness, accuracy, and management efficiency of vehicle tracking. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a vision-based continuous vehicle tracking method for large scenes disclosed in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the system architecture of the solution disclosed in the embodiments of the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the presentation of a target vehicle's trajectory in a GIS map according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a vision-based large-scene continuous vehicle tracking system disclosed in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please see Figure 1 This invention provides a vision-based method 100 for continuous vehicle tracking in large scenes, comprising the following steps:

[0028] Step S10: Use a feature extraction model to extract multi-dimensional features of the target vehicle from images captured by the surveillance camera network; the multi-dimensional features include the vehicle's basic appearance features and dynamic appearance features strongly related to dock operations.

[0029] In this step, the feature extraction model is built on a deep convolutional neural network, such as ResNet-50 as the backbone network, to map the input vehicle image into a high-dimensional feature vector. This model is trained using a dock vehicle dataset with multi-dimensional annotations to learn highly discriminative feature representations.

[0030] To address the technical challenges of variable vehicle appearance and high similarity among vehicles of the same model in bulk cargo terminal scenarios, this invention constructs a feature system that transcends traditional static appearance. The multi-dimensional features (i.e., improved ReID features) include basic vehicle appearance features and dynamic appearance features strongly correlated with terminal operations. The basic vehicle appearance features refer to the inherent and relatively stable attributes of the vehicle, such as its color, model, and brand. Dynamic appearance features strongly correlated with terminal operations include, but are not limited to:

[0031] Vehicle covering status: Identify the presence, color, and material of the tarpaulin;

[0032] Cargo shape profile: Extract the macroscopic shape features of the cargo on the flatbed truck, such as cuboids, cylinders, or irregular stacks;

[0033] Appearance color of goods: Identify the primary color of the goods or tarpaulin;

[0034] Empty and fully loaded conditions: a comprehensive judgment is made by analyzing the vehicle's suspension height and the relative deformation of the tires and axles.

[0035] By introducing the aforementioned dynamic features, the feature vector can comprehensively characterize the state changes of vehicles during dock operations, which can significantly improve the accuracy and robustness of subsequent re-identification and matching.

[0036] Regarding the surveillance camera network, taking a general cargo terminal as an example, the terminal includes three main transport roads, with 12 high-mast lights distributed along the roads. Each high-mast light is equipped with a 2-megapixel, 25fps high-definition camera, and the focal length of the camera lens is adjusted according to the height of the high-mast light and the coverage area (e.g., a 12m high-mast light is equipped with an 8mm focal length lens, covering an area with a radius of 50m). All cameras are connected to the back-end server through the terminal's internal Ethernet to ensure stable bandwidth for video stream data transmission (the bit rate of each camera is controlled at 4-8Mbps).

[0037] Step S20: When the target vehicle is simultaneously observed by multiple cameras in the monitoring camera network, the cooperative state estimate of the target vehicle in the global coordinate system is generated by combining the observation data of each camera with the network topology.

[0038] This step achieves a more accurate and stable global perception of the target vehicle's motion state through multi-camera collaborative observation.

[0039] Specifically, when the target vehicle is located in the overlapping or adjacent areas of the fields of view of multiple cameras, the system can simultaneously acquire image sequences of the vehicle from different perspectives, thus achieving the state of simultaneous observation by multiple cameras as described above.

[0040] The observation data from each camera originates from the vehicle image coordinate trajectory obtained by target detection and tracking algorithms within the single camera's field of view (such as YOLOv8 combined with the DeepSORT algorithm). Using pre-calibrated camera parameters, the image coordinates (u,v) are converted to world coordinates (X,Y) through perspective transformation. The network topology is defined jointly by the wharf's GIS map and the camera deployment plan, clarifying the spatial location of each camera, its field of view coverage, and the connectivity between roads.

[0041] Furthermore, by fusing world coordinate sequences from multiple cameras and employing data fusion algorithms (such as Kalman filtering or extended Kalman filtering), the optimal position, velocity, and heading angle of the target vehicle in the global coordinate system are estimated, i.e., cooperative state estimation. Understandably, this cooperative state estimation offers higher accuracy and better resistance to interference compared to observations from a single camera.

[0042] Step S30: Based on the cooperative state estimation, an ordered set of subsequent candidate cameras and their corresponding prediction spatiotemporal windows are predicted; according to the order of each candidate camera, within its corresponding prediction spatiotemporal window, re-identification and matching are performed based on the multi-dimensional features to achieve relay tracking of the target vehicle across cameras.

[0043] In this step, based on the cooperative state estimate obtained in step S20, and combined with the network topology (road layout and camera field of view), trajectory prediction is performed using a motion model (such as a constant speed model). Based on this, the camera nodes most likely to be passed by the vehicle in the future are inferred, forming an ordered set of subsequent candidate cameras (e.g., sorted from high to low probability of arrival). Simultaneously, a corresponding prediction spatiotemporal window is calculated for each candidate camera, i.e., the time interval and geographical area within which the vehicle is expected to appear in the camera's field of view.

[0044] During tracking, the highest-priority candidate camera is activated first, and newly appearing vehicles are searched within its prediction spatiotemporal window. For the searched candidate vehicles, the multi-dimensional features defined in step S10 are extracted, and similarity is calculated between them and the features of the target vehicle, such as calculating cosine similarity, using the following formula:

[0045]

[0046] If the similarity exceeds a preset threshold, then... If a match is found, it is determined to be the same vehicle, and the relay association is completed. If the match fails, the next priority candidate camera is automatically activated, and the above search and matching process is repeated until a successful association is achieved or the entire candidate set is traversed. Understandably, this method can significantly narrow the search range, thereby improving the efficiency and reliability of tracking.

[0047] Step S40: The world coordinates of the target vehicles that have been successfully tracked are stitched together in time sequence to form a complete spatiotemporal trajectory, and the spatiotemporal trajectory is overlaid and displayed on the high-precision GIS map of the dock.

[0048] In this step, the backend server sorts and stitches together the vehicle's world coordinates (X, Y) obtained after each successful relay tracking according to the timestamp, thus forming a complete spatiotemporal trajectory reflecting the vehicle's entire journey. Subsequently, this spatiotemporal trajectory is overlaid and merged with a pre-imported high-precision GIS map of the dock, and displayed in real-time on the monitoring terminal. The display effect can be seen in the attached image. Figure 3 , Figure 3 The image shows the spatiotemporal trajectory obtained when multiple cameras work together.

[0049] In addition, the solution of the present invention can also support high-frequency backtracking and querying of vehicle trajectories within any historical period, and can perform behavioral analysis based on trajectory data, such as route planning and abnormal stop point detection, to assist the terminal in traffic scheduling and safety supervision.

[0050] This invention effectively improves vehicle identification under complex appearance changes by constructing a multi-dimensional re-identification system that integrates dynamic features of terminal operations. Furthermore, it utilizes multi-camera collaborative observation to generate a global state estimate, thereby predicting ordered candidate camera sequences and their spatiotemporal windows, avoiding the inefficiency caused by blind searches. This invention significantly overcomes the challenges of tracking interruptions caused by license plate recognition failure, similar appearances, and blind spots in monitoring. Ultimately, in large-scale scenarios at bulk cargo terminals, it generates continuous, complete vehicle movement trajectories that can be displayed in real-time on GIS maps, improving the robustness, accuracy, and management efficiency of vehicle tracking.

[0051] As an example, the dynamic appearance features strongly related to terminal operations include at least one of vehicle covering status, cargo shape and outline, cargo appearance color, and empty / full status.

[0052] In this embodiment, the above four features are the appearance attributes that change most directly and frequently when vehicles are loading, unloading and transporting at the dock. They describe the operating status of the vehicle from different physical dimensions (material, geometry, color and load) and together constitute the unique identifier that distinguishes dock vehicles from urban road vehicles.

[0053] Specifically, during the re-identification and matching process, the four features mentioned above can form complementary judgment criteria. For example, when two vehicles are similar in color and model, the difference in their covering status or whether they are empty or fully loaded can serve as a key discrimination criterion; conversely, when the covering status changes due to loading and unloading operations, the stability of the cargo shape and appearance color can provide auxiliary identification information. This synergistic effect of multi-dimensional features ensures high accuracy and robustness of re-identification.

[0054] As an example, the feature extraction model is trained using a dock vehicle dataset with multi-dimensional annotations, and a triplet loss function is used for metric learning to bring the feature distance of the same vehicle closer together and push the feature distance of different vehicles further apart.

[0055] In this embodiment, the dataset is collected from the actual operation scenario of the bulk cargo terminal. For example, the construction process is as follows: collect vehicle video data of the terminal within one month, manually annotate the images of 500 different types of vehicles (trucks, flatbed trucks), and the annotation content includes: vehicle appearance features, covering status, cargo shape, cargo color, empty and full load status, generating a total of 100,000 annotated images.

[0056] The multi-dimensional annotations in the dataset clearly correspond to the aforementioned feature system, meaning they not only include basic vehicle appearance features (color, model), but more importantly, they include dynamic appearance features strongly correlated with dock operations, specifically:

[0057] Covering status (e.g., no covering, blue covering, green covering);

[0058] The shape of the goods (e.g., rectangular coal, cylindrical steel, irregular ore);

[0059] Goods color (e.g., black, silver-gray, brown);

[0060] Empty and full load status (e.g., empty, half load, full load).

[0061] Using the aforementioned dataset for training ensures that the feature extraction model can learn and encode dynamic operational features that are strongly correlated with the identity of dock vehicles from images, thereby enabling the feature extraction model to adapt to the complex environment of the dock and achieve high-precision re-identification.

[0062] Meanwhile, addressing the core challenge of highly similar appearances among vehicles of the same model in bulk cargo terminal scenarios, this invention employs a triplet loss function to perform metric learning on the feature extraction model. The specific mathematical form of the triplet loss function is as follows:

[0063]

[0064] in,( , , A triple is formed, representing the features of the anchor point, positive sample, and negative sample, respectively; α>0 is the margin parameter, used to control the minimum margin between intra-class and inter-class distances (preferably, α=0.3). This represents the L2 norm.

[0065] The optimization objective of the triplet loss function is to construct the optimal distribution structure in the feature space through metric learning. Specifically, this loss function achieves its technical effect through the following mechanisms:

[0066] Intra-class aggregation: By optimizing the feature distance between anchor samples and positive samples, the feature representations of the same vehicle under different operating states (such as changes in covering status, empty and full load conversion) maintain the minimum distance in the feature space, thereby enhancing the model's ability to recognize the consistency of vehicle identity.

[0067] Inter-class separation: By optimizing the feature distance between anchor samples and negative samples, the feature representations of different vehicles are forced to maintain sufficient distance in the feature space. This mechanism specifically addresses the technical challenge of vehicles of the same model having similar appearances in dock scenarios, improving the discriminative power of features by increasing the inter-class distance.

[0068] For example, a ReID feature extraction model was constructed based on the ResNet-50 network. The labeled dataset was divided into training and test sets in a 7:3 ratio. During training, a triplet loss function was used to optimize the model, with a learning rate of 0.001 and 100 training epochs. Based on experimental data, the above training ultimately enabled the feature extraction model to achieve a matching accuracy of 92% on the test set, which is 17 percentage points higher than the traditional ReID model. The experimental data verified the effectiveness and significant technological advancement of this training method.

[0069] This implementation method uses a dedicated dataset containing multi-dimensional dynamic features and combines it with a triplet loss function for model training. This enables the feature extraction model to achieve intra-class aggregation and inter-class separation in the feature space, thereby significantly improving the accuracy and robustness of vehicle re-identification and effectively solving the tracking interruption problem caused by similar vehicle appearances and changing states in dock scenarios.

[0070] As an example, by combining the observation data from each camera with the network topology, a cooperative state estimate of the target vehicle in the global coordinate system is generated, including:

[0071] Step S21: For each camera that observes the target vehicle, use its internal and external parameters to independently convert the vehicle image coordinate sequence obtained through target detection and tracking into a world coordinate sequence.

[0072] In this step, for each camera that observes the target vehicle, an image coordinate to world coordinate transformation is performed. Specifically, pre-calibrated camera intrinsic parameters (including focal length and principal point) and extrinsic parameters (including installation height, pitch angle, and yaw angle) are used to establish a mapping relationship between the image pixel coordinate system and the world coordinate system. After obtaining the vehicle's coordinate sequence in the image using a target detection and tracking algorithm (such as YOLOv8+DeepSORT), perspective transformation is used to independently convert the image coordinate sequence (u,v) obtained by the target detection and tracking algorithm (such as YOLOv8 combined with DeepSORT) into a world coordinate sequence (X,Y) based on the dock's global coordinate system.

[0073] Step S22: Based on the road constraints defined by the network topology, perform spatiotemporal alignment and validity verification on multiple world coordinate sequences from different cameras, and eliminate observation points with gross errors caused by perspective distortion or temporary occlusion.

[0074] In this step, since there may be slight differences in the frame rates acquired by each camera or transmission delays, the timestamps of all cameras are first unified to the same high-precision time base (such as a Network Time Protocol (NTP) server). Subsequently, the world coordinate sequences from different cameras are interpolated (e.g., linear interpolation) to generate coordinate observations at equally spaced time points within the same series, ensuring that all data are in synchronized time frames.

[0075] Simultaneously, based on the road constraints defined by the network topology, the observation locations from different cameras are validated. Specifically, road information (including road boundaries, lane lines, etc.) in the wharf GIS map is used to determine the validity of the observation locations. For example, if the observation location of a certain camera contradicts the road connectivity of several other camera observation locations (e.g., a location that is physically inaccessible), then that observation point is also considered invalid data and discarded.

[0076] This process can effectively eliminate observation errors caused by factors such as perspective distortion and temporary occlusion.

[0077] Step S23: The multiple world coordinate sequences that have passed verification are fused and input into the state estimation algorithm; the state estimation algorithm is based on the vehicle's kinematic model, simultaneously optimizes the target vehicle's position, velocity, and heading angle, and outputs the cooperative state estimate;

[0078] In this step, the verified world coordinate sequence {z_1,z_2,...,z_N} from N cameras is used as the observation value and input into a state estimation algorithm. This algorithm is preferably a Kalman filter or a variant thereof (such as the Extended Kalman Filter (EKF) or the Unscented Kalman Filter (UKF).

[0079] The system's state vector is defined as x=[X,Y,v_X,v_Y,θ] T Where (X,Y) represents the position in world coordinates, (v_X,v_Y) represents the velocity components in the X and Y directions, and θ is the heading angle. The prediction stage of the filter is based on a vehicle kinematics model (such as a uniform velocity model (CV) or a constant speed and velocity model (CTRV)) to predict the state.

[0080] During the filter update phase, the observation positions of multiple cameras are simultaneously used to correct the predicted values. The state estimation algorithm calculates the residuals and uncertainties (covariance) of each observation, and simultaneously optimizes all variables in the state vector using optimal estimation theory (such as minimum mean square error estimation). Finally, it outputs a more accurate and smoother co-state estimate x that incorporates all valid observation information. est This represents a significant improvement in both accuracy and robustness compared to observations from any single camera.

[0081] Step S24: Based on the covariance matrix of the cooperative state estimation, assess the overall confidence level of this cooperative observation.

[0082] In this step, the state estimation algorithm (such as a Kalman filter) outputs a cooperative state estimate x. est Simultaneously, it outputs the corresponding estimation error covariance matrix P, which is used to quantify the uncertainty of each component in the state estimation. Specifically, the overall confidence score C of the scalarization can be calculated based on the main diagonal elements of the covariance matrix P (i.e., the variance of each state variable) or the norm of the matrix. For example, C = 1 / trace(P), where trace(P) is the trace of the covariance matrix, representing the total error variance of the estimation. The higher the value of C, the higher the confidence of this cooperative state estimation.

[0083] The aforementioned confidence level can be used to dynamically adjust the strategy for subsequent steps. For example, when the confidence level C is high, more accurate trajectory prediction and camera sequence inference can be performed based on this high-confidence state estimate; when the confidence level C is low (such as due to a small number of observation cameras or high observation noise), a more conservative strategy can be adopted, such as shortening the prediction time range or expanding the search range of the candidate camera set, to balance the accuracy and robustness of tracking.

[0084] This implementation effectively eliminates gross error observation points through spatiotemporal alignment and road constraint verification, improving the quality of the data source. By fusing multi-observation data based on Kalman filters and kinematic models, the vehicle's position, velocity, and heading angle are simultaneously optimized, resulting in a more accurate and smoother state estimate than any single camera. Finally, through confidence evaluation based on the covariance matrix trace, key adaptive control is provided for downstream trajectory prediction and relay tracking, thereby significantly improving the accuracy, robustness, and overall intelligence level of continuous vehicle tracking at the system level.

[0085] As an example, based on the cooperative state estimation, an ordered set of subsequent candidate cameras and their corresponding prediction spatiotemporal windows are predicted, including:

[0086] Step S31: Based on the cooperative state estimation and its corresponding covariance matrix, the possible location distribution of the target vehicle at multiple future time points is generated through a probabilistic motion model to form a probabilistic trajectory cloud map; wherein, the probabilistic motion model considers the combined effects of vehicle dynamics constraints, road topology constraints and historical trajectory patterns;

[0087] In this step, a probabilistic motion model considering multiple constraints is constructed, which includes three core constraint modules:

[0088] 1) Vehicle dynamics constraint module: Based on the vehicle kinematic equations, a state transition matrix is ​​established, taking into account physical parameters such as maximum steering angle and acceleration limit, to ensure that the generated trajectory conforms to the actual vehicle motion characteristics;

[0089] 2) The road topology constraint module transforms the road boundaries, lane lines, and intersection connections in the monitoring network topology map into state space constraints, and eliminates inaccessible areas through inequality constraints.

[0090] 3) Historical trajectory pattern module: Based on the historical driving data of dock vehicles, a hidden Markov model is established to learn the typical path selection patterns of different types of vehicles under specific time periods and specific tasks.

[0091] Using position, velocity, and heading angle from the cooperative state estimation as initial states, and state uncertainty represented by the covariance matrix as the initial distribution, a probability position distribution with 0.1-second intervals is generated through forward propagation, for example, within the next 30 seconds. Specifically, the unscented transformation method can be used to capture the statistical characteristics of the state distribution after propagation through the nonlinear motion model by selecting sampling points, forming a probability trajectory cloud map represented by a Gaussian mixture model.

[0092] Step S32: Perform spatiotemporal correlation analysis on the probability trajectory cloud map and the camera field of view in the monitoring network topology map, calculate the probability of the target vehicle appearing in the field of view of each camera, and generate an ordered candidate camera set sorted by occurrence probability based on probability threshold screening and Bayesian inference update mechanism.

[0093] In this step, a precise geometric model of the camera's field of view is established. For each camera, based on its installation location, orientation angle, pitch angle, and focal length parameters, and combined with the 3D terrain data of the dock, a view frustum model with the camera's position as the vertex is constructed to accurately define the camera's effective monitoring range in 3D space.

[0094] Spatiotemporal correlation analysis was performed between the probability trajectory cloud map and the view frustum of each camera. By calculating the spatial intersection of the probability trajectory cloud map and the camera view frustum at each time slice, the probability value of the target vehicle appearing in the field of view of each camera was calculated using a numerical integration method. The calculation formula is as follows:

[0095]

[0096] Among them, V i Let p(x,y,z,t) represent the view frustum space of camera i, and let p(x,y,z,t) represent the probability density at time t at position (x,y,z).

[0097] Then, a preliminary screening is performed based on a preset probability threshold (preferably 0.05), retaining P. cami Cameras with a probability greater than 0.05 were selected as candidates. Finally, a Bayesian inference mechanism was employed, using road connectivity probability and historical camera switching patterns as prior distributions, and the current probability trajectory cloud map as the likelihood function. The posterior probability of each camera was updated using the Bayesian formula:

[0098]

[0099] Based on the updated posterior probability values, an ordered set of candidate cameras is generated, sorted from high to low probability.

[0100] Step S33: For each candidate camera, based on the spatial intersection of the probability trajectory cloud map and the camera's field of view, the time distribution of the target vehicle entering and leaving the field of view is statistically analyzed using the Monte Carlo sampling method. The prediction spatiotemporal window is determined in the form of a confidence interval, and the confidence index of the spatiotemporal window is calculated simultaneously.

[0101] In this step, for each candidate camera, the region in the probabilistic trajectory cloud map that spatially intersects with the camera's view frustum is extracted as the effective prediction region. Then, using the Monte Carlo sampling method, for example, 10,000 sample trajectories are randomly selected from the probabilistic trajectory cloud map, and the specific time points when each trajectory enters and leaves the camera's field of view are recorded.

[0102] Based on the sampled set of time points, construct the entry time t respectively. in and departure time t out The empirical distribution function. Selecting a 95% confidence level, determine the time interval of the prediction spatiotemporal window. , ],in This represents the 2.5th percentile of the entry time. This represents the 97.5th percentile of the departure time.

[0103] Simultaneously, the confidence index C of the spatiotemporal window is calculated. window :

[0104]

[0105] Where N valid N represents the number of valid trajectories through the camera's field of view. total This represents the total number of sampled trajectories. This confidence index comprehensively reflects the reliability and accuracy of the spatiotemporal window and is used to guide the adjustment of search strategies during subsequent tracking.

[0106] To address the technical challenges of inaccurate vehicle trajectory prediction and blind candidate camera search in large-scale scenarios, this paper constructs a probabilistic motion model that integrates vehicle dynamics, road topology, and historical trajectory patterns. This model generates a probabilistic trajectory cloud map representing the possible future location distribution, thus solving the problem of low reliability in traditional single trajectory prediction. By performing precise spatiotemporal correlation analysis between the probabilistic cloud map and the camera's view frustum, and employing a Bayesian inference mechanism to generate an ordered candidate camera set, a fundamental shift from blind search to probability-guided search is achieved. Finally, Monte Carlo sampling is used to determine a prediction spatiotemporal window with a confidence index, providing quantitatively reliable spatiotemporal constraints for subsequent tracking. Therefore, this implementation significantly improves the accuracy, robustness, and intelligence of cross-camera vehicle tracking, effectively solving the technical challenge of vehicle loss of tracking in complex scenarios.

[0107] In addition, when the target vehicle is not simultaneously observed by multiple cameras in the surveillance camera network, the following tracking and prediction methods can be adopted:

[0108] 1. Vehicle Motion State Estimation: The vehicle's image trajectory is acquired within the single camera's field of view using target detection and tracking algorithms (such as YOLOv8+DeepSORT). Combined with camera calibration parameters, the image coordinates (u,v) are converted to world coordinates (X,Y) through perspective transformation or direct linear transformation.

[0109] Based on the world coordinate sequence of consecutive frames {(X t ,Y t )}, calculate the vehicle's speed v and direction ϕ:

[0110]

[0111]

[0112] Where Δt is the frame time interval.

[0113] 2. Spatiotemporal behavior prediction: based on the current motion state (Position and Velocity) The future trajectory of the vehicle is predicted using a Kalman filter. A uniform velocity model is used for the state transition, and its state transition matrix F and prediction equations are as follows:

[0114]

[0115] Based on the predicted trajectory and the known camera network topology, determine the next camera field of view that the vehicle is most likely to enter and its expected arrival time window [t]. entry ,t exit [and the areas expected to appear.]

[0116] Relay matching verification: After the target vehicle disappears from the current camera's field of view, a search is conducted within the predicted time window and appearance area of ​​the next camera's field of view to find newly appearing vehicles. For each candidate vehicle found, spatial filtering is first performed (whether it is within the predicted area), and then the ReID feature similarity between the candidate vehicle and the target vehicle is calculated. When both the spatial constraint and the feature similarity constraint (sim≥τ) are satisfied, a relay association is completed, achieving continuous tracking across cameras.

[0117] To illustrate the implementation process of this invention, a coal-carrying flatcar at the dock was selected as the target vehicle for continuous tracking verification. The verification process involves positioning within a single camera and prediction and relay matching across cameras:

[0118] 1. Current vehicle location and status estimation within the camera:

[0119] Camera parameters:

[0120] Camera A is located at the high mast light at the entrance of the dock. Its parameters are as follows: installation height is h=30 m, depression angle is θ=30, physical focal length is fmm=4mm, camera pixel size is s=1.4μm / px, image resolution is 640×48, and image center coordinates are (u0,v0)=(320,240).

[0121] Focal length conversion (physical units to pixel units):

[0122]

[0123] Using the YOLOv8 algorithm, the center coordinates of the target vehicle in the image were detected as (u,v)=(420,300).

[0124] Based on the camera's perspective geometry model, the image coordinates (u,v) are transformed to the world coordinate system (with the ground projection point directly below the camera as the origin (0,0), the X-axis pointing east, and the Y-axis pointing north).

[0125]

[0126]

[0127] Substituting the parameters, the initial position of the vehicle in the world coordinate system is calculated:

[0128] , .

[0129] The negative sign indicates that the vehicle is located to the south of the origin (directly below the camera), which is consistent with the scenario logic of vehicles entering the dock from south to north.

[0130] 2. Vehicle motion state estimation

[0131] Select N=10 consecutive frames of images (camera frame rate ffps=25Hz, time interval Δt=0.04s) to obtain the vehicle's position sequence in the world coordinate system:

[0132]

[0133]

[0134] Calculate the vehicle's average speed and direction of travel:

[0135]

[0136]

[0137] Therefore, the velocity components in the X and Y directions are:

[0138] ,

[0139] 3. Prediction of the next camera and its location:

[0140] According to the GIS map of the wharf, the next camera in the 45-degree direction is camera B, and its coverage area is as follows:

[0141]

[0142] Entry time prediction: X(t) = 1.82 + 10.0t, Y(t) = -1.09 + 10.0t.

[0143] Find the minimum time t for the vehicle to enter the coverage area of ​​camera B. m :

[0144]

[0145] Taking the maximum value, we get t. m =4.82s

[0146] Kalman filter location prediction: Initial state vector Xk=[1.82,−1.09,10.0,10.0] T After a prediction period of t=4.82s (prediction step size Δtp=0.1s, total n=48 steps), the predicted position is:

[0147] , .

[0148] Considering process noise (σ) q =0.2), the uncertainty range of the predicted position is approximately ±0.8 m. Therefore, the predicted area where the vehicle appears in camera B is: [49.0,50.6]m×[46.3,47.9]m.

[0149] Relay matching verification: At t=4.82s, camera B detected a coal-carrying wagon within its prediction region [49.0,50.6]×[46.3,47.9].

[0150] Extract the ReID features f of the vehicle B The features f of the target vehicle in camera A A Perform matching and calculate cosine similarity: sim(f A ,f B =0.91 (≥τ=0.85).

[0151] Since both spatial location and appearance characteristics were verified, the system determined that it was the same vehicle and successfully completed the relay tracking from camera A to camera B.

[0152] 4. Multi-camera continuous relay and trajectory presentation

[0153] The vehicle continued to move, and based on the observation data of camera B, the system predicted and successfully associated it with the next camera C (coverage range [100,150]×[50,100]), with a ReID similarity of 0.89.

[0154] The backend server fuses the tracking results (time, world coordinates) from each camera in a time series:

[0155] Camera A (t=0 s, position (1.82,−1.09))

[0156] Camera B (t=4.82 s, position (49.84, 47.11))

[0157] Camera C (t=9.82 s, position (99.84, 97.11))

[0158] Finally, the target vehicle's continuous and smooth trajectory was generated and displayed in real time on the GIS map, with a travel direction of 45 (northeast) and an average speed of approximately 50.9 km / h.

[0159] Please see Figure 4 This invention also provides a vision-based large-scene continuous vehicle tracking system 200, the system comprising:

[0160] The feature extraction unit 201 is used to: extract multi-dimensional features of the target vehicle from images captured by the surveillance camera network using a feature extraction model; the multi-dimensional features include the vehicle's basic appearance features and dynamic appearance features strongly related to the dock operations.

[0161] The cooperative state estimation unit 202 is used to: when the target vehicle is observed simultaneously by multiple cameras in the monitoring camera network, integrate the observation data of each camera and the network topology to generate a cooperative state estimate of the target vehicle in the global coordinate system.

[0162] The tracking unit 203 is used to: predict an ordered set of subsequent candidate cameras and their corresponding prediction spatiotemporal windows based on the cooperative state estimation; and perform re-identification matching based on the multi-dimensional features within their corresponding prediction spatiotemporal windows according to the order of each candidate camera, so as to realize relay tracking of the target vehicle across cameras.

[0163] The trajectory management unit 204 is used to: stitch together the world coordinates of the target vehicle that has been successfully tracked in a time sequence to form a complete spatiotemporal trajectory, and overlay the spatiotemporal trajectory with a high-precision GIS map of the dock for display.

[0164] As an example, the dynamic appearance features strongly related to terminal operations include at least one of vehicle covering status, cargo shape and outline, cargo appearance color, and empty / full status.

[0165] As an example, the feature extraction model is trained using a dock vehicle dataset with multi-dimensional annotations, and a triplet loss function is used for metric learning to bring the feature distance of the same vehicle closer together and push the feature distance of different vehicles further apart.

[0166] As an example, the cooperative state estimation unit 202 is specifically used for:

[0167] For each camera that observes the target vehicle, the vehicle image coordinate sequence obtained through target detection and tracking is independently converted into a world coordinate sequence using its internal and external parameters.

[0168] Based on the road constraints defined by the network topology, spatiotemporal alignment and validity verification are performed on multiple world coordinate sequences from different cameras to eliminate observation points with gross errors caused by perspective distortion or temporary occlusion.

[0169] Multiple verified world coordinate sequences are fused and input into a state estimation algorithm; the state estimation algorithm, based on the vehicle's kinematic model, simultaneously optimizes the target vehicle's position, velocity, and heading angle, and outputs the cooperative state estimate.

[0170] Based on the covariance matrix of the cooperative state estimation, the overall confidence level of this cooperative observation is evaluated.

[0171] As an example, the tracking unit 203 is specifically used for:

[0172] Based on the cooperative state estimation and its corresponding covariance matrix, the possible location distribution of the target vehicle at multiple future time points is generated through a probabilistic motion model, forming a probabilistic trajectory cloud map; wherein, the probabilistic motion model considers the combined influence of vehicle dynamics constraints, road topology constraints and historical trajectory patterns;

[0173] The probability trajectory cloud map and the camera field of view in the monitoring network topology map are analyzed in a spatiotemporal correlation to calculate the probability of the target vehicle appearing in the field of view of each camera. Based on the probability threshold screening and Bayesian inference update mechanism, an ordered candidate camera set sorted by the probability of occurrence is generated.

[0174] For each candidate camera, based on the spatial intersection of the probability trajectory cloud map and the camera's field of view, the temporal distribution of the target vehicle entering and leaving the field of view is statistically analyzed using the Monte Carlo sampling method. The prediction spatiotemporal window is determined in the form of a confidence interval, and the confidence index of the spatiotemporal window is calculated simultaneously.

[0175] Additionally, please see Figure 2 The corresponding functions of the above-mentioned method 100 and system 200 of the present invention are mainly composed of Figure 2 The intelligent analysis host or the intelligent analysis host works in collaboration with the streaming media and control center to complete the task. The final spatiotemporal trajectory generated and overlaid on the high-precision GIS map of the dock is implemented on the client side, which will not be elaborated further.

[0176] The present invention also provides an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any of the foregoing embodiments.

[0177] The present invention also provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in any of the foregoing embodiments.

[0178] The present invention also provides a computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps of the method as described in any of the foregoing embodiments.

[0179] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A vision-based large scene vehicle continuous tracking method, characterized in that, The method comprises the following steps: Step S10, using a feature extraction model to extract multi-dimensional features of a target vehicle from images captured by a monitoring camera network; the multi-dimensional features include basic appearance features of the vehicle and dynamic appearance features strongly related to port operations; Step S20, when the target vehicle is observed by multiple cameras in the monitoring camera network at the same time, integrating observation data of each camera and network topology to generate a collaborative state estimation of the target vehicle in a global coordinate system; Step S30, based on the collaborative state estimation, predicting an ordered set of subsequent candidate cameras and respective corresponding predicted spatio-temporal windows; in the order of the candidate cameras, within the corresponding predicted spatio-temporal windows, performing re-identification matching based on the multi-dimensional features to realize relay tracking of the target vehicle across cameras; Step S40, splicing world coordinate points of the target vehicle successfully tracked in time sequence to form a complete spatio-temporal trajectory, and superimposing and displaying the spatio-temporal trajectory on a high-precision GIS map of the port.

2. The visual-based large scene vehicle continuous tracking method according to claim 1, wherein: The dynamic appearance features strongly related to port operations include at least one of vehicle covering condition, cargo shape profile, cargo apparent color, and empty / full load state.

3. The visual-based large scene vehicle continuous tracking method according to claim 1, wherein: The feature extraction model is trained using a port vehicle dataset containing multi-dimensional annotations, and a triplet loss function is used for metric learning to reduce the feature distance of the same vehicle and increase the feature distance of different vehicles.

4. The visual-based large-scale vehicle continuous tracking method according to claim 1, wherein: The collaborative state estimation of the target vehicle in the global coordinate system is generated by integrating the observation data of each camera and the network topology, comprising: Step S21, for each camera observing the target vehicle, using its internal and external parameters to independently convert the vehicle image coordinate sequence obtained through target detection and tracking into a world coordinate sequence; Step S22, based on the road constraints defined by the network topology, performing spatio-temporal alignment and validity checking on multiple world coordinate sequences from different cameras to eliminate gross error observation points caused by perspective distortion or temporary occlusion; Step S23, fusing the multiple world coordinate sequences that pass the checking and inputting them into a state estimation algorithm; the state estimation algorithm optimizes the position, speed and heading angle of the target vehicle based on the kinematic model of the vehicle, and outputs the collaborative state estimation; Step S24, based on the covariance matrix of the collaborative state estimation, evaluating the overall confidence of this collaborative observation.

5. The visual-based large-scale vehicle continuous tracking method according to claim 4, wherein: Based on the collaborative state estimation, an ordered set of subsequent candidate cameras and respective corresponding predicted spatio-temporal windows are predicted, comprising: Step S31, based on the collaborative state estimation and its corresponding covariance matrix, generating a possible position distribution of the target vehicle at multiple future time points through a probabilistic motion model to form a probability trajectory cloud map; wherein the probabilistic motion model considers the comprehensive effects of vehicle dynamics constraints, road topology constraints and historical trajectory patterns; Step S32, spatio-temporal correlation analysis is performed between the probability trajectory cloud map and the camera field of view range in the monitoring network topology map, the probability of the target vehicle appearing in each camera field of view is calculated, and an ordered candidate camera set sorted by appearance probability is generated based on a probability threshold screening and a Bayesian inference updating mechanism; Step S33, for each candidate camera, the spatial intersection of the probability trajectory cloud map and the camera field of view is used to statistically determine the time distribution of the target vehicle entering and leaving the field of view by a Monte Carlo sampling method, to determine a predicted spatio-temporal window in the form of a confidence interval, and to synchronously calculate the confidence index of the spatio-temporal window.

6. A vision-based large scene vehicle continuous tracking system, characterized by: The system comprises: The feature extraction unit is configured to extract multi-dimensional features of the target vehicle from images captured by the monitoring camera network using a feature extraction model, wherein the multi-dimensional features include basic appearance features of the vehicle and dynamic appearance features strongly related to port operations; The cooperative state estimation unit is configured to generate a cooperative state estimation of the target vehicle in a global coordinate system by integrating observation data of multiple cameras in the monitoring camera network and a network topology structure when the target vehicle is observed by the multiple cameras simultaneously; The tracking unit is configured to predict an ordered subsequent candidate camera set and respective predicted spatio-temporal windows based on the cooperative state estimation, and to perform re-identification matching based on the multi-dimensional features within the predicted spatio-temporal windows of the respective candidate cameras to achieve relay tracking of the target vehicle across cameras; The trajectory management unit is configured to splice world coordinate points of the target vehicle successfully tracked in a time sequence to form a complete spatio-temporal trajectory, and to superimpose and display the spatio-temporal trajectory on a high-precision GIS map of the port.

7. A vision-based large scene vehicle continuous tracking system according to claim 6, characterized in that: The tracking unit is specifically configured to: generate a probability trajectory cloud map by a probability motion model based on the cooperative state estimation and a corresponding covariance matrix, wherein the probability motion model considers the comprehensive influence of vehicle dynamics constraints, road topology constraints and historical trajectory patterns; perform spatio-temporal correlation analysis between the probability trajectory cloud map and camera field of view ranges in a monitoring network topology map, calculate the probability of the target vehicle appearing in each camera field of view, and generate an ordered candidate camera set sorted by appearance probability based on a probability threshold screening and a Bayesian inference updating mechanism; for each candidate camera, determine a predicted spatio-temporal window in the form of a confidence interval by a Monte Carlo sampling method based on the spatial intersection of the probability trajectory cloud map and the camera field of view, and synchronously calculate the confidence index of the spatio-temporal window.

8. An electronic device, comprising: A processor and a memory are included, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 5.

9. A readable storage medium, characterized by, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 5.

10. A computer program product, characterised in that, The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method according to any one of claims 1 to 5.

Images