Traffic perception and track splicing method and system based on trinocular camera

Abstract

The invention relates to the technical field of intelligent traffic, in particular to a traffic perception and track splicing method and system based on a trinocular camera, and the method comprises the steps: collecting traffic video streams covering different distance segments through short-distance, middle-distance and long-distance lenses, and extracting synchronous image frames; mapping the plurality of synchronous image frames to a world coordinate system through joint calibration; detecting a synchronous image frame of the video stream, obtaining an image area of the vehicle in the synchronous image frame, and obtaining a position coordinate corresponding to the image area; tracking and identifying the vehicle in the video stream based on the position coordinates, and generating a local trajectory fragment; on the basis of the local track fragments, predicting a space-time area where the vehicle enters an adjacent lens, performing matching according to vehicle appearance features, and associating different local track fragments to the same global vehicle identity; and splicing all the local track fragments according to the global vehicle identity identifier to form a continuous complete track. According to the invention, the coverage continuity and track integrity of the traffic sensing system in a longitudinal large range are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a traffic perception and trajectory stitching method and system based on a tri-camera. Background Technology

[0002] In the construction of intelligent transportation systems, continuous, accurate and wide-range trajectory perception of road vehicles is a fundamental and critical requirement. Whether it is to conduct traffic flow statistical analysis, automatic detection and full-process tracing of abnormal events, or to provide highly reliable global traffic situation information for vehicle-road cooperation, it all depends on continuous trajectory data that can seamlessly cover the target area and maintain the consistency of the target identity. With the development of intelligent transportation, traffic perception using roadside vision devices has become the mainstream method.

[0003] Existing technologies typically deploy two or more independent camera units or use integrated multi-lens hardware to cover a wider road area. However, the calibration error between cameras amplifies with distance, leading to deviations in the coordinate mapping of the same target by different cameras. This causes matching based on precise coordinates to fail. When a target moves from one camera's field of view to another, its appearance may change significantly due to drastic changes in angle and lighting, causing matching algorithms based on appearance features to fail. Moreover, existing methods usually only associate targets when they are already in the fields of view of two cameras simultaneously, i.e., there is an overlapping area. For targets that do not overlap or quickly pass through the edge of the field of view, there is a lack of effective prediction and active matching mechanisms, which can easily lead to trajectory loss.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the general background of this disclosure and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] In view of at least one of the above technical problems, the present invention provides a traffic perception and trajectory stitching method and system based on a tri-camera, which adopts a combination of close-range, medium-range and long-range three-lens collaborative processing and cross-camera trajectory prediction and feature matching to achieve consistency and stability in continuous tracking of traffic vehicles from far to near.

[0006] This invention provides a traffic perception and trajectory stitching method based on a tri-camera, comprising the following steps: Traffic video streams covering different distance segments (near, medium, and far) are captured using close-up, medium-range, and long-range lenses, and synchronized image frames are extracted. The multiple synchronized image frames are uniformly mapped to the same world coordinate system through joint calibration; In the world coordinate system, the synchronized image frame of each video stream is detected, the image region of the vehicle in the synchronized image frame is obtained, and the position coordinates of the image region in the world coordinate system are obtained. Based on the location coordinates, the vehicle is tracked and identified in the video stream of each shot, and a local trajectory segment of the same vehicle in each shot is generated. Based on the local trajectory segments, the spatiotemporal region where the vehicle enters the adjacent camera is predicted. Within the spatiotemporal region, the vehicle's appearance features are matched to associate the local trajectory segments of different cameras with the same global vehicle identity identifier. All the local trajectory segments are spliced ​​together based on the global vehicle identification to form a continuous and complete trajectory.

[0007] Furthermore, the process of uniformly mapping multiple synchronized image frames to the same world coordinate system through joint calibration includes: Based on the synchronized image frames from the close-range, medium-range, and long-range lenses, the internal parameters of each lens are calculated respectively. Calculate the first relative external parameter between the close-up lens and the medium-range lens, and the second relative external parameter between the medium-range lens and the telephoto lens based on the internal parameters. The world coordinate system is established with the imaging center of the mid-range lens as the origin, and the mapping relationship between the synchronized image frames of the close-range lens and the long-range lens and the world coordinate system is obtained through the first relative external parameter and the second relative external parameter. Based on the mapping relationship, the synchronized image frames of the close-up lens and the long-distance lens are uniformly mapped to the world coordinate system.

[0008] Furthermore, it also includes optimizing the mapping relationship: Based on the synchronized image frames from the close-up, medium-range, and long-range lenses, inherent static linear features in the actual traffic scene are extracted; Based on the currently valid mapping relationship, the static linear features corresponding to the same physical entity in the close-up, medium-range, and long-range shots are respectively mapped to the world coordinate system; Calculate the spatial position error between different shots mapped to the same static linear feature in the world coordinate system; With the goal of minimizing the spatial position error, the first relative external parameter and the second relative external parameter are iteratively adjusted to obtain the optimized mapping relationship; The optimized mapping relationship is used to uniformly map the synchronized image frames of the close-up and long-distance shots to the world coordinate system.

[0009] Furthermore, the process of generating local trajectory segments of the same vehicle in each shot includes: Create and maintain a trajectory list to record the vehicle trajectories in the current video stream. Each trajectory in the trajectory list includes motion state information and coordinate sequence, and each trajectory is marked as a continuous trajectory or a terminated trajectory. For each continuous trajectory in the trajectory list, based on the motion state information and coordinate sequence of the continuous trajectory, the expected position of the continuous trajectory in the subsequent synchronized image frames is predicted; Based on the predicted expected locations in the trajectory list and the vehicle's location coordinates, an association matching is performed; For the vehicles that are successfully matched, the motion state information and coordinate sequence corresponding to the continuous trajectory are updated using the location coordinates, and recorded in the trajectory list; When the vehicle corresponding to the continuous trajectory leaves the video stream of the current camera, the continuous trajectory is changed to the terminated trajectory; Remove the termination trajectory from the trajectory list and output the accumulated coordinate sequence of the termination trajectory as a local trajectory segment.

[0010] Furthermore, associating the local trajectory segments from different shots with the same global vehicle identity identifier includes the following process: When it is determined that the vehicle is about to leave the current camera's field of view based on the local trajectory segment, the expected time and expected spatial region of the vehicle entering the field of view of the adjacent camera are predicted based on the local trajectory segment and the vehicle kinematics model. The expected time and the spatial region constitute the spatiotemporal region. Within the spatial region, multiple candidate vehicles detected by adjacent lenses near the expected time are acquired; A probability hypothesis density filtering algorithm is used to calculate the proximity probability between the actual location of each candidate vehicle and the spatial region, as well as the similarity probability between the appearance features of each candidate vehicle and the appearance features of the vehicle that has driven out. Based on the proximity probability and the similarity probability, the departing vehicle is paired with the candidate vehicle, and the candidate vehicle is assigned the same global vehicle identity identifier as the departing vehicle.

[0011] Furthermore, the appearance features include depth features that are robust to changes in vehicle viewing angle and lighting conditions, and these depth features are extracted from the image region using a pre-trained deep neural network model.

[0012] Furthermore, it also includes detecting traffic events based on continuous complete trajectories, and tracking the lifecycle of vehicles based on the traffic events and the corresponding continuous complete trajectories.

[0013] Furthermore, the process of tracking a vehicle's lifecycle includes: Continuously monitor the continuous and complete trajectory under each global vehicle identity identifier and acquire trajectory data; When the trajectory data meets the preset event determination rules, a traffic event of the corresponding type is triggered, and a globally unique event identifier is assigned to the traffic event; The globally unique event identifier is associated and bound with the global vehicle identity identifier of the vehicle that triggered the traffic event; During the duration of the traffic incident, the status information of the traffic incident is continuously updated based on the vehicle's continuous and complete trajectory. The status information includes the latest location and duration of the traffic incident. When the conditions for the establishment of the traffic event are no longer met based on the continuous and complete trajectory, the traffic event is determined to have ended, and an event record containing the identifier, type, total duration, and associated vehicle information of the traffic event is generated.

[0014] This invention also provides a traffic perception and trajectory stitching system based on a trinocular camera, comprising: The extraction and mapping module acquires traffic video streams covering different distance segments (near, medium, and far) using close-range, medium-range, and long-range lenses and extracts synchronized image frames; through joint calibration, it maps multiple synchronized image frames to the same world coordinate system. The detection and acquisition module detects the synchronized image frame of each video stream in the world coordinate system, acquires the image region of the vehicle in the synchronized image frame, and acquires the position coordinates of the image region in the world coordinate system. The local generation module tracks and identifies vehicles in the video stream of each shot based on the position coordinates, and generates local trajectory segments of the same vehicle in each shot. The identification and association module predicts the spatiotemporal region where the vehicle enters an adjacent shot based on the local trajectory segment, and matches the vehicle's appearance features within the spatiotemporal region to associate the local trajectory segments of different shots with the same global vehicle identity identifier. The stitching module stitches together all the local trajectory segments based on the global vehicle identification identifier to form a continuous and complete trajectory.

[0015] Furthermore, the extraction mapping module also includes: shell; An optical lens assembly, comprising near-range, intermediate-range, and far-range lenses rigidly fixed to the housing in a longitudinal arrangement; An edge computing unit is connected to the optical lens assembly.

[0016] The technical solution of this invention can achieve the following technical effects: By using three lenses—close-range, medium-range, and long-range—to acquire and process data synchronously, and combining joint calibration, cross-camera spatiotemporal prediction, and probability matching mechanisms, a technological leap from local perception to global continuous tracking has been achieved. This effectively overcomes common challenges in multi-camera systems, such as blind spots, calibration error accumulation, unstable appearance features, and matching uncertainties. It enables continuous high-precision trajectory tracking of vehicle targets from far to near, while always maintaining the uniqueness and consistency of global identity identification.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the traffic perception and trajectory stitching method based on a tri-camera in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the process of mapping multiple synchronized image frames to the same world coordinate system through joint calibration in an embodiment of the present invention; Figure 3 This is a schematic diagram of the process for optimizing the mapping relationship in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process of generating local trajectory segments of the same vehicle in each shot in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the process of associating local trajectory segments from different cameras with the same global vehicle identity identifier in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the process of vehicle lifecycle tracking in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] This invention provides a method such as Figures 1 to 6 The traffic perception and trajectory stitching method based on a tri-camera shown includes the following steps: Traffic video streams covering different distance segments (near, medium, and far) are captured using close-up, medium-range, and long-range lenses, and synchronized image frames are extracted. Multiple synchronized image frames are uniformly mapped to the same world coordinate system through joint calibration; In the world coordinate system, detect the synchronized image frames of each video stream, obtain the image region of the vehicle in the synchronized image frame, and obtain the position coordinates of the image region in the world coordinate system; Based on the location coordinates, the vehicle is tracked and identified in the video stream of each shot, and a local trajectory segment of the same vehicle in each shot is generated. Based on local trajectory segments, predict the spatiotemporal region where the vehicle enters adjacent shots, and match the vehicle's appearance features within the spatiotemporal region to associate local trajectory segments from different shots with the same global vehicle identity identifier. All local trajectory segments are spliced ​​together based on the global vehicle identification to form a continuous and complete trajectory.

[0023] After the system starts up, the built-in synchronization trigger circuit sends a unified hardware synchronization signal to the image sensors of the three lenses (near-range, medium-range, and far-range) to ensure that the three video streams maintain microsecond-level time alignment at the exposure time. The three lenses, with their specific focal lengths and field of view, synchronously acquire traffic scene videos covering the near section of the road (0 to 100 meters), the medium section (75 to 250 meters), and the far section (200 to 500 meters). The system receives these three video streams in real time and extracts the time-stamped image frame sequence according to the same time series, effectively eliminating the problems of trajectory matching misalignment and increased computational complexity caused by time asynchrony.

[0024] To enable cross-lens data processing, a unified geometric benchmark needs to be established. The system performs joint calibration, mapping the pixel coordinates of the images acquired by the three lenses to the same world coordinate system, which can be a two-dimensional ground coordinate system with the origin of the point on the ground perpendicularly projected from the device mounting point.

[0025] For each set of synchronized image frames, the data is input to the target detection and localization unit. Real-time inference is performed on each frame, and the detection bounding boxes and their confidence scores for all vehicles are output. Subsequently, the localization subroutine processes each valid detection box, using the vehicle-ground contact point as a representative point. Using coordinate transformation in the corresponding world coordinate system, the pixel coordinates are transformed to the world coordinate system to obtain the vehicle's current two-dimensional ground coordinates, i.e., its position coordinates. At the same time, the vehicle image region is cropped and saved for subsequent feature extraction.

[0026] Each shot runs an independent single-target tracker to perform inter-frame correlation on the detected vehicles, forming a short-term trajectory within the field of view of that shot. This can be achieved using the SORT framework based on Kalman filtering and Hungarian matching algorithms. The sequence of timestamps and world coordinates generated by each tracker from initialization to termination is output as a local trajectory segment, which is also associated with the most representative appearance features of the corresponding vehicle during its lifecycle.

[0027] When a local trajectory segment in a camera is detected to be about to end and a vehicle is leaving the current camera's field of view, and the vehicle's movement trend is used to determine that it will enter the monitoring range of an adjacent camera, cross-camera association is initiated. Based on the motion state of the soon-to-end local trajectory segment, the system infers the possible spatiotemporal region (approximate time period and geographical range in the world coordinate system) where the vehicle may appear when it enters the field of view of the adjacent camera at a future time. Simultaneously, the adjacent camera will also generate local trajectory segments of vehicles within its field of view during the corresponding time period. Within the spatiotemporal region, the system calculates and matches the trajectory of the vehicle about to leave the camera with the newly appearing or continuing trajectory of the vehicle in the adjacent camera, based on the vehicle's appearance characteristics. If a match is found, the two local trajectory segments belonging to different cameras are determined to belong to the same vehicle, and they are assigned or associated with the same global vehicle identity identifier. If no match is found, a new global vehicle identity identifier is assigned to the newly appearing trajectory.

[0028] Based on the global vehicle identification identifier assigned to each local trajectory segment, all segments are grouped. For all local trajectory segments with the same global identifier, they are arranged in chronological order, and their world coordinate sequences are joined end to end for integration and smoothing. Finally, a continuous and complete trajectory covering the vehicle's entire driving process under multiple near, medium, and far-field cameras is generated. This trajectory serves as a unified carrier of time, global identification identifier, continuous world coordinate position, and other motion parameters, enabling long-distance, blind-spot-free continuous perception of traffic vehicles using tri-lens equipment. This overcomes the problems of trajectory breakage and identification confusion caused by traditional single-point or simple multi-point perception.

[0029] In some embodiments of the present invention, such as Figure 2As shown, the process of mapping multiple synchronized image frames to the same world coordinate system through joint calibration includes: Based on synchronized image frames from close-up, medium-range, and long-range lenses, calculate the internal parameters of each lens separately; Calculate the first relative external parameter between the close-up lens and the medium-range lens, and the second relative external parameter between the medium-range lens and the telephoto lens based on the internal parameters; A world coordinate system is established with the imaging center of the medium-range lens as the origin, and the mapping relationship between the synchronized image frames of the close-up lens and the long-range lens and the world coordinate system is obtained through the first relative external parameter and the second relative external parameter. Based on the mapping relationship, the synchronized image frames of close-up and long-distance shots are uniformly mapped to the world coordinate system.

[0030] First, standard camera intrinsic parameter calibration was performed on three lenses: close-range, medium-range, and long-range. Using a high-precision checkerboard calibration board, several sets of images of the calibration board were taken in various poses within the field of view of each lens. For each lens, using the multiple images, a classic camera calibration algorithm was employed to solve for the lens's intrinsic parameters, mainly including focal length and principal point coordinates, as well as coefficients characterizing lens distortion. This corrected the imaging geometric deformation of each lens, laying the foundation for subsequent accurate stereo vision calculations.

[0031] Treating the close-up and mid-range lenses as a binocular system, a checkerboard calibration board is simultaneously placed within the common overlapping field of view of both lenses, and multiple sets of synchronized images are acquired. Based on the calibrated internal parameters of the two lenses, the epipolar geometry principle in stereo vision is used to solve for the essential matrix or homography matrix through feature point matching and matrix calculation, such as checkerboard corner points. The rotation matrix and translation vector of the close-up lens relative to the mid-range lens are then calculated, and the transformation relationship between the coordinate system of the close-up lens and the coordinate system of the mid-range lens is defined, which is the first relative external parameter.

[0032] Similarly, the mid-range and telephoto lenses are considered as another binocular system. The calibration board is placed within the overlapping mid-to-far field of view shared by both lenses. Multiple sets of synchronized images are acquired, and the same method is used to calculate the rotation matrix and translation vector of the telephoto lens relative to the mid-range lens as the second relative external parameter.

[0033] The origin of the world coordinate system is chosen as the projection point of the imaging center of the medium-range lens onto the ground. A right-handed coordinate system is established with the horizontal plane as the reference, which serves as the world coordinate system. For the medium-range lens itself, the mapping from the pixel coordinate system to the world coordinate system can be directly defined by the internal parameters and the preset origin of the world coordinate system. For the close-range lens, the pixel coordinate system is first transformed to the medium-range lens coordinate system through the internal parameters, and then transformed to the world coordinate system. This transformation process is defined as the mapping relationship of the close-range lens. For the telephoto lens, its pixel coordinate system is transformed to the medium-range lens coordinate system through the internal parameters, and then transformed to the world coordinate system. This transformation process is defined as the mapping relationship of the telephoto lens.

[0034] In the subsequent real-time sensing process, for any synchronized image frame acquired by any lens, the system calls the pre-stored complete mapping parameters corresponding to that lens, including internal parameters, the first or second external parameters relative to the mid-range lens, and the mapping relationship to the world coordinate system. Through chained calculations and with the mid-range lens as the spatial reference, a unique world coordinate system is established, ensuring that the observation data from three lenses with different perspectives can be transformed into the same coordinate space with a real physical scale, providing a reliable common spatial reference for subsequent trajectory prediction and cross-lens matching.

[0035] In some embodiments of the present invention, such as Figure 3 As shown, it also includes optimizing the mapping relationship: Extract inherent static linear features in actual traffic scenes from synchronized image frames of close-up, medium-range, and long-range cameras; Based on the currently valid mapping relationship, the static linear features corresponding to the same physical entity in close-up, medium-range, and long-range shots are respectively mapped to the world coordinate system; Calculate the spatial position error between different shots mapped to the same static linear feature in the world coordinate system; With the goal of minimizing spatial location error, the first and second relative external parameters are iteratively adjusted to obtain the optimized mapping relationship. The optimized mapping relationship is used to uniformly map the synchronized image frames of close-up and long-distance shots to the world coordinate system.

[0036] While running and processing real-time traffic video streams, static linear features that are robust to changes in viewpoint and lighting and whose positions are stable in the world coordinate system over a long period of time are detected and extracted in parallel from synchronized image frames of three cameras: near-range, medium-range, and far-range. These features can include lane markings, such as solid lane edge lines, dashed lane dividers, and guide lines; or road edges, such as clear boundary lines formed by curbs and guardrail bases; or other fixed linear structures, such as bridge expansion joints and ground markings in specific areas. These extracted linear features are tracked and filtered, and only static features that appear consistently and stably across multiple frames and have consistent geometric shapes are retained for optimization calculations.

[0037] For the mapping relationship currently in use during the calibration period, the pixel coordinates of the static linear features extracted from the three lenses will be independently transformed into a unified world coordinate system according to the mapping relationship corresponding to each lens. For example, a section of dashed lane line in the mid-range lens image, a section of curb line in the close-up lens image, and a section of ground marking line in the long-range lens image will all be mapped into a two-dimensional plane in the world coordinate system.

[0038] Since these static linear features originate from real, fixed physical entities, theoretically, linear features corresponding to the same physical entity observed from different lenses and mapped to the world coordinate system should perfectly coincide. However, due to calibration errors or parameter drift, they will exhibit positional deviations after actual mapping. The feature association module identifies and associates features from different lenses that correspond to the same physical entity, for example, by judging based on the geographical proximity and directional consistency of the features. For each set of features associated with the same entity, the spatial positional error between their fitted curves in the world coordinate system is calculated, such as the average Euclidean distance between corresponding points or the distance between two fitted lines.

[0039] The weighted sum of the spatial position errors of all the associated feature pairs is calculated and used as the objective function. A nonlinear optimization algorithm is used to iteratively adjust the two sets of relative external parameters with the criterion of minimizing the objective function. During the optimization process, the coordinate system of the mid-range lens as the reference is kept fixed, and the internal parameters of the three lenses are also considered to be unchanged. The goal is to find the first and second sets of relative external parameters that best align the same static features from different lenses in the world coordinate system.

[0040] When the optimization iteration converges and the objective function value is lower than the preset threshold, a set of optimized first and second relative external parameters is obtained. Using this set of new parameters, the original mapping relationship model is updated, an optimized mapping relationship is generated, and the data quality and decision reliability of the entire intelligent transportation system are improved.

[0041] In some embodiments of the present invention, such as Figure 4As shown, the process of generating local trajectory segments of the same vehicle in each shot includes: Create and maintain a trajectory list to record vehicle trajectories in the current video stream. Each trajectory in the trajectory list includes motion state information and coordinate sequence, and each trajectory is marked as a continuous trajectory or a terminated trajectory. For each continuous trajectory in the trajectory list, predict the expected position of the continuous trajectory in subsequent synchronized image frames based on the motion state information and coordinate sequence of the continuous trajectory. Based on the predicted expected locations in the trajectory list and the vehicle's location coordinates, a correlation matching is performed; For vehicles that are successfully matched, the motion state information and coordinate sequence of the corresponding continuous trajectory are updated using the location coordinates and recorded in the trajectory list; When the vehicle corresponding to the continuous trajectory leaves the current camera's video stream, the continuous trajectory will be changed to a terminated trajectory; Remove the terminating trajectory from the trajectory list and output the accumulated coordinate sequence of the terminating trajectory as a local trajectory fragment.

[0042] Each shot independently initializes and maintains a trajectory list, which is a real-time collection of all vehicle trajectories currently being tracked within the view of that shot. Each element in the trajectory list is called a trajectory object, and each trajectory object contains at least a trajectory ID, a unique temporary identifier within the current shot; motion state information, data used to describe and predict the vehicle's motion state, which can be a Kalman filter state vector containing the vehicle's position, velocity, and corresponding state covariance matrix in the world coordinate system, used to represent the estimation uncertainty; coordinate sequence, a list arranged in chronological order, recording all world coordinate points traversed by the trajectory from its creation to the current moment; and a trajectory status flag, a binary flag used to indicate whether the trajectory is currently a continuous trajectory, indicating that the target vehicle is still within the view and being continuously tracked, or a terminated trajectory, indicating that the target vehicle has left the current view and tracking has ended.

[0043] For each state in the trajectory list marked as a continuous trajectory, when a new set of synchronized image frames arrives, based on the latest motion state information of the continuous trajectory, the possible position of the vehicle in the continuous trajectory in the next frame is predicted. The predicted expected position is usually represented as a two-dimensional world coordinate point and an elliptical or rectangular region around the point. The region represents the range in which the vehicle may appear.

[0044] Once vehicle detection for the current synchronized image frame is complete and a list of world coordinates for all vehicles is obtained, the association matching phase begins. This phase determines whether each detection result in the current synchronized image frame should be matched with an existing continuous trajectory in the trajectory list or represent a newly appearing vehicle.

[0045] For each successfully matched persistent trajectory, the Kalman filter of the corresponding trajectory is updated using the successfully matched detection coordinates as new observations, resulting in a corrected and more accurate latest motion state estimate. The current timestamp and the updated world coordinates are appended as new data points to the end of the coordinate sequence of the persistent trajectory. For detections in the current frame that fail to match any existing persistent trajectories, they are initialized as new trajectory objects, assigned new trajectory IDs, their motion state and coordinate sequence are initialized with the detection coordinates, they are marked as persistent trajectories, and added to the trajectory list.

[0046] For each continuous trajectory in the trajectory list, if no detection result is successfully matched in the association matching step within several consecutive frames, the system determines that the vehicle has left the current camera's field of view. At this time, the status marker of the continuous trajectory is changed to terminated trajectory.

[0047] Once a trajectory is marked as terminated, it is removed from the list of currently active trajectories. At the same time, the complete coordinate sequence accumulated by the trajectory object, together with the vehicle appearance features extracted or fused from the corresponding frame of the sequence, is packaged and output as a local trajectory segment. This enables continuous, real-time tracking of the vehicle within each shot and continuously generates completed local trajectory segments.

[0048] In some embodiments of the present invention, such as Figure 5 As shown, associating local trajectory segments from different shots with the same global vehicle identity identifier involves the following process: When it is determined that a vehicle is about to leave the current camera's field of view based on a local trajectory segment, the expected time and expected spatial region when the vehicle enters the field of view of an adjacent camera are predicted based on the local trajectory segment and the vehicle's kinematic model. The expected time and spatial region constitute the spatiotemporal region. Within a spatial region, acquire multiple candidate vehicles detected by adjacent cameras around a predetermined time. The probability hypothesis density filtering algorithm is used to calculate the proximity probability between the actual location of each candidate vehicle and the spatial region, as well as the similarity probability between the appearance features of each candidate vehicle and the appearance features of the vehicle that drove out. Based on proximity and similarity probabilities, vehicles that leave the vehicle are paired with candidate vehicles, and candidate vehicles are assigned the same global vehicle identity identifier as vehicles that leave the vehicle.

[0049] The system continuously monitors the status of local trajectory segments generated within each camera. When the current movement direction of a local trajectory segment points to the physical boundary of the camera's field of view, and the latest coordinates are close to the boundary threshold, the system determines that the vehicle is about to leave the current camera's field of view, records it as an out-of-field trajectory, and may enter the monitoring range of adjacent cameras. At this time, the cross-camera association process is immediately triggered.

[0050] The system predicts the expected time and spatial region when a vehicle enters an adjacent shot. Based on the motion state information of the vehicle's exit trajectory in the most recent few frames, it combines a simplified vehicle kinematics model, such as a uniform linear motion model or a uniform acceleration model, to extrapolate the vehicle's future motion. It calculates the expected time required for the vehicle to move from its current position along its current direction of motion until it enters the predefined field of view boundary of the adjacent shot at the predicted position. Based on the vehicle kinematics model, it calculates the set of all positions where the vehicle may appear within the field of view of the adjacent shot within the predicted time window. This set is defined as the expected spatial region.

[0051] Within the predicted time window, the system retrieves all newly appearing or recently detected vehicle targets from the processing results of adjacent shots. Specifically, it acquires the newly initialized local trajectory segments corresponding to these vehicle targets, or the detection results of the previous few frames, and uses them as candidate vehicles, recording the actual location and appearance features of these candidate vehicles when they are first observed within the predicted spatial region.

[0052] A probability hypothesis density filtering (PHD) algorithm is employed to handle various uncertainties in the matching process. For the exiting vehicle represented by the exit trajectory and multiple retrieved candidate vehicles, the PHD filter calculates the probability that the actual observed initial position falls within the spatial region predicted by the exit trajectory for each candidate vehicle. This probability is calculated based on the spatial uncertainty model of the prediction and reflects the degree of agreement between the candidate vehicle's spatial location and the prediction. For each candidate vehicle, the algorithm calculates the similarity probability between its appearance features and the appearance features of the exiting vehicle represented by the exit trajectory, which can be achieved through a pre-trained vehicle re-identification model.

[0053] The PHD filter maintains a strength function regarding the presence and association of a target, fusing the proximity and similarity probabilities of each candidate vehicle to form a comprehensive association likelihood probability. Simultaneously, the PHD filter naturally considers the possibility of no matching candidate vehicle (i.e., missed detection, vehicles leaving the vehicle not entering the next shot), and the existence of false candidate vehicles. Through iterative updates, the filter outputs an optimal association hypothesis: which candidate vehicle has the highest probability of matching the leaving vehicle.

[0054] Based on the decision results of the PHD filter, if the association probability between a candidate vehicle and the departing vehicle exceeds a preset confidence threshold, a successful match is determined. The system directly assigns the global vehicle identity identifier held by the departing vehicle to the candidate vehicle and its corresponding local trajectory segment. This candidate vehicle will continue to use its previous global vehicle identity identifier in subsequent shots, achieving cross-camera ID transfer. If the association probability of no candidate vehicle reaches the threshold, the system considers that the departing vehicle may not have been captured by adjacent shots or the match has failed. In this case, the global ID of the departing vehicle is temporarily archived. For new candidate vehicles, the system assigns a completely new global ID, achieving high-precision cross-camera identity continuity maintenance. This significantly improves the system's anti-interference capability and robustness. By only performing time-consuming deep feature extraction and matching calculations on a very small number of candidate vehicles, the system avoids the computational explosion problem caused by comparing all permutations and combinations among all detected targets, optimizing computational resource allocation and improving overall efficiency.

[0055] In some embodiments of the present invention, the appearance features include depth features that are robust to changes in vehicle viewing angle and illumination, the depth features being extracted from image regions using a pre-trained deep neural network model.

[0056] A convolutional neural network or visual Transformer pre-trained on a large-scale general image dataset can be selected as the backbone network. The classification head is removed, and a fully connected layer is connected to map the features to a specified dimension. Subsequently, using a vehicle re-identification dataset containing a large number of images of different vehicles under different viewpoints and lighting conditions, the network is fine-tuned using a metric learning loss function, enabling the model to learn to extract feature vectors that remain unchanged for the same vehicle and clearly distinguish between different vehicles. During real-time system operation, for each detected vehicle, its image region is cropped, scaled, and normalized before being input into the trained network. A fixed-dimensional floating-point vector is output from the network's feature layer as the vehicle's depth feature. This effectively overcomes the vulnerability of traditional handmade features to viewpoint changes, lighting differences, and partial occlusion, significantly improving the accuracy and reliability of cross-camera matching.

[0057] In some embodiments of the present invention, traffic events are detected based on continuous complete trajectories, and vehicle lifecycle tracking is performed based on the traffic events and the corresponding continuous complete trajectories.

[0058] Based on the above embodiments, such as Figure 6 As shown, the process of vehicle lifecycle tracking includes: Continuously monitor the continuous and complete trajectory under each global vehicle identification identifier and acquire trajectory data; When the trajectory data meets the preset event determination rules, the corresponding type of traffic event is triggered, and a globally unique event identifier is assigned to the traffic event; Associate and bind the globally unique event identifier with the global vehicle identity identifier of the vehicle that triggered the traffic event; During the duration of a traffic incident, the status information of the traffic incident is continuously updated based on the vehicle's continuous and complete trajectory. The status information includes the latest location and duration of the traffic incident. When the conditions for establishing a traffic event based on a continuous and complete trajectory are no longer met, the traffic event is determined to be terminated, and an event record containing the traffic event's identifier, type, total duration, and related vehicle information is generated.

[0059] The system continuously monitors each continuous and complete trajectory data stream with a globally unique identifier, including world coordinates, speed, and direction information that change over time. The system has a pre-configured event determination rule base. When the analysis of a trajectory data point reveals that its behavioral characteristics meet specific rules, event detection is triggered. For example, if a vehicle's trajectory shows its speed dropping from 80 km / h to 0 within 2 seconds and remaining there for more than 60 seconds, it is determined to be an abnormal parking traffic event. Once a traffic event is triggered, the system immediately assigns a globally unique event identifier to it and strongly binds this identifier to the globally unique identifier of the vehicle that triggered the event. During the event's duration, the system continuously tracks the vehicle's subsequent trajectory and updates the event's status information in real time based on the trajectory data, including the latest location, duration, and changes in severity. The system marks the event as ended when the trajectory data determines that the event conditions are no longer met, such as when the parked vehicle leaves. It then generates a structured event record report containing the full picture of the event, including the traffic event's identifier, type, total duration, and associated vehicle information. By establishing event detection on continuous and reliable trajectory data and introducing global event identification management, a leap from discrete instantaneous alarms to continuous process control has been achieved. This completely solves the industry pain point of repeated alarms and inability to trace back the same event due to its span across different cameras or time periods. At the same time, trajectory-based lifecycle tracking allows each event to be traced back to the triggering vehicle and its complete behavioral chain, greatly improving the precision of traffic event management and post-event analysis capabilities, and providing high-value, actionable insights for traffic operation and maintenance.

[0060] Based on the same inventive concept as the traffic perception and trajectory stitching method based on a trinocular camera in the foregoing embodiments, the present invention also provides a traffic perception and trajectory stitching system based on a trinocular camera, comprising: The extraction and mapping module acquires traffic video streams covering different distance segments (near, medium, and far) using close-range, medium-range, and long-range lenses and extracts synchronized image frames; through joint calibration, it maps multiple synchronized image frames to the same world coordinate system. The detection and acquisition module detects the synchronized image frames of each video stream in the world coordinate system, acquires the image region of the vehicle in the synchronized image frame, and acquires the position coordinates of the image region in the world coordinate system. The local generation module tracks and identifies vehicles in the video stream of each shot based on their position coordinates, generating local trajectory segments of the same vehicle in each shot. The identification and association module predicts the spatiotemporal region where a vehicle enters an adjacent shot based on local trajectory segments. Within the spatiotemporal region, it matches the vehicle's appearance features and associates the local trajectory segments of different shots with the same global vehicle identity identifier. The stitching module stitches together all local trajectory segments based on the global vehicle identification identifier to form a continuous and complete trajectory.

[0061] The stitching system described above in this invention can effectively realize traffic perception and trajectory stitching methods based on tri-cameras, and the technical effects it can achieve are as described in the above embodiments, and will not be repeated here.

[0062] In some embodiments of the present invention, the mapping extraction module further includes: The casing is manufactured using a combination of an aluminum alloy frame and carbon fiber composite skin. While ensuring structural strength and high heat dissipation efficiency, the total weight is kept below 5 kg, facilitating on-site installation. With an IP67 protection rating, it completely prevents dust intrusion and can withstand short-term immersion, making it sufficient to withstand outdoor wind, rain, and sandstorms. Its wide operating temperature design, ranging from -30℃ to 70℃, enables stable operation in extreme climates across most parts of China. The unique internal cooling solution combines air cooling with thermally conductive silicone pads. A small vortex fan generates airflow, and the high heat generated by the edge computing unit is conducted to the metal casing through a large-area thermally conductive silicone pad for dissipation. This ensures that the temperature rise of core components does not exceed 10℃ when the system is running at full load, avoiding performance degradation or system crashes due to overheating.

[0063] The optical lens assembly includes close-range, medium-range, and long-range lenses rigidly fixed to the housing in a longitudinal arrangement, each responsible for monitoring tasks at different distances, achieving seamless visual coverage from 0 meters to 500 meters. The close-up lens is located at the top or front, with a focal length of 4-8mm, providing an ultra-wide-angle field of view of over 120°. It focuses on monitoring the panoramic traffic flow within a range of 0-100 meters and is particularly good at detecting small-volume debris, pedestrians, or non-motorized vehicles running in from the road.

[0064] The mid-range lens, located in the center with a focal length of 12-25mm, is the mainstay for tracking and event detection. Its field of view of 75-250 meters covers the area where vehicle interaction is most frequent, and it supports HDR to ensure clear imaging even in scenes with strong contrast between light and dark.

[0065] The telephoto lens is located at the bottom or rearmost position, with a focal length of 50-100mm. It is responsible for capturing details of targets 200-500 meters away. The supported optical image stabilization function effectively suppresses image blur caused by pole vibration and wind load, while the fog penetration function improves the observation capability in adverse weather conditions such as rain, fog, and haze.

[0066] The three lenses are connected to the image processing interface on the core board via high-speed data cables, such as FPC flexible circuit boards, to ensure that the three high-definition video streams can be transmitted to the subsequent processing unit with low latency and synchronously.

[0067] The edge computing unit is connected to the optical lens assembly, and the video stream captured by each lens is processed in parallel on the edge computing unit.

[0068] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of the application as defined herein, and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. A traffic perception and trajectory stitching method based on a three-camera, characterized in that, Includes the following steps: Traffic video streams covering different distance segments (near, medium, and far) are captured using close-up, medium-range, and long-range lenses, and synchronized image frames are extracted. The multiple synchronized image frames are uniformly mapped to the same world coordinate system through joint calibration; In the world coordinate system, the synchronized image frame of each video stream is detected, the image region of the vehicle in the synchronized image frame is obtained, and the position coordinates of the image region in the world coordinate system are obtained. Based on the location coordinates, the vehicle is tracked and identified in the video stream of each shot, and a local trajectory segment of the same vehicle in each shot is generated. Based on the local trajectory segments, the spatiotemporal region where the vehicle enters the adjacent camera is predicted. Within the spatiotemporal region, the vehicle's appearance features are matched to associate the local trajectory segments of different cameras with the same global vehicle identity identifier. All the local trajectory segments are spliced ​​together based on the global vehicle identification to form a continuous and complete trajectory.

2. The method of claim 1, wherein, The process of mapping multiple synchronized image frames to the same world coordinate system through joint calibration includes: Based on the synchronized image frames from the close-range, medium-range, and long-range lenses, the internal parameters of each lens are calculated respectively. Calculate the first relative external parameter between the close-up lens and the medium-range lens, and the second relative external parameter between the medium-range lens and the telephoto lens based on the internal parameters. The world coordinate system is established with the imaging center of the mid-range lens as the origin, and the mapping relationship between the synchronized image frames of the close-range lens and the long-range lens and the world coordinate system is obtained through the first relative external parameter and the second relative external parameter. Based on the mapping relationship, the synchronized image frames of the close-up lens and the long-distance lens are uniformly mapped to the world coordinate system.

3. The method of claim 2, wherein, It also includes optimizing the mapping relationship: Based on the synchronized image frames from the close-up, medium-range, and long-range lenses, inherent static linear features in the actual traffic scene are extracted; Based on the currently valid mapping relationship, the static linear features corresponding to the same physical entity in the close-up, medium-range, and long-range shots are respectively mapped to the world coordinate system; Calculate the spatial position error between different shots mapped to the same static linear feature in the world coordinate system; With the goal of minimizing the spatial position error, the first relative external parameter and the second relative external parameter are iteratively adjusted to obtain the optimized mapping relationship; The optimized mapping relationship is used to uniformly map the synchronized image frames of the close-up and long-distance shots to the world coordinate system.

4. The method of claim 1, wherein, The process of generating local trajectory segments of the same vehicle in each shot includes: Create and maintain a trajectory list to record vehicle trajectories in the current video stream. Each trajectory in the trajectory list includes motion state information and coordinate sequence, and each trajectory is marked as a continuous trajectory or a terminated trajectory. For each continuous trajectory in the trajectory list, based on the motion state information and coordinate sequence of the continuous trajectory, the expected position of the continuous trajectory in the subsequent synchronized image frames is predicted; Based on the predicted expected locations in the trajectory list and the vehicle's location coordinates, an association matching is performed; For the vehicles that are successfully matched, the motion state information and coordinate sequence corresponding to the continuous trajectory are updated using the location coordinates, and recorded in the trajectory list; When the vehicle corresponding to the continuous trajectory leaves the video stream of the current camera, the continuous trajectory is changed to the terminated trajectory; Remove the termination trajectory from the trajectory list and output the accumulated coordinate sequence of the termination trajectory as a local trajectory segment.

5. The method of claim 1, wherein, Associating the local trajectory segments from different shots with the same global vehicle identity identifier includes the following process: When it is determined that the vehicle is about to leave the current camera's field of view based on the local trajectory segment, the expected time and expected spatial region of the vehicle entering the field of view of the adjacent camera are predicted based on the local trajectory segment and the vehicle kinematics model. The expected time and the spatial region constitute the spatiotemporal region. Within the spatial region, multiple candidate vehicles detected by adjacent lenses near the expected time are acquired; A probability hypothesis density filtering algorithm is used to calculate the proximity probability between the actual location of each candidate vehicle and the spatial region, as well as the similarity probability between the appearance features of each candidate vehicle and the appearance features of the vehicle that has driven out. Based on the proximity probability and the similarity probability, the departing vehicle is paired with the candidate vehicle, and the candidate vehicle is assigned the same global vehicle identity identifier as the departing vehicle.

6. The method of claim 5, wherein, The appearance features include depth features that are robust to changes in vehicle viewing angle and lighting conditions, and these depth features are extracted from the image region using a pre-trained deep neural network model.

7. The method of claim 1, wherein, It also includes detecting traffic events based on continuous complete trajectories and tracking the lifecycle of vehicles based on the traffic events and the corresponding continuous complete trajectories.

8. The method of claim 7, wherein, The process of tracking a vehicle's lifecycle includes: Continuously monitor the continuous and complete trajectory under each global vehicle identity identifier and acquire trajectory data; When the trajectory data meets the preset event determination rules, a traffic event of the corresponding type is triggered, and a globally unique event identifier is assigned to the traffic event; The globally unique event identifier is associated and bound with the global vehicle identity identifier of the vehicle that triggered the traffic event; During the duration of the traffic incident, the status information of the traffic incident is continuously updated based on the vehicle's continuous and complete trajectory. The status information includes the latest location and duration of the traffic incident. When the conditions for the establishment of the traffic event are no longer met based on the continuous and complete trajectory, the traffic event is determined to have ended, and an event record containing the identifier, type, total duration, and associated vehicle information of the traffic event is generated. 9.A traffic perception and trajectory stitching system based on a three-camera, using the traffic perception and trajectory stitching method based on a three-camera according to any one of claims 1 to 8. include: The extraction and mapping module acquires traffic video streams covering different distance segments (near, medium, and far) using close-range, medium-range, and long-range lenses and extracts synchronized image frames. The multiple synchronized image frames are uniformly mapped to the same world coordinate system through joint calibration; The detection and acquisition module detects the synchronized image frames of each video stream in the world coordinate system, acquires the image region of the vehicle in the synchronized image frame, and acquires the position coordinates of the image region in the world coordinate system. The local generation module tracks and identifies vehicles in the video stream of each shot based on the position coordinates, and generates local trajectory segments of the same vehicle in each shot. The identification and association module predicts the spatiotemporal region where the vehicle enters an adjacent shot based on the local trajectory segment, and matches the vehicle's appearance features within the spatiotemporal region to associate the local trajectory segments of different shots with the same global vehicle identity identifier. The stitching module stitches together all the local trajectory segments based on the global vehicle identification identifier to form a continuous and complete trajectory.

10. The traffic perception and trajectory stitching system based on a trinocular camera according to claim 9, characterized in that, The extraction mapping module further includes: shell; An optical lens assembly, including near-range, intermediate-range, and far-range lenses rigidly fixed to the housing in a longitudinal arrangement; An edge computing unit is connected to the optical lens assembly.

Images