A road vehicle recognition and tracking system based on double fisheye camera cooperation

Abstract

The application discloses a kind of road vehicle identification and tracking system based on double fish-eye camera cooperation, belong to wisdom traffic field, including: at least two fish-eye cameras are deployed in target road both sides;The fish-eye camera is calibrated, to ground establishes panoramic coordinate system, calculates the camera extrinsic matrix of each fish-eye camera coordinate system to panoramic coordinate system;Read the collection image of fish-eye camera, and after executing image pre-processing and distortion correction, vehicle target identification in image is carried out, and the detection result containing vehicle position, boundary frame and confidence is generated;According to the camera extrinsic matrix, the detection result of each fish-eye camera is projected to panoramic coordinate system;Vehicle target list is constructed;According to vehicle target list, cross-frame tracking is executed, and vehicle trajectory is continuously output.The application solves the problem of missing detection and tracking interruption caused by mutual occlusion of vehicles under single camera view, improves the integrity and accuracy of vehicle tracking.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation, specifically to a road vehicle recognition and tracking system based on dual fisheye camera collaboration. Background Technology

[0002] In intelligent transportation systems for urban roads and highways, video-based vehicle detection and tracking is one of the core technologies. However, in practical applications, severe vehicle occlusion is a major cause of tracking failures and trajectory interruptions. When vehicles are parallel, following too closely, or partially / completely occluded by large vehicles, detection algorithms from a monocular camera's perspective are prone to missed detections or ID switching.

[0003] In existing technologies, the following solutions are commonly used to address the occlusion problem: Using high-angle cameras: This involves erecting high poles to obtain a "bird's-eye view," reducing occlusion. However, this method has high infrastructure costs, inflexible deployment, and limited field of view. Using radar or lidar: Radar waves have strong penetrating power and are less affected by occlusion. However, they are expensive and cannot provide rich texture and color information, limiting their effectiveness in tasks such as vehicle classification. Deep learning-based contextual reasoning: This involves training a model to learn the shape of partially occluded vehicles. This method requires high-quality training data and remains ineffective for severe occlusion. Fisheye cameras, due to their ultra-wide field of view, can cover multiple lanes with a single camera, offering low cost and easy deployment. However, their inherent distortion and single-view limitations mean they also cannot avoid occlusion problems. Therefore, there is an urgent need for a vehicle tracking solution that maintains the advantages of low cost and wide coverage while effectively overcoming vehicle occlusion. Summary of the Invention

[0004] This application provides a road vehicle recognition and tracking system based on dual fisheye camera collaboration, aiming to solve the problems of missed detection and tracking interruption caused by mutual occlusion of vehicles under the single camera view in the prior art.

[0005] In view of the above problems, this application provides a road vehicle recognition and tracking system based on dual fisheye camera collaboration.

[0006] The first aspect disclosed in this application provides a road vehicle recognition and tracking system based on dual fisheye camera collaboration. The system includes: a deployment module for deploying at least two fisheye cameras on both sides of a target road, the fisheye cameras being used to simultaneously acquire wide-angle road images containing vehicles; a calibration module for calibrating the fisheye cameras, establishing a panoramic coordinate system with the ground, and calculating the camera extrinsic parameter matrix from each fisheye camera coordinate system to the panoramic coordinate system; a target detection module for reading the acquired images from the fisheye cameras, performing image preprocessing and distortion correction, and then recognizing vehicle targets within the images to generate detection results including vehicle position, bounding box, and confidence level; a projection module for projecting the detection results of each fisheye camera onto the panoramic coordinate system according to the camera extrinsic parameter matrix; a target association and fusion module for spatiotemporally associating the projected vehicle detection results from the fisheye cameras, performing matching analysis based on intersection-over-union ratio or Euclidean distance, and constructing a vehicle target list; and a tracking module for performing cross-frame tracking according to the vehicle target list and continuously outputting vehicle trajectories.

[0007] Another aspect of this application discloses a road vehicle recognition and tracking method based on dual fisheye camera collaboration. The method includes: deploying at least two fisheye cameras on both sides of a target road, the fisheye cameras being used to simultaneously acquire wide-angle road images containing vehicles; calibrating the fisheye cameras, establishing a panoramic coordinate system with the ground, and calculating the camera extrinsic parameter matrix from each fisheye camera coordinate system to the panoramic coordinate system; reading the acquired images from the fisheye cameras, performing image preprocessing and distortion correction, and then recognizing vehicle targets within the images to generate detection results including vehicle position, bounding box, and confidence level; projecting the detection results from each fisheye camera to the panoramic coordinate system according to the camera extrinsic parameter matrix; spatiotemporally associating the projected vehicle detection results from the fisheye cameras, performing matching analysis based on intersection-over-union ratio (IoU) or Euclidean distance, and constructing a vehicle target list; and performing cross-frame tracking based on the vehicle target list, continuously outputting vehicle trajectories.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] By employing a complete technical solution that involves deploying fisheye cameras on both sides of the road to collaboratively acquire images, mapping multi-path detection results to a unified panoramic coordinate system through calibration and projection, performing correlation fusion based on spatiotemporal intersection-union or Euclidean distance, and finally executing cross-frame tracking, this solution solves the technical problems of easy tracking interruption and blind spots in monocular vision systems when vehicles are occluded. It achieves the technical effect of effectively eliminating occlusion interference by utilizing complementary information from multiple perspectives, realizing continuous and stable vehicle tracking, and accurate trajectory output.

[0010] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] Figure 1 This application provides a schematic diagram of the structure of a road vehicle recognition and tracking system based on dual fisheye camera collaboration.

[0012] Figure 2 This application provides a flowchart illustrating a road vehicle recognition and tracking method based on dual fisheye camera collaboration.

[0013] Explanation of reference numerals in the attached diagram: Deployment module 11, Calibration module 12, Target detection module 13, Projection module 14, Target association and fusion module 15, Tracking module 16. Detailed Implementation

[0014] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0015] The overall concept of the technical solution provided in this application is as follows:

[0016] This application provides a road vehicle recognition and tracking system based on dual fisheye camera collaboration. Fisheye cameras are deployed on both sides of the road to simultaneously acquire images. After calibration, the vehicle coordinates detected by each camera are uniformly projected onto the ground panoramic coordinate system. Based on this, the intersection-union ratio or Euclidean distance of the vehicle detection boxes is calculated for correlation matching, and the dual-view data is fused to generate a global vehicle list. Finally, a tracking algorithm is used to achieve cross-frame trajectory correlation, thereby forming a complete technical closed loop from collaborative perception, coordinate unification, data fusion to continuous tracking.

[0017] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0018] Example 1, as Figure 1 As shown in the figure, this application provides a road vehicle recognition and tracking system based on dual fisheye camera collaboration. The system includes:

[0019] Deployment module 11 is used to deploy at least two fisheye cameras on both sides of the target road, wherein the fisheye cameras are used to simultaneously acquire wide-angle road images containing vehicles;

[0020] Specifically, target roads refer to specific road sections where vehicle identification and tracking are required, such as urban intersections, highway ramps, tunnel exits, or regular road sections prone to congestion and obstruction. These areas are typically key nodes in traffic monitoring. A fisheye camera is a special lens camera with an extremely short focal length and a very wide field of view, typically reaching 180° or even over 230°. Its advantage is that a single camera can cover a very wide area, such as multiple lanes.

[0021] Specifically, the optimal installation locations are determined based on road width and monitoring requirements, typically by selecting poles at suitable heights and relatively opposite positions on both sides of the road. At least two fisheye cameras are deployed with specific pitch and orientation angles to ensure that their ultra-wide-angle fields of view can overlap and cover the same road area, forming an effective collaborative monitoring network. By configuring synchronization signal lines or network synchronization protocols, synchronous acquisition by all cameras is achieved, thereby obtaining wide-angle road images containing vehicles from different perspectives at the same time, providing a high-quality, time-difference-free multi-source data foundation for subsequent image processing and analysis.

[0022] This module, by constructing a dual-view collaborative acquisition system, provides complementary visual information for subsequent processing, fundamentally overcoming the visual blind spots and occlusion problems under a single camera viewpoint, and laying the hardware foundation for the high-reliability perception of the entire system.

[0023] The calibration module 12 is used to calibrate the fisheye camera, establish a panoramic coordinate system with the ground, and calculate the camera extrinsic parameter matrix from each fisheye camera coordinate system to the panoramic coordinate system.

[0024] Specifically, in computer vision, calibration refers to the process of determining the camera's internal and external parameters through a series of calculations. Calibration is particularly crucial for fisheye cameras, providing the parameters needed to correct severe image distortion and establishing the spatial geometric relationship between the camera and the real world. The panoramic coordinate system, also known as the world coordinate system, specifically refers to a unified coordinate system established with the ground as the reference. Imagine it as a huge, precise "map" laid out on the road; the positions of all vehicles captured by cameras are ultimately projected onto this unified coordinate system, thus eliminating positional differences caused by different viewing angles. The fisheye camera coordinate system is a three-dimensional coordinate system with the camera's optical center as the origin and the camera's optical axis as the Z-axis. Each pixel in the image has a corresponding orientation in this coordinate system. The camera extrinsic parameter matrix is ​​a mathematical transformation matrix that defines how to transform three-dimensional points in the fisheye camera coordinate system to the panoramic coordinate system. Simply put, it describes "where the camera is and which direction it's looking."

[0025] Specifically, a calibration board of known precise size, such as a 30mm×30mm checkerboard with each square, is placed within the common field of view of the cameras. The origin of the calibration board coordinate system is defined at one of its corner points, with the Z-axis perpendicular to the board surface, thus establishing the calibration board coordinate system. The calibration board is placed flat on the ground (at this point, the calibration board coordinate system and the target panoramic coordinate system only have a fixed translation relationship, i.e., the height difference between the board surface and the ground), and the calibration board is moved or rotated to acquire multiple images from different angles. The Zhang Zhengyou calibration method is used to detect the pixel coordinates of the checkerboard corner points in each image. Using the known physical coordinates of the corner points (in the calibration board coordinate system) and the camera intrinsic parameters, a least-squares problem is solved to calculate the camera extrinsic parameter matrix corresponding to each image, i.e., the rotation matrix R and translation vector T from the calibration board coordinate system to the camera coordinate system. Through a fixed rigid body transformation, the camera extrinsic parameters based on the calibration board coordinate system are transformed to the panoramic coordinate system with the ground as the origin, thus obtaining the final extrinsic parameter matrix of each camera relative to the panoramic coordinate system. The extrinsic parameter matrix contains the spatial position and orientation information of each camera, thereby establishing a spatial mapping relationship from their respective independent fisheye camera coordinate systems to a unified panoramic coordinate system, laying the geometric foundation for subsequently projecting the detection results from different perspectives onto the same "map".

[0026] By establishing a precise spatial transformation relationship from each camera's perspective to a unified ground coordinate system, a common spatiotemporal reference benchmark is provided for vehicle detection data from different cameras. This is the core and prerequisite for achieving accurate geometric fusion of multi-view information.

[0027] The target detection module 13 is used to read the images acquired by the fisheye camera, and after performing image preprocessing and distortion correction, to identify vehicle targets within the image and generate detection results including vehicle position, bounding box and confidence level.

[0028] Specifically, image preprocessing includes: image scaling (adjusting the image to a fixed size required by the model), color space conversion (e.g., BGR to RGB), and normalization (scaling pixel values ​​from 0-255 to 0-1 or standardizing the distribution), to make the data more suitable for model processing. Distortion correction is a key preprocessing step specific to fisheye cameras. Using calibrated camera parameters, mathematical transformations are used to correct geometric distortions caused by lens characteristics, restoring curved wide-angle images to normal images conforming to perspective projection. Bounding boxes are used to visually indicate the location and extent of vehicles in the image. Confidence is a value between 0 and 1, representing the detection model's certainty in its judgment. For example, a bounding box labeled "car" with a confidence of 0.95 means the model is 95% confident that the area within the box is indeed a car. This parameter is used to filter out unreliable detection results in subsequent processes.

[0029] Specifically, raw wide-angle road images synchronously acquired by fisheye cameras deployed on both sides of the road are read through a data transmission network. Image preprocessing and critical distortion correction operations are performed on each raw image sequentially. The correction process uses pre-calibrated camera internal parameters and distortion coefficients to eliminate barrel distortion caused by the fisheye lens, resulting in a geometrically accurate corrected image. The corrected image is then identified using a target detection algorithm (such as YOLO or RCNN series) to obtain all vehicle targets in the image. A structured detection result is generated for each vehicle, which includes the pixel coordinates of the vehicle's bounding box and the confidence score that the object within the bounding box is a vehicle. This provides accurate single-view perception data for subsequent coordinate projection and data fusion.

[0030] By correcting and intelligently analyzing the original fisheye images, the complex image data is transformed into a structured and quantified list of vehicle target information, providing accurate and reliable single-view perception input for subsequent multi-view data fusion and tracking. This forms the data foundation for the entire system to achieve precise vehicle perception and trajectory reconstruction.

[0031] Projection module 14 is used to project the detection results of each fisheye camera onto the panoramic coordinate system according to the camera extrinsic parameter matrix.

[0032] Specifically, the single-view detection result output by the target detection module is typically obtained as the pixel coordinates of the bottom midpoint of the vehicle bounding box in the image, as it is the closest point of contact between the vehicle and the ground. Using the camera intrinsic parameter matrix obtained during the camera calibration phase, it is converted into normalized coordinates (X, Y, 1) in the camera coordinate system through inverse perspective transformation. This point lies on a ray emanating from the camera's optical center, but its specific depth is unknown. A crucial step is introducing a key geometric constraint: the Z=0 plane of the panoramic coordinate system. Assuming the vehicle contact point is located on the ground plane, the ray equation is combined with the aforementioned three-dimensional ray equation to solve for the specific three-dimensional coordinates (X, Y, 1) of this point in the camera coordinate system. c Y c Z c ), where Z c The distance from the camera's optical center to the ground is determined; using the pre-calibrated camera extrinsic parameter matrix (including the rotation matrix R and translation vector T), the coordinate transformation formula [X...] is applied. w Y w Z w ]=R*[X c Y c Z c The obtained 3D point is transformed from the camera coordinate system to a unified panoramic coordinate system, thus obtaining the vehicle's true 2D coordinates on the ground, completing the mapping from image pixels to world coordinates. Due to the ground assumption, Z... wThis can be considered as 0, thus obtaining the vehicle's two-dimensional coordinates (X, Y, F, G, I) in the panoramic coordinate system. w Y w ), complete the projection.

[0033] By using geometric transformation, vehicle detection results from different camera perspectives, existing in the image pixel coordinate system, are uniformly mapped to a panoramic coordinate system based on the ground. This transforms the originally isolated, viewpoint-dependent image position information into unified spatial coordinates with actual physical meaning, providing a common and quantifiable spatial reference benchmark for the subsequent accurate association and fusion of detection results from different camera perspectives.

[0034] The target association and fusion module 15 is used to perform spatiotemporal association of the projected vehicle detection results from the fisheye camera, perform matching analysis based on the intersection-union ratio or Euclidean distance, and construct a list of vehicle targets.

[0035] Specifically, the projected vehicle detection results refer to the data processed by the "projection module." That is, the vehicle bounding boxes, originally existing in their respective fisheye camera images, have been transformed into coordinate representations in a unified panoramic coordinate system. Spatiotemporal correlation refers to the process of establishing correspondences between data from different sources in time and space. Spatial correlation refers to determining whether two targets detected from different camera perspectives are close enough in the panoramic coordinate system that they are likely the same physical entity. Temporal correlation ensures that the data used for correlation analysis was collected at the same point in time or within a very short time interval, guaranteed by synchronous camera acquisition. The intersection-union ratio (IU / U) is a metric used in computer vision to measure the degree of overlap between two bounding boxes. The formula is: the area of ​​their intersection divided by the area of ​​their union. The closer the value is to 1, the higher the overlap. Euclidean distance refers to the straight-line distance between the center points of two vehicle detection boxes in the panoramic coordinate system (two-dimensional plane). The smaller the distance, the closer the two points are in spatial location. The vehicle target list is a unified, non-repeating set of vehicle information. Each entry in the list represents a vehicle uniquely identified by the system, containing information such as its fused position, ID, and speed in the panoramic coordinate system.

[0036] Specifically, the system acquires projected vehicle detection results from two fisheye cameras, projected onto a panoramic coordinate system. It then performs spatiotemporal correlation on all existing detection results at the current moment, the core of which is matching analysis. This involves calculating the intersection-over-union ratio (IoU) or Euclidean distance between the center points of any two vehicle detection boxes from different cameras. If the IoU between a detection box from camera A and another detection box from camera B is greater than a preset threshold (e.g., 0.5) or the Euclidean distance is less than another preset threshold, they are determined to be the same vehicle. Then, based on the matching results, the detection information from both perspectives is fused, such as by taking the average position or selecting results with high confidence. An entry is created for each uniquely identified vehicle, ultimately constructing a global, non-repeating list of vehicle targets. This list integrates information from both perspectives, effectively compensating for occlusion-related missed detections from a single perspective.

[0037] By performing objective and quantitative correlation matching within a unified coordinate space of multi-view data, the perception information from different cameras was successfully fused into a single, global vehicle target view. This fundamentally solved the problem of target omission caused by single-view occlusion and generated more accurate and complete vehicle position information, providing high-quality data input for subsequent stable tracking.

[0038] The tracking module 16 is used to perform cross-frame tracking based on the vehicle target list and continuously output vehicle trajectories.

[0039] Specifically, cross-frame tracking refers to the process of maintaining a unique identity (ID) for each individual vehicle target within a continuous time series (video frames) and connecting their positions in different frames. Its core objective is to solve the data association problem of "who is the same vehicle in different frames." A vehicle trajectory refers to a continuous set of all historical positions (coordinate point sequences) of the same vehicle ID over a period of time. It describes the continuous movement path of the vehicle in the panoramic coordinate system and is the final output of the system.

[0040] Specifically, the system obtains the fused list of vehicle targets at the current moment; the tracking module starts cross-frame tracking, uses a Kalman filter tracker to predict the latest position of historical vehicles, and performs similarity-based Hungarian matching with the currently detected vehicle position; unmatched vehicles are used to predict future coordinates through the Kalman filter tracker, and target matching is performed in subsequent frames; if an unmatched historical vehicle meets the third threshold, it is removed from the vehicle target list, and the tracking of the corresponding vehicle is abandoned.

[0041] By using motion prediction and data association algorithms, independent vehicle detection points in discrete frames are linked into continuous and smooth motion trajectories. Each target is assigned and maintained with a unique identity ID, thereby achieving continuous, stable, and ID-free tracking and monitoring of road vehicles. The final output is a vehicle motion trajectory with temporal sequence and uniqueness, providing a key data foundation for intelligent traffic management.

[0042] Furthermore, the system contains two fisheye cameras, which are deployed opposite each other on both sides of the target road. The optical axes of the two fisheye cameras are opposite each other or at a certain angle. The resolution, frame rate, and exposure parameters of the two fisheye cameras are kept uniformly calibrated, and the two fisheye cameras constitute a cooperative sensing unit.

[0043] Specifically, a relative deployment refers to two cameras positioned on opposite sides of the target road, with their installation points roughly facing each other in space. This layout aims to ensure that the main field of view of one camera can cover part of the blind spot or easily obstructed area of ​​the other camera, thus achieving cross-coverage of the field of view. The optical axis can be simply understood as the direction of the central axis of the camera lens. A collaborative sensing unit is a definition of an overall system composed of two fisheye cameras. It emphasizes that these two cameras do not work independently, but rather as a whole unit, working together to complete the sensing task through information interaction and fusion, achieving a "1+1>2" effect.

[0044] Specifically, on both sides of the selected target road, two installation points with opposite positions are chosen to ensure effective cross-coverage of the field of view; two fisheye cameras are deployed, and their optical axes are adjusted to be opposite each other or form an optimal angle to maximize coverage of key monitoring areas and reduce blind spots; the resolution, frame rate, and exposure parameters of the two cameras are uniformly calibrated to ensure that the acquired images maintain a high degree of consistency in spatiotemporal and photometric attributes, laying the foundation for subsequent accurate correlation and fusion; the two fisheye cameras, which are physically and parametrically aligned, are logically bound together to form a unified collaborative sensing unit, enabling it to perform synchronous image acquisition, data transmission, and collaborative analysis as a whole system.

[0045] By constructing a collaborative perception unit with unified parameters and complementary fields of view, the system is provided with highly consistent dual-path visual data in space and time, laying the hardware foundation for the effective fusion of multi-view information and being a prerequisite for overcoming the blind spots and occlusion problems of single-view.

[0046] Furthermore, when the target detection module 13 performs image distortion correction, it obtains the camera's internal parameters based on the Zhang Zhengyou calibration method, and the correction formula is as follows:

[0047] ;

[0048] ;

[0049] in, These are internal camera parameters. , The image coordinates under ideal imaging conditions. These are the image coordinates after distortion correction.

[0050] Specifically, the Zhang Zhengyou calibration method is a widely used camera calibration method. By having the camera take multiple images of a known pattern from different angles, the camera's intrinsic parameters and distortion parameters are solved using the correspondence between the pixel coordinates of the pattern's corner points in these images and their known world coordinates.

[0051] The camera's internal parameters are a set of parameters that determine the camera's imaging geometry. In this formula, they specifically refer to: Radial distortion parameter: used to correct radial distortion caused by lens shape (image points deviating radially from or closer to the center), which is the main component of fisheye distortion. Tangential distortion parameter: used to correct distortion caused by the lens not being parallel to the imaging plane. Ideal imaging coordinates and corrected coordinates: Assume the coordinates of a three-dimensional point projected onto the image plane in an ideal pinhole camera model without distortion. In a real fisheye camera, due to distortion, the coordinates of this point on the image are actually different.

[0052] Specifically, a checkerboard calibration board is used to acquire multiple images of the camera at different positions and angles within the camera's field of view. Algorithms such as the Zhang Zhengyou calibration method are then applied to process these images and determine the unique intrinsic parameters of each camera, including key radial and tangential distortion parameters. During actual operation, after the system reads the original fisheye image, for each pixel requiring processing, it is first mapped onto a distortion-free ideal imaging plane based on the camera's intrinsic parameters to obtain its normalized ideal coordinates and calculate the squared distance from the center. ; then, , The calibrated distortion parameters are substituted into the correction formula for calculation. The formula, by superimposing radial and tangential distortion models, accurately solves inversely for the actual pixel coordinates of the ideal point in the original distorted image. By performing this mapping for each pixel in the image, or by building a lookup table for rapid resampling, a new image with corrected geometry and restored straight lines can be generated, thus providing accurate image input for subsequent vehicle detection.

[0053] By using precise mathematical modeling to compensate for the inherent optical distortion of the fisheye lens, the severely distorted original image is restored to a corrected image that conforms to perspective geometry. This fundamentally eliminates the interference caused by distortion on the shape, position, and scale information of the vehicle target, providing a geometrically accurate image basis for the subsequent target detection module. It is a key preprocessing step to ensure the recognition accuracy and measurement reliability of the entire system.

[0054] Furthermore, in the target detection module 13, the detection results are used to detect the two-dimensional or three-dimensional vehicle frame coordinate information of each vehicle through target detection algorithms, including YOLO series algorithms and RCNN series algorithms.

[0055] Specifically, 2D vehicle frame coordinate information typically refers to a 2D bounding box, using a rectangle whose sides are parallel to the image coordinate axes to enclose the vehicle. Common formats include the pixel coordinates of the top-left and bottom-right corners of the rectangle, or the pixel coordinates of the center point and its width and height. This provides the vehicle's position and approximate range on the image plane. 3D vehicle frame coordinate information refers to a 3D bounding box, which, in addition to containing the vehicle's 2D position in the image, also attempts to estimate the vehicle's dimensions (length, width, height) and orientation (yaw angle) in 3D space. The YOLO series of algorithms is a very popular single-stage object detection algorithm. Its core idea is to treat object detection as a regression problem, directly predicting and classifying bounding boxes on the image grid. Its advantage is its speed, making it very suitable for real-time applications. The R-CNN series of algorithms is a representative of a class of two-stage object detection algorithms (such as R-CNN, Fast R-CNN, Faster R-CNN). Its core idea is to first generate candidate regions that may contain the target, and then classify these regions and perform accurate bounding box regression. Its advantage is high accuracy.

[0056] Specifically, after the system completes the preprocessing and distortion correction of the fisheye image, the corrected image is input into a pre-trained deep learning model. This model is built on mature object detection algorithms such as the YOLO series or RCNN series. The model performs forward propagation inference on the image, extracts deep features of the image through its convolutional neural network, and decodes these features in the output layer. Finally, it detects each vehicle target in the image and generates the corresponding detection result. This result not only includes the category of the target (such as car, truck), but more importantly, it provides its precise geometric position information. That is, depending on the algorithm capability, it can be two-dimensional frame coordinate information describing the vehicle's planar position in the image, or it can be three-dimensional frame coordinate information that can estimate the vehicle's true size and orientation and has more spatial representation. At the same time, it also gives a confidence score that represents the reliability of the detection.

[0057] Taking the YOLOv5s model as an example, its structure mainly consists of Backbone (CSPDarknet), Neck (PANet), and Head (three detection scales). During training, a large-scale labeled road vehicle dataset, such as BDD100K and COCO, is prepared. Vehicles in each image are accurately labeled using two-dimensional bounding boxes or three-dimensional boxes including orientation angles. The images are uniformly scaled to 640x640 pixels and data augmentation is performed, such as Mosaic and random color adjustment. The training process uses the SGD optimizer with an initial learning rate of 0.01, combined with a cosine annealing strategy. The loss function is composed of a weighted average of classification loss, bounding box regression loss, and object confidence loss. The model parameters are optimized through multiple rounds of iteration until the loss converges, finally obtaining a weight file that can efficiently and accurately output vehicle position and confidence for system deployment.

[0058] Taking Faster R-CNN as an example, its model construction includes a shared convolutional backbone network, such as ResNet-50+FPN, for feature extraction; a region proposal network responsible for generating candidate regions; and a RoI pooling layer and subsequent classification and bounding box regression networks. During training, the backbone network is pre-trained using the ImageNet dataset, and end-to-end training is performed on a dataset containing vehicle bounding box annotations. The training process adopts a multi-task loss function, simultaneously optimizing the candidate box generation of RPN (distinguishing foreground / background and initially regressing coordinates) and the accurate classification (vehicle type) and bounding box fine-tuning of the final detection head. The network weights are iteratively updated through optimizers such as SGD, ultimately resulting in a model that can generate high-precision two-dimensional vehicle detection results.

[0059] By introducing a powerful and optional deep learning object detection algorithm, the corrected image is transformed into a structured list of vehicle targets with accurate geometric information and category confidence. This provides high-quality, quantifiable single-view perception data for subsequent projection of image coordinates onto the real-world coordinate system and multi-view fusion, and is the core algorithm for the entire system to achieve high-precision vehicle recognition and tracking.

[0060] Furthermore, in the target association and fusion module 15, the matching analysis performed based on the intersection-union ratio or Euclidean distance includes:

[0061] If the intersection-union ratio of the vehicle detection frames in the projected vehicle detection results of two fisheye cameras is greater than the first preset threshold or the Euclidean distance is less than the second preset threshold, then they are determined to be the same vehicle, and information fusion of the projected vehicle detection results is performed to construct a vehicle target list.

[0062] Specifically, the first preset threshold is a pre-set IoU threshold value, such as 0.5, used to determine whether the overlap is high enough. When the IoU is greater than this value, the match is considered successful. The second preset threshold is a pre-set distance threshold value, such as 1 meter, used to determine the proximity. When the Euclidean distance is less than this value, the match is considered successful.

[0063] Specifically, vehicle detection bounding box data from two fisheye cameras projected onto a panoramic coordinate system are acquired and matched. All bounding box pairs (each pair containing a box from the left camera and a box from the right camera) are traversed, and their intersection-over-union (IoU) ratio and Euclidean distance (EMD) are calculated. Pre-defined decision rules are applied: if the IoU ratio of any bounding box pair is greater than a first preset threshold or the Euclidean distance is less than a second preset threshold, the pair is determined to represent the same vehicle. For successfully matched vehicles, information fusion is performed, such as weighted averaging of the coordinates of the two boxes or selecting the result with higher confidence, to generate a more accurate representation of the vehicle's location. Finally, all fused vehicles and unmatched vehicles (i.e., those detected by only one camera) are uniformly included in a global vehicle target list, thereby constructing a complete and unique vehicle perception view.

[0064] By setting dual-threshold matching rules, the detection results from multiple perspectives were quickly and accurately correlated and fused, effectively solving the problem of missed detections caused by single-view occlusion, generating a more complete and reliable global vehicle list, and providing a high-quality data foundation for subsequent tracking.

[0065] Furthermore, if the vehicle detection results from both fisheye cameras are present, the information fusion method is to select the vehicle detection result with higher confidence or perform weighted average processing of the bounding box coordinates. If only one vehicle detection result is present, the corresponding vehicle detection result is filled into the vehicle target list.

[0066] Specifically, the projected vehicle detection results all exist, meaning that within the same matching period, both fisheye cameras successfully detected the same vehicle that might exist, generating their respective bounding boxes in the panoramic coordinate system. This typically occurs when the vehicle is located in an overlapping area with clear and unobstructed views from both cameras. Weighted averaging of bounding box coordinates is a data fusion method. It involves calculating a weighted average of the coordinates of the two bounding boxes. The weights can be assigned based on confidence levels (higher confidence levels have greater weights) or simply by taking an arithmetic mean. The goal is to obtain a more accurate and stable estimate in spatial location.

[0067] Specifically, after completing the matching determination, the system executes differentiated fusion strategies based on different matching situations: If the projected vehicle detection results of both fisheye cameras exist, i.e., the match is successful, the system will perform information fusion. Specifically, it will prioritize selecting the result with higher confidence as the final output to utilize the most reliable observation data, or perform weighted averaging on the bounding box coordinates of the two detection results to smooth noise and improve positioning accuracy through data fusion. Secondly, if only one projected vehicle detection result exists, i.e., a camera fails to detect the target due to occlusion, the system will not discard it as a false detection, but will directly fill this unique detection result from the unoccluded viewpoint into the global vehicle target list, thereby effectively avoiding missed detections caused by occlusion and ensuring the continuity of vehicle tracking.

[0068] By formulating refined data fusion and adoption rules based on different scenarios, the positioning accuracy and reliability are improved not only when both views are visible, but more importantly, the observation results of the unobstructed view can be effectively adopted when a single view is occluded. This achieves the ultimate goal of dual-camera collaboration at the algorithm level, significantly reduces the target miss rate caused by occlusion, and greatly enhances the perception robustness of the system in complex traffic scenarios.

[0069] Furthermore, the tracking module 16 is also used to: predict the latest position of historical vehicles using a Kalman filter tracker and perform similarity-based Hungarian matching with the currently detected vehicle position; predict the future coordinates of unmatched vehicles using a Kalman filter tracker and perform target matching in subsequent frames; if an unmatched historical vehicle meets the third threshold, it is removed from the vehicle target list and the tracking of the corresponding vehicle is abandoned.

[0070] Specifically, the Kalman filter tracker is an algorithm that uses the system's historical state and current observations to optimally estimate the system's future state. In target tracking, it builds a dynamic model (including position, velocity, etc.) for each tracked target (vehicle) to predict the target's most likely location in the next frame, and updates the model after obtaining new observations to make the trajectory smoother and more stable. Historical vehicles refer to vehicle targets that have been successfully tracked by the system in previous frames and assigned a unique ID. The tracker maintains a growing trajectory and a motion state estimate based on Kalman filtering for these targets. Hungarian matching is a classic combinatorial optimization algorithm for solving data association problems. It is used to optimally assign two sets of elements, here "predicted historical vehicles" and "currently detected vehicles," in a one-to-one manner. Unmatched vehicles include two cases: a target is detected in the current frame but does not match any historical trajectory (possibly a newly appeared vehicle); a historical trajectory is not assigned to any currently detected target (possibly the vehicle was briefly occluded or has moved away). The third threshold is a preset frame number threshold (e.g., 10 consecutive frames). It is used to determine whether a trajectory that has not matched successfully for a long time should be terminated to avoid wasting tracker memory and computational resources.

[0071] Specifically, for each historical vehicle, the tracking module uses its corresponding Kalman filter tracker to predict its latest position in the current frame based on the state (position, speed) of the previous frame; it calculates the similarity (e.g., center point distance) between all predicted positions and the positions of all newly detected vehicles in the current frame's "vehicle target list," and performs Hungarian matching based on this cost matrix to determine which historical trajectory the current detection should belong to; for unmatched historical trajectories (usually meaning the vehicle may be occluded or temporarily disappear), the system does not delete them immediately, but continues to predict their future coordinates through the Kalman filter tracker and continuously attempts to match them with new detected targets in subsequent frames; if a historical trajectory meets the requirement that the number of consecutive unmatched frames reaches the third threshold, the system determines that the target has stably disappeared (e.g., driven out of the field of view), removes it from the active vehicle target list, abandons the tracking of the corresponding vehicle, and thus completes the lifecycle management of the trajectory.

[0072] By integrating the predictive capabilities of Kalman filtering with the global optimal matching of the Hungarian algorithm, and combining it with a threshold-based trajectory management mechanism, stable and continuous correlation of vehicle trajectories in the time dimension is achieved. This effectively handles complex scenarios such as vehicle interaction and occlusion, significantly reduces the ID switching frequency during tracking, and ensures the effective utilization of system resources.

[0073] In summary, the road vehicle recognition and tracking system based on dual fisheye camera collaboration provided in this application has the following technical effects:

[0074] By constructing a complete technology chain consisting of a deployment module, a calibration module, a target detection module, a projection module, a target association and fusion module, and a tracking module, wide-angle images are simultaneously acquired using fisheye cameras deployed on both sides of the road to obtain complementary fields of view. Then, through calibration and projection, the detection results from different perspectives are unified into a panoramic coordinate system. Subsequently, spatiotemporal association and fusion are performed based on intersection-union ratio or Euclidean distance to eliminate missed detections caused by single-view occlusion. Finally, vehicle trajectories are continuously output through cross-frame tracking. Thus, through systematic multi-source collaborative technology, the technical effect of effectively overcoming single-view visual blind spots and occlusion interference, and achieving continuous and stable vehicle tracking and accurate trajectory generation is achieved.

[0075] Example 2 is based on the same inventive concept as the road vehicle recognition and tracking system based on dual fisheye cameras in the previous examples, such as... Figure 2 As shown in the figure, this application provides a road vehicle recognition and tracking method based on dual fisheye camera collaboration, the method including:

[0076] Step S100: Deploy at least two fisheye cameras on both sides of the target road. The fisheye cameras are used to simultaneously acquire wide-angle road images containing vehicles. Step S200: Calibrate the fisheye cameras, establish a panoramic coordinate system with the ground, and calculate the camera extrinsic parameter matrix from each fisheye camera coordinate system to the panoramic coordinate system. Step S300: Read the acquired images from the fisheye cameras, perform image preprocessing and distortion correction, and then perform vehicle target recognition within the images to generate detection results including vehicle position, bounding box, and confidence level. Step S400: Project the detection results of each fisheye camera to the panoramic coordinate system according to the camera extrinsic parameter matrix. Step S500: Spatiotemporally correlate the projected vehicle detection results from the fisheye cameras, perform matching analysis based on intersection-over-union ratio or Euclidean distance, and construct a vehicle target list. Step S600: Perform cross-frame tracking according to the vehicle target list and continuously output vehicle trajectories.

[0077] Furthermore, the method uses two fisheye cameras, which are deployed opposite each other on both sides of the target road. The optical axes of the two fisheye cameras are opposite each other or at a certain angle. The resolution, frame rate, and exposure parameters of the two fisheye cameras are kept uniformly calibrated, and the two fisheye cameras constitute a cooperative sensing unit.

[0078] Furthermore, when performing image distortion correction in step S300, the camera's internal parameters are obtained based on the Zhang Zhengyou calibration method, and the correction formula is as follows:

[0079] ;

[0080] ;

[0081] in, These are internal camera parameters. , The image coordinates under ideal imaging conditions. These are the image coordinates after distortion correction.

[0082] Furthermore, in step S300, the detection results are used to detect the two-dimensional or three-dimensional vehicle frame coordinate information of each vehicle through a target detection algorithm, which includes the YOLO series algorithm and the RCNN series algorithm.

[0083] Furthermore, in step S500, performing matching analysis based on the intersection-union ratio or Euclidean distance includes: when the intersection-union ratio of the vehicle detection frames of the projected vehicle detection results of the two fisheye cameras is greater than a first preset threshold or the Euclidean distance is less than a second preset threshold, they are determined to be the same vehicle, and information fusion of the projected vehicle detection results is performed to construct a vehicle target list.

[0084] Furthermore, if the vehicle detection results from both fisheye cameras are present, the information fusion method is to select the vehicle detection result with higher confidence or perform weighted average processing of the bounding box coordinates. If only one vehicle detection result is present, the corresponding vehicle detection result is filled into the vehicle target list.

[0085] Furthermore, step S600 also includes: using a Kalman filter tracker to predict the latest position of historical vehicles and performing similarity-based Hungarian matching with the currently detected vehicle position; predicting the future coordinates of unmatched vehicles using the Kalman filter tracker and performing target matching in subsequent frames; if an unmatched historical vehicle meets a third threshold, it is removed from the vehicle target list and the tracking of the corresponding vehicle is abandoned.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A road vehicle recognition and tracking system based on dual fisheye camera collaboration, characterized in that, The system includes: A deployment module is used to deploy at least two fisheye cameras on both sides of the target road, the fisheye cameras being used to simultaneously acquire wide-angle road images containing vehicles; The calibration module is used to calibrate the fisheye camera, establish a panoramic coordinate system with the ground, and calculate the camera extrinsic parameter matrix from each fisheye camera coordinate system to the panoramic coordinate system. The target detection module is used to read the images captured by the fisheye camera, and after performing image preprocessing and distortion correction, it performs vehicle target recognition within the image and generates detection results including vehicle position, bounding box and confidence level. The projection module is used to project the detection results of each fisheye camera onto the panoramic coordinate system according to the camera extrinsic parameter matrix; The target association and fusion module is used to perform spatiotemporal association of the vehicle detection results projected by the fisheye camera, perform matching analysis based on the intersection-union ratio or Euclidean distance, and construct a list of vehicle targets. The tracking module is used to perform cross-frame tracking based on the vehicle target list and continuously output vehicle trajectories.

2. The road vehicle recognition and tracking system based on dual fisheye camera collaboration as described in claim 1, characterized in that, The system contains two fisheye cameras, which are deployed opposite each other on both sides of the target road. The optical axes of the two fisheye cameras are opposite each other or at a certain angle. The resolution, frame rate, and exposure parameters of the two fisheye cameras are kept uniformly calibrated, and the two fisheye cameras constitute a cooperative sensing unit.

3. The road vehicle recognition and tracking system based on dual fisheye camera collaboration as described in claim 1, characterized in that, When the target detection module performs image distortion correction, it obtains the camera's internal parameters based on the Zhang Zhengyou calibration method. The correction formula is as follows: ； ； in, These are internal camera parameters. , The image coordinates under ideal imaging conditions. These are the image coordinates after distortion correction.

4. The road vehicle recognition and tracking system based on dual fisheye camera collaboration as described in claim 1, characterized in that, In the target detection module, the detection results are used to detect the two-dimensional or three-dimensional vehicle frame coordinate information of each vehicle through target detection algorithms, including the YOLO series algorithms and the RCNN series algorithms.

5. A road vehicle recognition and tracking system based on dual fisheye camera collaboration as described in claim 1, characterized in that, In the target association and fusion module, matching analysis based on intersection-union ratio or Euclidean distance includes: If the intersection-union ratio of the vehicle detection frames in the projected vehicle detection results of two fisheye cameras is greater than the first preset threshold or the Euclidean distance is less than the second preset threshold, then they are determined to be the same vehicle, and information fusion of the projected vehicle detection results is performed to construct a vehicle target list.

6. The road vehicle recognition and tracking system based on dual fisheye camera collaboration as described in claim 5, characterized in that, If both fisheye cameras detect vehicles, the information fusion method is to select the vehicle detection result with higher confidence or perform a weighted average of the bounding box coordinates. If only one vehicle detection result is detected, the corresponding vehicle detection result is added to the vehicle target list.

7. A road vehicle recognition and tracking system based on dual fisheye camera collaboration as described in claim 1, characterized in that, The tracking module is also used for: The Kalman filter tracker is used to predict the latest location of historical vehicles and perform similarity-based Hungarian matching with the location of currently detected vehicles. For vehicles that are not matched, predict their future coordinates using a Kalman filter tracker, and then perform target matching in subsequent frames. If an unmatched historical vehicle meets the third threshold, it is removed from the vehicle target list, and the corresponding vehicle tracking is abandoned.

8. A road vehicle recognition and tracking method based on dual fisheye camera collaboration, characterized in that, The method is implemented by the road vehicle recognition and tracking system based on dual fisheye camera collaboration as described in any one of claims 1-7, including: At least two fisheye cameras are deployed on both sides of the target road, and the fisheye cameras are used to simultaneously acquire wide-angle road images containing vehicles; The fisheye camera is calibrated, a panoramic coordinate system is established with the ground, and the camera extrinsic parameter matrix from each fisheye camera coordinate system to the panoramic coordinate system is calculated. After reading the images captured by the fisheye camera and performing image preprocessing and distortion correction, vehicle target recognition is performed within the image to generate detection results including vehicle position, bounding box and confidence level. The detection results of each fisheye camera are projected onto the panoramic coordinate system based on the camera extrinsic parameter matrix. The projected vehicle detection results from the fisheye camera are spatiotemporally correlated, and a matching analysis is performed based on the intersection-union ratio or Euclidean distance to construct a list of vehicle targets. Perform cross-frame tracking based on the vehicle target list and continuously output vehicle trajectories.

Images