A visual analysis method for collaborative perception at traffic intersections

Abstract

This invention relates to a visual analysis method for collaborative perception at traffic intersections. The method includes the following steps: S1, using a fast intersection calibration algorithm and acquiring camera images from different locations; S2, for each camera image, inputting it into a target detection model to obtain target detection results; S3, inputting the target detection results from each location into a target tracking model to output target tracking results; S4, transforming the target tracking results into a unified global coordinate system based on an affine transformation matrix, performing global cross-camera ID matching to obtain cross-camera tracking results; S5, smoothing the cross-camera tracking results to obtain smoothed tracking results; S6, performing visual perception based on the smoothed tracking results to obtain visual perception results. Compared with existing technologies, this invention has advantages such as ensuring the correct association of the same target under different camera perspectives and achieving consistent recognition across cameras.

Description

Technical Field

[0001] This invention relates to the field of traffic scene perception, and in particular to a visual analysis method for collaborative perception at traffic intersections. Background Technology

[0002] With the continuous expansion of urban traffic scale and the increasing complexity of traffic flow, the importance of intelligent transportation systems in improving traffic management efficiency and ensuring road safety is becoming increasingly prominent. As key nodes in the traffic network, traffic intersections face complex traffic environments and ever-changing traffic behaviors, placing higher demands on perception technologies. Traditional single-camera monitoring systems have many shortcomings in terms of coverage, limited field of view, and multi-target recognition and tracking, making it difficult to comprehensively and accurately capture the dynamic changes at intersections. To address these challenges, multi-camera collaborative perception technology has emerged. By deploying multiple cameras at different locations within a traffic intersection, comprehensive monitoring of the entire intersection can be achieved.

[0003] Visual analysis for multi-camera intersection collaborative perception needs to address two major challenges: (1) rapid calibration of multiple cameras, and (2) perception fusion and cross-camera target association. Regarding the first challenge, traditional calibration methods, such as the chessboard calibration method, while providing high calibration accuracy, face numerous challenges in practical applications. Specifically, the chessboard calibration method requires precise placement of a standard chessboard pattern within the camera's field of view and ensures clear recognition from multiple camera perspectives. This is often difficult to achieve in complex traffic intersection environments, and traditional chessboard calibration methods often require a long calibration time, making it difficult to meet the needs of rapid deployment and dynamic adjustment. Regarding the second challenge, perception fusion and cross-camera target association, current cross-camera perception solutions for traffic intersections are relatively limited, and traditional methods have significant shortcomings in perception fusion and cross-camera target association. The large differences in perspective between different cameras make it difficult to maintain cross-perspective consistency, thus affecting the accuracy and stability of target association. It performs poorly in handling feature consistency issues under cross-perspective conditions, making it difficult to achieve efficient and reliable target tracking and recognition. This not only limits the perception capabilities of multi-camera systems, but also restricts their application potential in complex traffic environments. Summary of the Invention

[0004] The purpose of this invention is to improve the global consistency of multi-camera collaborative perception, achieve rapid calibration and view fusion, ensure that the same target can be correctly associated under different camera views, and realize consistent recognition across cameras. This invention provides a visual analysis method for collaborative perception at traffic intersections.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A visual analysis method for collaborative perception at traffic intersections, comprising the following steps:

[0007] S1. A fast intersection calibration algorithm is adopted. By setting several key points at different locations of traffic intersections, an affine transformation matrix is ​​generated, and camera images at different locations are acquired.

[0008] S2. For each camera image, input it into the target detection model to obtain the target detection result;

[0009] S3. The target detection results at each location are input into the target tracking model, and the target tracking results are output.

[0010] S4. Based on the affine transformation matrix, the target tracking results are transformed into a unified global coordinate system, and global cross-lens ID matching is performed to obtain cross-camera tracking results;

[0011] S5. Smooth the cross-camera tracking results to obtain smoothed tracking results;

[0012] S6. Perform visual perception based on the smooth tracking results to obtain the visual perception results.

[0013] Furthermore, the affine transformation matrix is:

[0014]

[0015] Among them, a 11 ,a 12 ,a 21 ,a 22 The parameter t is the linear part. x ,t y These are the parameters of the translation vector.

[0016] Furthermore, the specific steps of S2 are as follows:

[0017] The camera image at each location is input into the backbone network to obtain feature map F. Feature map F is processed by a multi-scale aggregation module to obtain fused features. The fused features are input into the classification and regression branches respectively. The classification and regression branches output detection boxes to obtain the target detection results.

[0018] Furthermore, the training process for the object detection model is as follows:

[0019] Construct an object detection network, and based on the set of predicted bounding boxes D = {d1, d2, ..., d...} output by the object detection network... n} and the true target set G = {g1, g2, ..., g} in the training set. m Maximizing the transfer matrix involves using the predicted bounding box obtained by maximizing the transfer matrix as the current predicted bounding box output by the object detection network. Specifically, maximizing the transfer matrix involves:

[0020]

[0021] O = SimOTA(D, G)

[0022] Where n is the number of predicted boxes in the predicted box set, m is the number of true targets, and O ij Let d represent an n×m matrix, if d i Assigned to g j Then O ij =1 otherwise 0, SimOTA represents the dynamic label assignment strategy, D represents the set of predicted bounding boxes output by the object detection network, d i Let g represent a prediction bounding box, and G represent the set of true targets in the training set. i Let S represent a real target, and S is also an m×n matrix, where S ij Represents the prediction box d i and the real target g i The matching score.

[0023] Furthermore, the loss function of the object detection model includes classification loss, regression loss, and objectness loss, where the classification loss is:

[0024]

[0025] in, For classification loss, N pred p represents the number of predicted boxes. i α and γ are adjustment parameters for the predicted class probabilities;

[0026] The regression loss is:

[0027]

[0028] in, Let b represent the regression loss, and b be the true bounding box. To predict the bounding box;

[0029] The objectness loss is:

[0030]

[0031] in, Represents the objectness loss, y i For real labels, The confidence level of the predicted object.

[0032] Furthermore, the specific steps of S3 are as follows:

[0033] 31) The set of detection boxes for the target detection results is divided into a set of high-confidence detection boxes. and low confidence detection box set And input it into the target tracking model from the set of confidence detection boxes. Select a current detection box d t,i Based on high-confidence detection box set and the target trajectory set T of the previous frame t-1 Construct a matching matrix, and define frame t-1, which is the set of target trajectories from the previous frame, as follows: N track This represents the number of tracked targets; each trajectory is calculated. With the current detection box d t,i IoU, to obtain the matching matrix :

[0034]

[0035] 32) Using the Hungarian algorithm in the matching matrix M track The optimal allocation is solved by matching high-confidence detection boxes with the target trajectories of the previous frame to obtain matching results. In these matching results, if no match is found for a target trajectory from the previous frame, it is considered that the trajectory may have disappeared; this trajectory is denoted as an unmatched trajectory. If no match is found for a detection box, it is considered a new target; this detection box is denoted as an unmatched detection box. If a match is found, the matching relationship between the successfully matched detection box and the target trajectory is as follows:

[0036] A track =Hungarian(M track )

[0037] A track It represents the matching relationship between the detection box and the target trajectory obtained by solving the Hungarian algorithm, representing a matching pair.

[0038] 33) Obtain the target trajectory set T of the current frame based on the matching results. t Specifically:

[0039] For successfully matched trajectories, update them with the corresponding detection boxes, and then update the trajectories. The position and status are updated to the detection box d. t,i Position and status;

[0040] For unmatched detection boxes, in the trajectory Create a new target trajectory in the process.

[0041] For unmatched trajectories, if there are multiple consecutive unmatched frames, then start from the trajectory Remove the trajectory from the current frame to obtain the target trajectory set T. t At the update time, repeat steps 31) to 33) to obtain high-confidence tracking results;

[0042] 34) Complementary matching of low-confidence detection boxes: For the set of low-confidence detection boxes... Repeat steps 31) to 33) with the previously unmatched trajectory set to obtain low-confidence tracking results. The low-confidence tracking results and high-confidence tracking results are then integrated into the target tracking results.

[0043] Furthermore, the specific steps of S4 are as follows:

[0044] 41) Based on the affine transformation matrix, the tracking results across cameras are unified into the global coordinate system to obtain a unified tracking result;

[0045] 42) Preset multiple prior regions. For targets within the prior regions in the unified tracking results, extract feature vectors. For target i in the prior region P... j eigenvectors f within i And the target k in the prior region P d eigenvectors f within k Calculate the cosine similarity of the feature vectors;

[0046] 43) Find the optimal target matching relationship A based on cosine similarity. reid ;

[0047] 44) Based on the target ID corresponding to the optimal target matching relationship, determine whether different IDs under different shots are the same target, and obtain the cross-camera tracking results.

[0048] Furthermore, the cosine similarity is:

[0049]

[0050] in, Represents the eigenvector f i and eigenvector f k The cosine similarity.

[0051] Furthermore, the specific steps of S5 are as follows:

[0052] For the cross-camera tracking results at time frame t, if a target exists, then:

[0053] The current position of each target in time frame t is smoothed using an exponentially weighted average method to obtain the position smoothing result;

[0054] The current motion direction angle of each target in time frame t is smoothed using an exponential weighted average method to obtain the smoothed motion direction angle result;

[0055] The velocity vector of each target in time frame t is smoothed by an exponential weighted average method to obtain the smoothed velocity vector result. The three smoothed results are used as the smoothed tracking result of the target.

[0056] If no target exists, the current position is estimated based on the velocity vector smoothing result of the previous frame as the position smoothing result.

[0057] Furthermore, the estimated current position is:

[0058]

[0059] in, Current position This is the position smoothing result of the previous frame. This is the smoothed result of the velocity vector from the previous frame.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] This invention establishes several key points at different locations within a traffic intersection and generates an affine transformation matrix using a fast intersection calibration algorithm. This matrix is ​​used to transform the images from four cameras into a unified bird's-eye view, achieving seamless fusion of multi-camera footage. Target detection and tracking are then performed. From a single camera's perspective, traffic targets are identified and tracked, acquiring their category information and coordinate positions within the image. The detected target information is subsequently mapped to a global coordinate system for further processing under a unified perspective. Cross-camera ID matching ensures the correct association of the same target across different camera perspectives. By analyzing target features and motion trajectories, consistent recognition across cameras is achieved, ensuring accurate matching of target information across multiple viewpoints. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the main steps of the present invention;

[0063] Figure 2 Example diagram for rapid intersection marking;

[0064] Figure 3 A framework diagram for target detection and tracking in traffic scenarios;

[0065] Figure 4 A schematic diagram for setting up a priori regions across shots;

[0066] Figure 5 This is a visual analysis example of collaborative perception at a multi-camera intersection. Detailed Implementation

[0067] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0068] This invention proposes a visual analysis method for collaborative perception at traffic intersections, primarily targeting complex traffic environments, especially intersections covered by multiple cameras. It aims to achieve efficient and accurate target detection and tracking, thereby comprehensively improving the monitoring and analysis capabilities of traffic intersections. Specifically, this invention first proposes a fast intersection calibration algorithm. By selecting key points at the intersection and combining them with affine transformation, it accurately obtains the intrinsic and extrinsic parameters of the cameras, and then integrates multiple cameras into a unified bird's-eye view (BEV), achieving cross-view target fusion and providing a unified and comprehensive visual foundation for subsequent analysis. Secondly, considering the special characteristics of traffic scenarios, this invention deeply optimizes the target detection and tracking algorithms, significantly improving target detection accuracy and the robustness of multi-target tracking. This effectively ensures the real-time performance and accuracy of the system in complex dynamic environments, thereby ensuring the stable operation of the system under changing traffic conditions. Furthermore, to address the issue of target identity consistency in multi-camera scenarios, this invention introduces a cross-camera ID matching algorithm. This algorithm can correctly associate targets from multiple perspectives, avoiding identity confusion caused by perspective transitions. Finally, to address the problem of temporary target loss or unstable trajectories, this invention designs trajectory smoothing and prediction modules. This module generates coherent target motion information through reasonable trajectory completion and prediction, providing more complete and detailed dynamic data support for traffic management. Finally, the optimized target is fed into downstream tasks related to the traffic environment for visual perception, further optimizing real-time performance and robustness to meet diverse practical needs. The modules of this invention work closely together to achieve comprehensive monitoring and precise analysis of complex traffic intersections. To adapt to edge computing scenarios, this invention features a lightweight algorithm design, enabling efficient operation on edge devices and ensuring real-time availability in traffic scenarios. By using high-performance, low-power edge devices such as the Jetson Nano, it provides excellent processing performance and stable operation in resource-constrained environments without relying on cloud computing, further enhancing the system's deployment flexibility and making it suitable for intelligent monitoring and management needs of various practical traffic intersections. Compared with existing advanced methods, this invention significantly improves target detection and tracking accuracy and cross-camera matching accuracy, offering advantages such as wider scene coverage and higher processing efficiency, providing more reliable technical support for real-time monitoring and intelligent management of traffic intersections.

[0069] A fast intersection calibration algorithm is adopted. By setting several key points at different locations at the intersection, an affine transformation matrix is ​​generated to achieve the fusion and alignment of perception results from multiple cameras under a unified bird's-eye view (BEV). Under a single camera view, the perception result is acquired through a target detection and tracking module and mapped to a global coordinate system. Subsequently, a cross-camera ID matching module and a trajectory smoothing and prediction module are used to generate a complete intersection perception result. The intersection visual analysis method includes a target detection and tracking module, a cross-camera ID matching module, and a trajectory smoothing and prediction module. The target detection and tracking module is used to acquire the coordinate position information of the target category under a single camera view. The cross-camera ID matching module is used to achieve accurate association of target identity in a multi-view environment. The trajectory smoothing and prediction module generates a coherent motion path by completing and optimizing the target trajectory. The flowchart of the method is as follows: Figure 1 As shown. The method includes the following steps:

[0070] S1. A fast intersection calibration algorithm is adopted. By setting several key points at different locations of traffic intersections, an affine transformation matrix is ​​generated, and camera images at different locations are acquired.

[0071] S2. For each camera image, input it into the target detection model to obtain the target detection result;

[0072] S3. The target detection results at each location are input into the target tracking model, and the target tracking results are output.

[0073] S4. Based on the affine transformation matrix, the target tracking results are transformed into a unified global coordinate system, and global cross-lens ID matching is performed to obtain cross-camera tracking results;

[0074] S5. Smooth the cross-camera tracking results to obtain smoothed tracking results;

[0075] S6. Perform visual perception based on the smooth tracking results to obtain the visual perception results.

[0076] The intersection visual analysis method also includes the following downstream task modules:

[0077] Color detection module: Identifies and classifies vehicle colors.

[0078] Vehicle model detection module: Classifies and identifies the vehicle models of passing vehicles.

[0079] License plate recognition module: Automatically extracts and identifies the license plate number of detected vehicles.

[0080] Anomaly Detection Module: Monitors and identifies abnormal situations at traffic intersections, including the detection of traffic accidents, spill events, and falling goods.

[0081] Lane detection module: Accurately detects the position and status of lane lines to assist in vehicle path planning.

[0082] Equipment and facility detection module: Identifies and locates equipment and facilities at traffic intersections, including traffic lights, signs, sidewalks, green belts, and lane markings.

[0083] These downstream task modules work closely with the target detection and tracking module, cross-camera ID matching module, and trajectory smoothing and prediction module corresponding to the method of this invention, ultimately achieving comprehensive monitoring and analysis of complex traffic intersections. Furthermore, all modules have been lightweight and optimized, enabling integration and efficient deployment on edge devices such as Jetson Nano, ensuring real-time and stable monitoring and analysis capabilities even in resource-constrained environments.

[0084] The process of placing key points at intersections and calibrating camera views in intersection marking includes the following steps:

[0085] 1) Scene initialization: In a real intersection environment, it is necessary to first select suitable key points as calibration references, obvious ground markings, and manually placed markers at predetermined 5-meter intervals to ensure coverage of the entire intersection area, especially within 10 meters in the lane, to provide comprehensive calibration basis and ensure clear visibility from different camera perspectives. Considering the differences in geometry and structure of different intersections, it may be necessary to adjust the number and spacing of key points appropriately according to the specific situation to ensure the accuracy and reliability of calibration.

[0086] 2) The connection between the camera and the real scene: After the physical arrangement of the key points is completed, the camera view is associated with the real scene. The pixel coordinates of the pre-arranged key points are identified one by one and manually selected. The pixel coordinates of the key points under each camera view are mapped one-to-one with the real world coordinates in the top view.

[0087] 3) Coordinate transformation: Based on the correspondence between camera image coordinates and real-world scene coordinates obtained in step 2, coordinate pairs for affine transformation are constructed. For example, at least three non-collinear key points are selected to ensure the uniqueness and stability of the transformation matrix. For each camera's viewpoint, a mask region is designed to select image areas with high detection confidence and close proximity to the camera. This enables fast and accurate calibration of multi-camera intersections and establishes an efficient transformation relationship between image coordinates and real-world scene coordinates.

[0088] The perception aspect of object detection specifically includes:

[0089] A traffic scene detection network employing a design that separates classification and regression tasks can detect various targets at traffic intersections in real time, such as vehicles, pedestrians, and cyclists. It assigns a high-confidence score to each detection box, significantly improving the detection accuracy and real-time performance of the multi-camera collaborative perception system. The optimized anchor point design is suitable for rapidly changing traffic scenes, enabling efficient vehicle and pedestrian detection. An advanced label assignment strategy helps accurately identify different types of traffic participants, improving the system's accuracy.

[0090] By utilizing the pre-trained large-scale visual classification network CSPDarknet as its backbone, high-level semantic features of images can be efficiently extracted, thereby enhancing perception capabilities in traffic scenes. Based on an IoU matching strategy, labels are dynamically assigned, enabling detection boxes to better match real traffic targets, especially in high-density traffic flows, reducing mismatches and overlapping detections. A detection method without non-maximum suppression (NMS-Free) is explored, further improving detection speed and real-time performance by directly outputting the final detection results.

[0091] The specific perception involved in target tracking is as follows:

[0092] This paper optimizes the combination of detection and tracking through innovative strategies, fully utilizing low-confidence detection and an efficient target association strategy to significantly improve tracking accuracy and robustness. Based on target detection boxes extracted from a traffic scene detection network, a simple yet effective multi-target tracking framework reduces complexity. An IoU matching strategy improves the tracking accuracy of vehicles and pedestrians, ensuring accurate identification and tracking in congested traffic scenarios. Low-confidence detection is handled to reduce target loss, ensuring stable monitoring under various weather and lighting conditions. An efficient association strategy is employed, combining a simple yet effective ranking method to match detection boxes with existing tracked targets. This strategy avoids complex feature matching calculations, improving speed and efficiency. Stable target tracking is ensured by prioritizing high-confidence detection and gradually introducing low-confidence detection.

[0093] In cross-camera ID matching, the results from each individual camera are first mapped to a unified global coordinate system. Then, by extracting prior information and enhancing features, targets across cameras are processed, and the Hungarian algorithm is used to optimize positional association, thereby achieving global cross-camera ID matching.

[0094] To address potential issues such as inaccurate target detection, abnormal jitter, or missing data from a global perspective, this module utilizes perception results from consecutive frames to filter out unstable situations and supplement missing data, achieving continuous and stable target perception and tracking. The trajectory smoothing and prediction module is characterized by establishing a stable dictionary containing information such as position, angle, and velocity for each target, addressing problems like inaccurate target detection, abnormal jitter, or missing data from a global perspective. By leveraging perception data from consecutive frames, and in the event of missing detection data, this module predicts and completes the target's current position and motion state based on historical trajectory information and motion models, ensuring stable perception and tracking in complex traffic environments.

[0095] The downstream vision task module is characterized by its ability to accurately perform key functions such as license plate recognition, equipment and facility detection, color classification, anomaly detection, vehicle model recognition, and lane line detection, based on the results of basic target detection. This module works efficiently with the target detection and tracking module, the cross-camera ID matching module, and the trajectory smoothing and prediction module to form a closed-loop perception and analysis system, thereby significantly improving the ability to comprehensively monitor, dynamically analyze, and accurately manage complex traffic intersections.

[0096] The method also includes:

[0097] 1) Performance evaluation of single-camera detection and tracking: The performance of a single camera in target detection and tracking is evaluated through a series of quantitative and qualitative indicators.

[0098] 2) Comprehensive performance evaluation of unified visual perception results at multi-camera intersections: The overall performance of multi-camera collaborative work is comprehensively considered, mainly including the accuracy of collaborative perception.

[0099] This invention first performs rapid calibration and view fusion. By setting several key points at different locations at traffic intersections, an affine transformation matrix is ​​generated using a rapid intersection calibration algorithm. This matrix is ​​used to transform the images from four cameras into a unified bird's-eye view, achieving seamless fusion of multi-camera footage. Next, target detection and tracking are performed. From a single camera's perspective, target detection and tracking identify and track traffic targets, acquiring their category information and coordinate positions in the image. The detected target information is then mapped to a global coordinate system for subsequent processing under a unified view. Cross-camera ID matching ensures the same target can be correctly associated from different camera perspectives. By analyzing target features and motion trajectories, consistent recognition across cameras is achieved, ensuring accurate matching of target information across multiple perspectives. Trajectory smoothing and prediction complete and optimize the target's motion trajectory, predicting the target's future position through algorithms, providing a more coherent and stable motion trajectory. Finally, the optimized target is fed into downstream tasks related to the traffic environment for visual perception, and real-time performance and robustness are further optimized to meet diverse practical needs.

[0100] The models in this invention are all lightweight and can be efficiently deployed on edge devices, such as Jetson Nano, ensuring real-time traffic monitoring and analysis even in resource-constrained environments.

[0101] Compared with the prior art, the present invention has the following beneficial effects:

[0102] I. This invention proposes a fast and accurate multi-camera calibration method. By uniformly arranging key points within the intersection and utilizing an affine transformation matrix, it achieves fast and accurate calibration of multi-camera images, reducing the time and manpower costs required by the traditional chessboard calibration method.

[0103] Second, this invention achieves high-precision target detection and tracking, significantly improving the accuracy and robustness of traffic scene detection and tracking, ensuring continuous and stable target tracking, and reducing false detections and missed detections.

[0104] Third, this invention possesses comprehensive multi-view data fusion capabilities. Through cross-lens ID matching, it achieves seamless fusion of target information from different camera perspectives, providing a unified global perspective perception. This comprehensive data fusion capability ensures accurate location and tracking of traffic targets under multi-camera collaboration, improving the coverage and accuracy of the overall perception system.

[0105] Fourth, this invention also has the advantage of scalability. It adopts a modular design structure, facilitating the rapid addition or updating of corresponding modules based on new visual perception targets or traffic scenario requirements, without requiring large-scale reconstruction of the entire model. Simultaneously, the model's training and optimization processes are highly automated, automatically adjusting based on new data feedback to further improve the model's adaptability and performance in different scenarios.

[0106] Fifth, this invention optimizes the system's robustness and stability. Through trajectory smoothing and prediction modules, it establishes stable optimizations for each target, including information such as position, angle, and velocity. Furthermore, it supplements missing data using historical trajectory information and motion models, effectively addressing issues such as inaccurate target detection, abnormal jitter, and data gaps. This mechanism ensures stable operation and continuous sensing capabilities in complex traffic environments, significantly improving system reliability.

[0107] VI. This invention employs a highly efficient comprehensive performance evaluation, which includes a comprehensive performance evaluation of the detection and tracking performance of a single camera and the unified visual perception results of multiple cameras at intersections, enabling real-time monitoring and optimization of system performance.

[0108] Furthermore, this invention significantly improves target detection and tracking accuracy and cross-camera matching accuracy, and has outstanding advantages such as wider scene coverage and higher processing efficiency, providing more reliable technical support for real-time monitoring and intelligent management of traffic intersections.

[0109] The workflow of the module corresponding to the method of this invention is as follows:

[0110] A rapid intersection calibration algorithm is employed, which generates an affine transformation matrix by setting several key points at different locations within the intersection, enabling the fusion of four camera feeds from a unified bird's-eye view. Under a single camera view, the perception results are acquired through a target detection and tracking module and mapped to a global coordinate system. Subsequently, a cross-camera ID matching module and a trajectory smoothing and prediction module are used to generate a complete intersection perception result. The intersection visual analysis method includes a target detection and tracking module, a cross-camera ID matching module, and a trajectory smoothing and prediction module. The target detection and tracking module acquires the target category coordinates under a single camera view; the cross-camera ID matching module ensures correct association of targets across multiple perspectives; and the trajectory smoothing and prediction module completes, optimizes, and predicts the target trajectory, providing a more coherent motion trajectory.

[0111] The specific steps performed by the above module are as follows:

[0112] S1. A fast intersection calibration algorithm is adopted. By setting several key points at different locations at traffic intersections, an affine transformation matrix is ​​generated to achieve the fusion of images from four cameras under a unified bird's eye view (BEV).

[0113] 11) Scene initialization: In a real intersection environment, select suitable key points as calibration references, such as obvious ground markings and manually placed markers. Arrange these key points at predetermined 5-meter intervals, ensuring coverage of the entire intersection area, especially within a 10-meter range of the lanes, to provide comprehensive calibration data. Ensure that the key points are clearly visible from different camera perspectives, and appropriately adjust the number and spacing of key points according to the geometry and structural differences of the intersection to guarantee the accuracy and reliability of the calibration.

[0114] 12) Associating the camera with the real scene: After completing the placement of key points, associate the camera viewpoint with the real scene. For each camera viewpoint, design a mask region, selecting image regions with high detection confidence and close to the camera. Identify and manually select the pixel coordinates (x, y) of key points in each camera image one by one. i y i ), and compare it with the real-world coordinates (X) in the top view. i Y i A one-to-one mapping is performed.

[0115]

[0116] Where (x, y) are points in the image coordinate system, (X, Y) are points in the real scene coordinate system, and a 11 a 12 a 21 a 22 t represents the linear part of the affine transformation matrix. x , t y These are the parameters of the translation vector.

[0117] 13) Coordinate transformation: Based on the correspondence between the camera image coordinates and the real scene coordinates obtained in step 12, construct coordinate pairs for affine transformation. Select at least three non-collinear keypoints. To ensure the uniqueness and stability of the transformation matrix. For the i-th point.

[0118]

[0119] The affine transformation parameters can be obtained, and then the transformation matrix can be constructed to realize the projection of image coordinates to real coordinates.

[0120] S2. Construct a target detection model for traffic scenes. Use a traffic scene detection network designed with classification and regression tasks separated. The optimized anchor point design is suitable for rapidly changing traffic scenes, achieving efficient vehicle and pedestrian detection.

[0121] 21) In this example, CSPDarknet is used as the backbone network to extract low-level and high-level features of the image. CSPDarknet improves the efficiency and effectiveness of feature extraction by optimizing the gradient information and feature flow of the network through Cross Stage Partial Networks (CSPNet). Assuming the input image is I, the feature map F is obtained after feature extraction by the backbone network:

[0122] F = CSPDarknet(I)

[0123] The feature pyramid is further enhanced using the Path Aggregation Network (PANet), achieving more effective feature fusion through bottom-up and top-down information flow:

[0124] P i =PANet(F i )

[0125] 22) Anchor-free center point prediction method is adopted. For each pixel, it is predicted whether it is the center of the target, and the size of the bounding box is directly regressed. The output is: {class probability, center point offset (Δx, Δy), bounding box size (w, h)}.

[0126] By separating the classification and regression tasks into different branches, the mutual interference between the two is avoided, thus improving detection performance. Let C and R be the outputs of the classification and regression branches, respectively:

[0127] C = Classification_Head(P i )

[0128] R = Regression_Head(P) i )

[0129] 23) SimOTA, a dynamic label allocation strategy based on optimal transport theory, dynamically assigns labels according to the matching degree between each predicted bounding box and the ground truth target, optimizing the matching of positive and negative samples during training. The set of predicted bounding boxes is D = {d1, d2, ..., d...}. n} and the true target set G = {g1, g2, ..., g m The total matching degree is maximized through the optimal transfer matrix T.

[0130]

[0131] T = SimOTA(D, G)

[0132] 24) The loss function consists of classification loss, regression loss, and objectness loss. In the decoupled header structure, classification and regression losses are calculated separately: classification loss (using Focal Loss), regression loss (GIoU), and objectness loss.

[0133]

[0134] Where, p i α and γ are the adjustment parameters for the predicted class probabilities.

[0135]

[0136] Where b is the true bounding box. To predict the bounding box.

[0137]

[0138] Among them, y i For real labels, The confidence level of the predicted object.

[0139] S3. Construct a target tracking model for traffic scenarios. Through innovative strategies, the combination of detection and tracking is optimized. It makes full use of low-confidence detection and efficient target association strategies, which significantly improves tracking accuracy and robustness.

[0140] 31) In this example, in frame t, a set of detection boxes is generated using the target detector:

[0141]

[0142] For high-confidence and low-confidence detection classification, each detection box d in frame t... t,i Includes bounding box coordinates Confidence Based on the confidence threshold τ, the detection boxes are classified into high-confidence and low-confidence categories. By combining high-confidence and low-confidence detections, it is ensured that targets such as vehicles and pedestrians can be continuously and accurately tracked even in high-density traffic and complex environments.

[0143]

[0144]

[0145] 32) High confidence matching, simple association strategy based on IoU. The optimization of the target tracking model and Hungarian algorithm in traffic scenarios enables the target tracking model in traffic scenarios to achieve real-time target tracking at high-traffic intersections, meeting the strict real-time requirements of intelligent transportation systems.

[0146]

[0147] A track =Hungarian(M track )

[0148] A track It represents the matching relationship between the detection box and the target trajectory obtained by solving the Hungarian algorithm, representing a matching pair. This indicates which detection boxes d t,i And which trajectories A matching relationship has been established. A track The results are directly used to update the target trajectory set T t .

[0149] Update to a new detection box, or remove unmatched targets.

[0150]

[0151] For successfully matched trajectories, update them with the corresponding detection boxes, and then update the trajectories. The position and status are updated to the detection box d. t,i The location and status of the target (e.g., the bounding box location information, category information, etc.).

[0152] 33) Perform IoU matching again and update the tracking status, creating new targets for high-confidence tracking. For low-confidence detection boxes that still meet the conditional matching, new tracking targets are created. In complex traffic scenarios such as fast-moving vehicles and frequent occlusion, the target tracking model can maintain stable target tracking through a flexible detection box association and update mechanism, improving the overall robustness and reliability of the system. Supplementary matching of low-confidence detection boxes is also implemented for the low-confidence detection box set. Repeat steps 31) to 33) for the previously unmatched trajectory set, attempting to add low-confidence detection boxes to the unmatched trajectories.

[0153] S4. A cross-camera ID matching module is constructed based on the perception results. First, the results from a single camera are mapped to a unified global coordinate system. Then, by extracting prior information and enhancing features, targets across cameras are processed, and the Hungarian algorithm is used to optimize positional association, thereby achieving global cross-camera ID matching.

[0154] 41) In this example, S1 obtains the camera calibration parameters P, and the coordinates of the center point of the single-camera detection box in the image are (x... i y i If the camera calibration parameter P is given, then the coordinates (X, Y) in the global coordinate system are... iY i It can be calculated using the following affine transformation formula:

[0155]

[0156] 42) Within the field of view boundary of each camera, multiple prior regions are pre-defined based on the geometric layout of the traffic intersection and driving routes. These prior regions are used to predict the possible locations of targets, reducing the search space for matching and improving matching efficiency. For targets within the prior regions, the pre-trained Re-Identification (ReID) model TransReID effectively extracts distinguishable features. Let target i be in prior region P. j The eigenvectors within are f i The target k is in the prior region R d The feature vector within (d represents 4 cameras with different viewing angles) is f. k Then the cosine similarity The calculation is as follows:

[0157]

[0158] 43) Based on the cosine similarity calculated in the previous step, construct a similarity matrix S for targets across cameras, where S mn In this context, n and m represent the number of targets detected by the two cameras in the current time frame, respectively. The optimal matching matrix A is then solved using the Hungarian algorithm. reid To maximize the overall similarity of matching pairs, determine the correspondence between targets across cameras, update the global ID of the target, and achieve consistent tracking across cameras.

[0159] S5. In response to potential issues such as inaccurate target detection, abnormal jitter, or missing data in the global view obtained from S4, the trajectory smoothing and prediction module can effectively address these problems in traffic intersection scenarios, ensuring continuous and stable perception and tracking of the multi-camera system in complex dynamic environments.

[0160] 51) For each target i in time frame t, its current position is The location smoothing process uses an exponentially weighted average method:

[0161]

[0162] α is a smoothing factor, 0 < α < 1, which controls the weight of the new position relative to historical positions. This is the current position after smoothing.

[0163] 52) For each target i in time frame t, its current motion direction angle is: The angle smoothing process also uses an exponentially weighted average.

[0164]

[0165] 53) The velocity vector of target i in time frame t is It can be calculated through positional changes:

[0166]

[0167] Where Δt is the time interval.

[0168] The speed smoothing process also uses an exponentially weighted average:

[0169]

[0170] 54) If target i is not detected in time frame t (i.e., data is missing), estimate the current position based on the smoothing velocity of the previous frame:

[0171]

[0172] S6. In traffic scenarios, there are various visual perception targets, such as license plate recognition, color detection, anomaly detection, lane line detection, equipment and facility detection, and vehicle type detection. Corresponding visual perception models for intersection cameras are constructed for these targets. These models are developed with lightweight and robust design principles to achieve efficient perception in complex scenarios.

[0173] 61) In terms of lightweight design, the computational and storage requirements of the model are significantly reduced through optimization of the model structure and algorithm logic, enabling it to run efficiently in environments with limited hardware resources. Specifically, the model employs depthwise separable convolution technology, which decomposes standard convolution into depthwise convolution and pointwise convolution, thereby reducing the number of convolution kernel parameters and effectively compressing model parameters. Simultaneously, model pruning techniques are used to remove neurons and connections from the model based on their impact on the results, reducing unnecessary intermediate computational steps and effectively avoiding performance bottlenecks caused by model bloat.

[0174] 62) Regarding robustness, this invention incorporates an adaptive illumination adjustment algorithm. This algorithm dynamically adjusts image contrast and brightness based on the image's brightness histogram information, automatically adapting to changes in image features under different lighting conditions, significantly improving target recognition accuracy in complex lighting scenarios. Secondly, an image denoising filtering algorithm is introduced. This algorithm, based on the analysis of image signals of different frequencies, removes high-frequency noise signals generated by severe weather, effectively filtering image noise caused by severe weather and ensuring perception accuracy under such conditions. Furthermore, this invention designs a target recognition algorithm based on contextual information. By analyzing the target's surrounding environment and its relationship with other targets, it makes reasonable inferences about occluded targets, enabling the model to accurately identify targets even when faced with complex target occlusion and image clutter caused by traffic congestion. This robustness ensures that the model maintains efficient perception capabilities in various complex scenarios, effectively compensating for the shortcomings of existing technologies in complex traffic scenarios and providing reliable, accurate, and efficient visual perception support for traffic management and monitoring.

[0175] S7. Evaluate the overall performance of the visual analysis method for multi-camera intersection collaborative perception in the complex traffic environment by combining the detection performance of a single camera and the unified visual perception results of multiple cameras at the intersection.

[0176] In this embodiment, the automatic evaluation metrics used include AP and its derived metrics AP. 50 AP 75 AP S AP M AP L It is a key indicator for evaluating the performance of target detection models. The manual evaluation indicators used are mainly based on tracking accuracy and target tracking precision.

[0177] 71) The detection performance of a single camera is comprehensively reflected by the combination of different IoU thresholds and target sizes, providing an important basis for model optimization and application.

[0178] 72) For unified visual perception at multi-camera intersections, manual evaluation metrics are used: Tracking accuracy: Overall tracking performance is evaluated by comprehensively considering detection errors (missed detections and false detections). Target tracking accuracy: This measures the matching accuracy of the bounding box position during tracking, reflecting the fine-grained accuracy of the tracking.

[0179] Corresponding to the steps above, Figure 2 Example diagram for rapid intersection marking; Figure 3 A framework diagram for target detection and tracking in traffic scenarios; Figure 4 A schematic diagram for setting up a priori regions across shots; Figure 5 This is a visual analysis example of collaborative perception at a multi-camera intersection.

[0180] To verify the performance of the method in this application, the following experiments were designed.

[0181] This embodiment is compared with other advanced object detection models on our roadside traffic dataset, and the specific details are as follows:

[0182] (1) YOLOv3, a target detection model based on multi-scale prediction capability; (2) EfficientDet, a target detection model using scalable architecture and compound scaling techniques; (3) PP-YOLOv2, a target detection model based on a more advanced backbone network optimized by YOLO; (4) YOLOv5, a highly flexible and efficient target detection model.

[0183] The specific experimental results are shown in Tables 1 and 2.

[0184] Table 1 Performance of different object detection models on roadside traffic dataset (%).

[0185]

[0186]

[0187] AP is the average precision that combines precision and recall in object detection. 50 AP 75 These are the AP values ​​when the crossover ratio (CUP) thresholds are 50% and 75%, respectively. S AP M AP L This refers to the average accuracy for detecting small, medium, and large target objects. Experimental results demonstrate that our method outperforms other models with higher vehicle detection accuracy on roadside traffic datasets.

[0188] Table 2. Human evaluation results of the invention in real-world scenarios.

[0189] direction Number of tests Actual vehicle passage accuracy Detection rate north 59 78 75％ 93％ South 71 84 80％ 84％ West 114 124 91％ 91％ East 191 210 91％ 91％

[0190] The table presents the results of a 30-minute manual evaluation of the proposed method at intersections in different directions (north, south, west, and east). The results fully demonstrate its effectiveness and reliability. Particularly noteworthy are the westbound and eastbound intersections, where both the accuracy and detection rate reached 91%. The data strongly illustrates that our method maintains high accuracy in vehicle detection even under complex traffic flow conditions, closely matching actual vehicle traffic patterns, thus fully demonstrating the stability and efficiency of the method in various traffic flow scenarios.

[0191] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A visual analysis method for collaborative perception at traffic intersections, characterized in that, The method includes the following steps: S1. A fast intersection calibration algorithm is adopted. By setting several key points at different locations of traffic intersections, an affine transformation matrix is ​​generated, and camera images at different locations are acquired. S2. For each camera image, input it into the target detection model to obtain the target detection result; S3. The target detection results at each location are input into the target tracking model, and the target tracking results are output. S4. Based on the affine transformation matrix, the target tracking results are transformed into a unified global coordinate system, and global cross-lens ID matching is performed to obtain cross-camera tracking results; S5. Smooth the cross-camera tracking results to obtain smoothed tracking results; S6. Perform visual perception based on the smooth tracking results to obtain the visual perception results; The specific steps of S5 are as follows: For the cross-camera tracking results at time frame t, if a target exists, then: The current position of each target in time frame t is smoothed using an exponentially weighted average method to obtain the position smoothing result; The current motion direction angle of each target in time frame t is smoothed using an exponential weighted average method to obtain the smoothed motion direction angle result; The velocity vector of each target in time frame t is smoothed by an exponential weighted average method to obtain the smoothed velocity vector result. The three smoothed results are used as the smoothed tracking result of the target. If no target exists, the current position is estimated based on the velocity vector smoothing result of the previous frame as the position smoothing result.

2. The visual analysis method for collaborative perception at traffic intersections according to claim 1, characterized in that, The affine transformation matrix is: Among them, a 11 ,a 12 ,a 21 ,a 22 The parameter t is the linear part. x ,t y These are the parameters of the translation vector.

3. The visual analysis method for collaborative perception at traffic intersections according to claim 1, characterized in that, The specific steps of S2 are as follows: The camera image at each location is input into the backbone network to obtain feature map F. Feature map F is processed by a multi-scale aggregation module to obtain fused features. The fused features are input into the classification and regression branches respectively. The classification and regression branches output detection boxes to obtain the target detection results.

4. A visual analysis method for collaborative perception at traffic intersections according to claim 3, characterized in that, The training process for the object detection model is as follows: Construct an object detection network, and based on the set of predicted bounding boxes D = {d1, d2, ..., d...} output by the object detection network... n } and the true target set G = {g1, g2, ..., g} in the training set. m Maximizing the transfer matrix involves using the predicted bounding box obtained by maximizing the transfer matrix as the current predicted bounding box output by the object detection network. Specifically, maximizing the transfer matrix involves: O = SimOTA(D, G) Where n is the number of predicted boxes in the predicted box set, m is the number of true targets, and O ij Let d represent an n×m matrix, if d i Assigned to g j Then O ij =1 otherwise 0, SimOTA represents the dynamic label assignment strategy, D represents the set of predicted bounding boxes output by the object detection network, d i Let g represent a prediction bounding box, and G represent the set of true targets in the training set. i Let S represent a real target, and S is also an m×n matrix, where S ij Represents the prediction box d i and the real target g i The matching score.

5. A visual analysis method for collaborative perception at traffic intersections according to claim 4, characterized in that, The loss function of the object detection model includes classification loss, regression loss, and objectness loss, where the classification loss is: in, For classification loss, N pred p represents the number of predicted boxes. i α and γ are adjustment parameters for the predicted class probabilities; The regression loss is: in, Let b represent the regression loss, and b be the true bounding box. To predict the bounding box; The objectness loss is: in, Represents the objectness loss, y i For real labels, Confidence level of the predicted object.

6. A visual analysis method for collaborative perception at traffic intersections according to claim 1, characterized in that, The specific steps for S3 are as follows: 31) The set of detection boxes for the target detection results is divided into a set of high-confidence detection boxes. and low confidence detection box set And input it into the target tracking model from the set of confidence detection boxes. Select a current detection box d t,i Based on high-confidence detection box set and the target trajectory set T of the previous frame t-1 Construct a matching matrix, and define frame t-1, which is the set of target trajectories in the previous frame, as T. t-1 = N track This represents the number of tracked targets; each trajectory is calculated. With the current detection box d t,i IoU, to obtain the matching matrix 32) Using the Hungarian algorithm in the matching matrix M track The optimal allocation is solved by matching high-confidence detection boxes with the target trajectories of the previous frame to obtain matching results. In these matching results, if no match is found for a target trajectory from the previous frame, it is considered that the trajectory may have disappeared; this trajectory is denoted as an unmatched trajectory. If no match is found for a detection box, it is considered a new target; this detection box is denoted as an unmatched detection box. If a match is found, the matching relationship between the successfully matched detection box and the target trajectory is as follows: A track =Hungarian(M track ) A track It represents the matching relationship between the detection box and the target trajectory obtained by solving the Hungarian algorithm, representing a matching pair. 33) Obtain the target trajectory set T of the current frame based on the matching results. t Specifically: For successfully matched trajectories, update them with the corresponding detection boxes, and then update the trajectories. The position and status are updated to the detection box d. t,i Position and status; For unmatched detection boxes, in the trajectory Create a new target trajectory in the process. For unmatched trajectories, if there are multiple consecutive unmatched frames, then start from the trajectory Remove the trajectory from the current frame to obtain the target trajectory set T. t At the update time, repeat steps 31) to 33) to obtain high-confidence tracking results; 34) Complementary matching of low-confidence detection boxes: For the set of low-confidence detection boxes... Repeat steps 31) to 33) with the previously unmatched trajectory set to obtain low-confidence tracking results. The low-confidence tracking results and high-confidence tracking results are then integrated into the target tracking results.

7. A visual analysis method for collaborative perception at traffic intersections according to claim 1, characterized in that, The specific steps of S4 are as follows: 41) Based on the affine transformation matrix, the tracking results across cameras are unified into the global coordinate system to obtain a unified tracking result; 42) Preset multiple prior regions. For targets within the prior regions in the unified tracking results, extract feature vectors. For target i in the prior region P... j eigenvectors f within i And the target k in the prior region P d eigenvectors f within k Calculate the cosine similarity of the feature vectors; 43) Find the optimal target matching relationship A based on cosine similarity. reid ; 44) Based on the target ID corresponding to the optimal target matching relationship, determine whether different IDs under different shots are the same target, and obtain the cross-camera tracking results.

8. A visual analysis method for collaborative perception at traffic intersections according to claim 7, characterized in that, The cosine similarity is: in, Represents the eigenvector f i and eigenvector f k The cosine similarity.

9. A visual analysis method for collaborative perception at traffic intersections according to claim 1, characterized in that, The estimated current position is: in, Current position This is the position smoothing result of the previous frame. This is the smoothed result of the velocity vector from the previous frame.

Images