Visual analysis method for traffic intersection collaborative perception

Abstract

The invention relates to a visual analysis method for traffic intersection collaborative perception, and the method comprises the following steps: S1, employing an intersection rapid calibration algorithm, and obtaining camera images at different positions; s2, for the camera image of each position, inputting a target detection model to obtain a target detection result; s3, inputting the target detection result of each position into a target tracking model, and outputting a target tracking result; s4, converting the target tracking result into a unified global coordinate system based on the affine transformation matrix, and executing global cross-lens ID matching to obtain a cross-camera tracking result; s5, smoothing the cross-camera tracking result to obtain a smooth tracking result; and S6, performing visual perception based on the smooth tracking result to obtain a visual perception result. Compared with the prior art, the method has the advantages of ensuring that the same target can be correctly associated under different camera visual angles, realizing cross-camera consistency identification and the like.

Description

Technical Field

[0001] The present invention relates to the field of traffic scene perception, and in particular to a visual analysis method for collaborative perception at traffic intersections. Background Art

[0002] With the continuous expansion of urban traffic and the increasing complexity of traffic flow, the importance of intelligent transportation systems in improving traffic management efficiency and ensuring road safety has become increasingly prominent. As a key node in the traffic network, traffic intersections have complex traffic environments and changeable traffic behaviors that place higher demands on perception technology. Traditional single-camera monitoring systems have many shortcomings in terms of coverage, viewing angle limitations, and multi-target recognition and tracking, making it difficult to fully and accurately capture the dynamic changes of intersections. To meet these challenges, multi-camera collaborative perception technology has emerged. By deploying multiple cameras at different locations at a traffic intersection, all-round monitoring of the entire intersection can be achieved.

[0003] Visual analysis of multi-camera intersection collaborative perception needs to solve two major difficulties: (1) rapid calibration of multiple cameras, and (2) perception fusion and cross-lens target association. For the first difficulty, although traditional calibration methods, such as the chessboard calibration method, can provide high calibration accuracy, they face many challenges in practical applications. Specifically, the chessboard calibration method requires the precise placement of a standard chessboard pattern within the camera field of view and ensures that it can be clearly identified from the perspectives of multiple cameras. This is often difficult to achieve in complex traffic intersection environments. Traditional chessboard calibration methods often require a long calibration time and are difficult to meet the needs of rapid deployment and dynamic adjustment. For the second difficulty, perception fusion and cross-lens target association, currently, cross-camera perception solutions for traffic intersections are relatively limited, and traditional methods have significant deficiencies in perception fusion and cross-lens target association. The large difference in perspectives between different cameras makes it difficult to maintain cross-perspective consistency, which in turn affects the accuracy and stability of target association. It performs poorly in dealing with feature consistency issues under cross-perspectives, making it difficult to achieve efficient and reliable target tracking and recognition. This not only restricts the perception capabilities of multi-camera systems, but also limits their application potential in complex traffic environments. Summary of the invention

[0004] The purpose of the present invention is to improve the global consistency of multi-camera collaborative perception, realize fast calibration and perspective fusion, ensure that the same target can be correctly associated under different camera perspectives, and realize consistent recognition across cameras, thereby providing a visual analysis method for collaborative perception at traffic intersections.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A visual analysis method for collaborative perception of traffic intersections, the method comprising the following steps:

[0007] S1, using the intersection fast calibration algorithm, by setting a number of key points at different positions of the traffic intersection, generating an affine transformation matrix, and obtaining camera images at different positions;

[0008] S2. For the camera image at each position, input the target detection model to obtain the target detection result;

[0009] S3, the target detection result at each position is input into the target tracking model, and the target tracking result is output;

[0010] S4, transforming the target tracking result into a unified global coordinate system based on the affine transformation matrix, performing global cross-lens ID matching, and obtaining a cross-camera tracking result;

[0011] S5, performing smoothing processing on the cross-camera tracking result to obtain a smooth tracking result;

[0012] S6. Perform visual perception based on the smooth tracking result to obtain a visual perception result.

[0013] Furthermore, the affine transformation matrix is:

[0014]

[0015] Among them, a 11 ,a 12 ,a 21 ,a 22 is the linear part parameter, t x ,t y is the parameter of the translation vector.

[0016] Furthermore, the specific steps of S2 are:

[0017] The camera image at each position is input into the backbone network to obtain the feature map F. The feature map F passes through the multi-scale aggregation module to obtain the fusion feature. The fusion feature is input into the classification and regression branches respectively. The classification and regression branches output the detection box to obtain the target detection result.

[0018] Furthermore, the training process of the target detection model is:

[0019] Construct a target detection network, based on the prediction box set D = {d1, d2, ..., d n} and the true target set G in the training set = {g1,g2,…,g m}Maximize the transfer matrix, and use the predicted box under the maximized transfer matrix as the current predicted box output by the target detection network. The maximized transfer matrix is ​​specifically:

[0020]

[0021] O = SimOTA(D, G)

[0022] Among them, n is the number of prediction boxes in the prediction box set, m is the number of real targets, O ij represents an n×m matrix, if d i Assigned to g j , then O ij =1, otherwise 0, SimOTA represents the dynamic label assignment strategy, D represents the set of prediction boxes output by the target detection network, d i represents a prediction box, G represents the real target set in the training set, and g i represents a real target, S is also an m×n matrix, where S ij Denotes the predicted box d i and the real target g i The matching score.

[0023] Furthermore, the loss function of the target detection model includes classification loss, regression loss and objectness loss, where the classification loss is:

[0024]

[0025] in, is the classification loss, N pred represents the number of prediction boxes, p i is the predicted class probability, α and γ are adjustment parameters;

[0026] The regression loss is:

[0027]

[0028] in, represents the regression loss, b is the true bounding box, To predict the bounding box;

[0029] The objectness loss is:

[0030]

[0031] in, Represents objectness loss, y i is the true label, is the confidence of the predicted object.

[0032] Furthermore, the specific steps of S3 are:

[0033] 31) The detection box set of the target detection result is divided into a high confidence detection box set And a set of low confidence detection boxes And input into the target tracking model, from the confidence detection box set Select a current detection box d t,i , based on a set of high confidence detection boxes And the target trajectory set T of the previous frame t-1 Construct the matching matrix and define the t-1 frame, that is, the target trajectory set of the previous frame as N track Represents the number of tracked targets and calculates each trajectory With the current detection box d t,i IoU, get the matching matrix :

[0034]

[0035] 32) Use the Hungarian algorithm to match the matrix M track Solve the optimal allocation above, match the high confidence detection frame with the target track of the previous frame, and obtain the matching result. In the matching result, if a target track of the previous frame is not matched, it is considered that the track may disappear, and the track is set as an unmatched track. If a detection frame is not matched to the track, it is considered to be a new target, and the detection frame is set as an unmatched detection frame. If a match is found, the matching relationship between the successfully matched detection frame and the target track is:

[0036] A track =Hungarian(M track )

[0037] A track It is the matching relationship between the detection box and the target trajectory obtained by the Hungarian algorithm, representing a matching pair

[0038] 33) Based on the matching results, the current frame target trajectory set T is obtained t , specifically:

[0039] For the successfully matched trajectory, update it to the corresponding detection frame and The position and state of the detection box d t,i location and status;

[0040] For unmatched detection boxes, in the trajectory Create a new target trajectory in

[0041] For unmatched trajectories, if multiple consecutive frames are not matched, Remove the trajectory from the current frame to obtain the target trajectory set T t , update the time, repeat 31)~33) to obtain a high confidence tracking result;

[0042] 34) Supplementary matching of low confidence detection boxes, for low confidence detection box sets The operations of 31) to 33) are repeated for the previously unmatched trajectory set to obtain a low-confidence tracking result. The low-confidence tracking result and the high-confidence tracking result are integrated into the target tracking result.

[0043] Furthermore, the specific steps of S4 are:

[0044] 41) Unifying the cross-camera tracking results into the global coordinate system based on the affine transformation matrix to obtain a unified tracking result;

[0045] 42) Preset multiple prior regions, extract feature vectors for targets in the prior regions in the unified tracking results, and extract feature vectors for target i in the prior region P. j The eigenvector f i and target k in the prior region P d The eigenvector f k Calculate the cosine similarity of feature vectors;

[0046] 43) Find the optimal target matching relationship A based on cosine similarity reid ;

[0047] 44) Based on the target ID corresponding to the optimal target matching relationship, determine whether different IDs under cross-lens conditions are the same target, and obtain the cross-camera tracking result.

[0048] Furthermore, the cosine similarity is:

[0049]

[0050] in, Denotes the eigenvector f i and the eigenvector f k The cosine similarity of .

[0051] Furthermore, the specific steps of S5 are:

[0052] For the cross-camera tracking results at time frame t, if there is a target, then:

[0053] The current position of each target in time frame t is smoothed using the exponential weighted average method to obtain a position smoothing result;

[0054] The current moving direction angle of each target in time frame t is smoothed using an exponential weighted average method to obtain a moving direction angle smoothing result;

[0055] The velocity vector of each target in time frame t is smoothed by using the exponential weighted average method to obtain the velocity vector smoothing result. The three smoothing results are used as the smooth tracking results of the target.

[0056] If there is no target, the current position is estimated based on the velocity vector smoothing result of the previous frame as the position smoothing result.

[0057] Furthermore, the estimated current position is:

[0058]

[0059] in, is the current location, is the position smoothing result of the previous frame, The velocity vector smoothing result for the previous frame.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention sets a number of key points at different positions of the traffic intersection and uses the intersection fast calibration algorithm to generate an affine transformation matrix. This matrix is ​​used to convert the images of four cameras to a unified bird's-eye view to achieve seamless fusion of multi-camera images. Then, target detection and tracking are performed to identify and track traffic targets in the perspective of a single camera, and obtain their category information and coordinate positions in the image. The detected target information is then mapped to the global coordinate system for subsequent processing in a unified perspective. Cross-lens ID matching is used to ensure that the same target can be correctly associated in different camera perspectives. By analyzing the characteristics and motion trajectory of the target, consistent recognition across cameras is achieved to ensure accurate matching of target information in multiple perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the main steps of the present invention;

[0063] Figure 2 This is an example of a fast intersection calibration diagram;

[0064] Figure 3 A diagram of the framework for target detection and tracking in traffic scenes;

[0065] Figure 4 Schematic diagram for setting the cross-lens prior region;

[0066] Figure 5 This is an example of visual analysis for collaborative perception at multi-camera intersections. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0068] The present invention proposes a visual analysis method for collaborative perception of traffic intersections, which is mainly aimed at complex traffic environments, especially intersection scenes covered by multiple cameras, and is committed to achieving efficient and accurate target detection and tracking, so as to comprehensively improve the monitoring and analysis capabilities of traffic intersections. Specifically, the present invention first proposes a fast intersection calibration algorithm, which accurately obtains the internal and external parameters of the camera by selecting key points of the intersection and combining the affine transformation method, and then integrates multiple cameras into a unified bird's-eye view (BEV), realizes cross-view target fusion, and provides a unified and comprehensive visual basis for subsequent analysis. Secondly, considering the particularity of traffic scenes, the present invention deeply optimizes the target detection and tracking algorithm, greatly improves the target detection accuracy and the robustness of multi-target tracking, effectively guarantees the real-time and accuracy of the system in complex dynamic environments, thereby ensuring the stable operation of the system under variable traffic conditions. In addition, in order to solve the problem of target identity consistency in multi-camera scenes, the present invention introduces a cross-lens ID matching algorithm, which can realize the correct association of multi-view targets and avoid the identity confusion problem caused by perspective conversion. In addition, in order to solve the problem of temporary loss of targets or unstable trajectories, the present invention designs a trajectory smoothing and prediction module. The module generates coherent target motion information through reasonable trajectory completion and prediction, providing more complete and detailed dynamic data support for traffic management. Finally, the optimized target is sent to the downstream tasks related to the traffic environment to realize visual perception, and further optimizes the real-time performance and robustness to meet the diverse practical needs. The modules of the present invention work closely together to finally realize the all-round monitoring and precise analysis of complex traffic intersections. In order to adapt to the edge computing scenario, the present invention particularly optimizes the lightweight design of the algorithm, which can run efficiently on the edge device to ensure the real-time availability in the traffic scenario. By using high-performance and low-power edge devices such as Jetson Nano, it can still provide excellent processing performance and stable operation guarantee in a resource-limited environment without relying on cloud computing, further improving the deployment flexibility of the system, and is suitable for the intelligent monitoring and management needs of various actual traffic intersections. Compared with the existing advanced methods, the present invention has significantly improved the accuracy of target detection and tracking, and the accuracy of cross-lens matching, and has outstanding advantages such as wider scene coverage and higher processing efficiency, providing more reliable technical support for real-time monitoring and intelligent management of traffic intersections.

[0069] By adopting the fast intersection calibration algorithm, several key points are set at different positions of the traffic intersection to generate an affine transformation matrix to achieve the fusion and alignment of the perception results of multiple cameras under a unified bird's-eye view (BEV). Under the perspective of a single camera, the perception results are obtained through the target detection and tracking module and mapped to the global coordinate system. Subsequently, the cross-lens ID matching module and the trajectory smoothing and prediction module are used to generate complete intersection perception results. The intersection visual analysis method includes a target detection and tracking module, a cross-lens ID matching module, and a trajectory smoothing and prediction module. The target detection and tracking module is used to obtain the coordinate position information of the target category under a single camera. The cross-lens ID matching module is used to achieve accurate association of the target identity in a multi-perspective environment. The trajectory smoothing and prediction module generates a coherent motion path by completing and optimizing the target trajectory. The flow chart of the method is shown as follows. Figure 1 The method comprises the following steps:

[0070] S1, using the intersection fast calibration algorithm, by setting a number of key points at different positions of the traffic intersection, generating an affine transformation matrix, and obtaining camera images at different positions;

[0071] S2. For the camera image at each position, input the target detection model to obtain the target detection result;

[0072] S3, the target detection result at each position is input into the target tracking model, and the target tracking result is output;

[0073] S4, transforming the target tracking result into a unified global coordinate system based on the affine transformation matrix, performing global cross-lens ID matching, and obtaining a cross-camera tracking result;

[0074] S5, performing smoothing processing on the cross-camera tracking result to obtain a smooth tracking result;

[0075] S6. Perform visual perception based on the smooth tracking result to obtain a visual perception result.

[0076] The intersection visual analysis method also includes the following downstream task modules:

[0077] Color Detection Module: Identifies and classifies the color of the vehicle.

[0078] Vehicle type detection module: classifies and identifies the vehicle types of passing vehicles.

[0079] License plate recognition module: automatically extracts and identifies the license plate number of the detected vehicle.

[0080] Anomaly Detection Module: Monitors and identifies abnormal situations at traffic intersections, including detection of traffic accidents, spillage events, and cargo drops.

[0081] Lane line detection module: accurately detects the position and status of lane lines to assist vehicle driving path planning.

[0082] Equipment and facility detection module: Identify and locate equipment and facilities at traffic intersections, including traffic lights, signs, sidewalks, green belts and lane lines.

[0083] These downstream task modules work closely with the target detection and tracking module, cross-lens ID matching module, and trajectory smoothing and prediction module corresponding to the method of the present invention, and finally achieve all-round monitoring and analysis of complex traffic intersections. In addition, the modules are all lightweight and optimized, and can be integrated and efficiently deployed on edge devices such as Jetson Nano, ensuring that real-time and stable monitoring and analysis capabilities can still be provided in resource-constrained environments.

[0084] The process of placing key points at intersections and calibrating camera view angles in intersection calibration includes the following steps:

[0085] 1) Scene initialization. In a real intersection environment, we first need to select key points suitable for calibration reference, obvious ground markings, and manually placed points at a predetermined interval of 5 meters to ensure that the entire intersection area can be covered, especially within the 10-meter range of the lane, to provide a comprehensive calibration basis and ensure that it can be clearly seen from different camera perspectives. Considering the geometric shapes and structural differences of different intersections, the number and spacing of key points may need to be appropriately adjusted according to the specific situation to ensure the accuracy and reliability of calibration;

[0086] 2) Association between the camera and the real scene. After completing the physical arrangement of the key points, the camera view is associated with the real scene. The pixel coordinates of the pre-arranged key points are identified and manually selected one by one. The pixel coordinates of the key points under each camera view are mapped one by one to the real world coordinates in the top view.

[0087] 3) Coordinate transformation: Based on the correspondence between the camera image coordinates obtained in the second step and the real scene coordinates, construct coordinate pairs for affine transformation. For example, select at least three non-collinear key points to ensure the uniqueness and stability of the transformation matrix. For each camera’s viewing angle, design a mask area to select image areas with high detection confidence and close to the camera. This enables fast and accurate calibration of multi-camera intersections and establishes an efficient conversion relationship between image coordinates and real scene coordinates.

[0088] The perception of target detection is specifically:

[0089] The traffic scene detection network designed with classification and regression tasks separated can detect various types of targets at traffic intersections in real time, such as vehicles, pedestrians, cyclists, etc., and assign high confidence scores to each detection frame, significantly improving the detection accuracy and real-time performance of the multi-camera collaborative perception system. The optimized anchor point design is suitable for rapidly changing traffic scenes and achieves efficient vehicle and pedestrian detection. The advanced label allocation strategy helps to accurately identify different types of traffic participants and improve the accuracy of the system.

[0090] Using the pre-trained large-scale visual classification network CSPDarknet as its backbone network, it can efficiently extract high-level semantic features of images, thereby enhancing perception capabilities in traffic scenes. Based on the IoU matching strategy, labels are dynamically assigned so that the detection box can better match the real traffic target, especially in high-density traffic flow, reducing mismatches and overlapping detections. The detection method without non-maximum suppression (NMS-Free) is explored, and the detection speed and real-time performance are further improved by directly outputting the final detection results.

[0091] The perception of target tracking is as follows:

[0092] The combination of detection and tracking is optimized through innovative strategies, making full use of low-confidence detections and efficient target association strategies, significantly improving tracking accuracy and robustness. Based on the target detection box extracted by the traffic scene detection network, the complexity is reduced through a simple but effective multi-target tracking framework. The tracking accuracy of vehicles and pedestrians is improved using the IoU matching strategy to ensure accurate identification and tracking in crowded traffic scenes. Low-confidence detections are handled to reduce target loss and ensure stable monitoring in various weather and lighting conditions. An efficient association strategy is used, combined with a simple but effective sorting method to match the detection box with the existing tracking target. This strategy avoids complex feature matching calculations and improves speed and efficiency. Stable tracking of targets is ensured by prioritizing high-confidence detections and gradually introducing low-confidence detections.

[0093] In cross-camera ID matching, the results of a single camera are first mapped to a unified global coordinate system. Then, by extracting prior information and enhancing features, the cross-camera targets are processed, and the position association is optimized using the Hungarian algorithm, thereby achieving global cross-camera ID matching. By extracting prior information and enhancing features, the cross-camera targets are processed, and the position association is optimized using the Hungarian algorithm, thereby achieving global cross-camera ID matching.

[0094] In view of the inaccurate detection, abnormal jitter or missing of the target from a global perspective, the unstable situation is filtered out and the missing data is supplemented by using the perception results of the previous and next frames to achieve continuous and stable perception and tracking of the target. The characteristic of the trajectory smoothing and prediction module is that it establishes a stable dictionary containing information such as position, angle and speed for each target in view of the inaccurate detection, abnormal jitter or missing data of the target from a global perspective. This module uses the perception data of the previous and next frames to predict and complete the current position and motion state of the target based on the historical trajectory information and motion model in the case of missing detection data, ensuring stable perception and tracking in complex traffic environments.

[0095] The characteristic of the visual downstream task module is that it relies on the basic target detection results to accurately realize the key functions of license plate recognition, equipment and facility detection, color classification, anomaly monitoring, vehicle model recognition and lane line detection. This module works efficiently with the target detection and tracking module, the cross-lens ID matching module and the trajectory smoothing and prediction module to form a closed-loop perception and analysis system, thereby significantly improving the all-round monitoring, dynamic analysis and precise management capabilities of complex traffic intersections.

[0096] The method also includes:

[0097] 1) Single-camera detection and tracking performance evaluation: A series of quantitative and qualitative indicators are used to evaluate the performance of a single camera in target detection and tracking.

[0098] 2) Comprehensive performance evaluation of unified visual perception results of multi-camera intersections: Comprehensively consider the overall performance of multiple cameras working together, mainly including the accuracy of collaborative perception.

[0099] The present invention first performs rapid calibration and perspective fusion. By setting several key points at different positions of the traffic intersection, an affine transformation matrix is ​​generated using the intersection rapid calibration algorithm. This matrix is ​​used to convert the images of four cameras to a unified bird's-eye view to achieve seamless fusion of multi-camera images. Then target detection and tracking are performed. Under the perspective of a single camera, target detection and tracking are responsible for identifying and tracking traffic targets, obtaining their category information and coordinate positions in the image. The detected target information is then mapped to a global coordinate system for subsequent processing under a unified perspective. Cross-lens ID matching is used to ensure that the same target can be correctly associated under different camera perspectives. By analyzing the characteristics and motion trajectory of the target, consistent recognition across cameras is achieved to ensure accurate matching of target information under multiple perspectives. Trajectory smoothing and prediction complement and optimize the motion trajectory of the target, and the future position of the target is predicted by an algorithm to provide a more coherent and stable motion trajectory. Finally, the optimized target is sent to downstream tasks related to the traffic environment to achieve visual perception, and further optimizes real-time performance and robustness to meet diverse practical needs.

[0100] The models in this invention are all lightweight in design and can be efficiently deployed on edge devices, such as JetsonNano, to ensure real-time traffic monitoring and analysis in resource-constrained environments.

[0101] Compared with the prior art, the present invention has the following beneficial effects:

[0102] 1. The present invention proposes a fast and accurate multi-camera calibration, which achieves fast and accurate calibration of multi-camera images by evenly arranging key points in the intersection and using affine transformation matrix, reducing the time and labor cost required by the traditional chessboard calibration method.

[0103] 2. The present invention realizes high-precision target detection and tracking, greatly improves the accuracy and robustness of traffic scene detection and tracking, ensures continuous and stable tracking of targets, and reduces false detections and missed detections.

[0104] 3. The comprehensive multi-view data fusion capability of the present invention achieves seamless fusion of target information from different camera perspectives through cross-lens ID matching, providing unified global perspective perception. This comprehensive data fusion capability ensures that traffic targets can be accurately located and tracked under the coordination of multiple cameras, improving the coverage and accuracy of the overall perception system.

[0105] Fourth, the present invention also has the advantage of scalability. It adopts a modular design structure, which facilitates the rapid addition or update of corresponding modules according to new visual perception goals or traffic scene requirements, without the need for large-scale reconstruction of the entire model. At the same time, the training and optimization process of the model has a high degree of automation, and can automatically adjust according to new data feedback, further improving the adaptability and performance of the model in different scenarios.

[0106] 5. The present invention optimizes the robustness and stability of the system. Through the trajectory smoothing and prediction module, a stable optimization including position, angle and speed information is established for each target. The missing data is supplemented through historical trajectory information and motion models, effectively dealing with problems such as inaccurate target detection, abnormal jitter and data missing. This mechanism ensures the stable operation and continuous perception capability of the system in complex traffic environments, significantly improving the reliability of the system.

[0107] 6. The present invention adopts efficient comprehensive performance evaluation, which includes comprehensive performance evaluation of the detection and tracking performance of a single camera and the unified visual perception results of multiple camera intersections, and can monitor and optimize system performance in real time.

[0108] In addition, the present invention has significant improvements in target detection and tracking accuracy, cross-lens matching accuracy, etc., and has outstanding advantages such as wider scene coverage and higher processing efficiency, providing more reliable technical support for real-time monitoring and intelligent management of traffic intersections.

[0109] The workflow of the module corresponding to the method of the present invention is:

[0110] By adopting the fast intersection calibration algorithm, a number of key points are set at different positions of the traffic intersection to generate an affine transformation matrix to achieve the fusion of the four-way camera images under a unified bird's-eye view. Under the perspective of a single camera, the perception results are obtained through the target detection and tracking module and mapped to the global coordinate system. Subsequently, the cross-lens ID matching module and the trajectory smoothing and prediction module are used to generate a complete intersection perception result. The intersection visual analysis method includes a target detection and tracking module, a cross-lens ID matching module, and a trajectory smoothing and prediction module. The target detection and tracking module is used to obtain the target category coordinate position information under a single camera, the cross-lens ID matching module is used to ensure that the target can be correctly associated under multiple perspectives, and the trajectory smoothing and prediction module is used to complete, optimize and predict the target trajectory to provide a more coherent motion trajectory.

[0111] The specific steps performed by the above modules are:

[0112] S1. Adopting the fast intersection calibration algorithm, by setting several key points at different positions of the traffic intersection, an affine transformation matrix is ​​generated to achieve the fusion of the four-way camera images under the unified bird's eye view (BEV).

[0113] 11) Scene initialization: In the real intersection environment, select key points suitable for calibration reference, such as obvious ground markings and manually placed marking points. Arrange these key points at a predetermined interval of 5 meters to ensure coverage of the entire intersection area, especially within the 10-meter range of the lane, to provide a comprehensive calibration basis. Ensure that the key points are clearly visible from different camera perspectives, and adjust the number and spacing of key points appropriately according to the geometric shape and structural differences of the intersection to ensure the accuracy and reliability of the calibration.

[0114] 12) Association between the camera and the real scene. After completing the arrangement of key points, associate the camera view with the real scene. For each camera view, design a mask area and select the image area with high detection confidence and close to the camera. Identify and manually select the pixel coordinates (x i ,y i ) and compare it to the real-world coordinate (X i , Y i ) for one-to-one mapping.

[0115]

[0116] Where (x, y) is the point in the image coordinate system, (X, Y) is the point in the real scene coordinate system, and a 11 , a 12 , a 21 , a 22 is the linear part parameter of the affine transformation matrix. x , t y is the parameter of the translation vector.

[0117] 13) Coordinate transformation: Based on the correspondence between the camera image coordinates obtained in step 12 and the real scene coordinates, construct a coordinate pair for affine transformation. Select at least three non-collinear key points. To ensure the uniqueness and stability of the transformation matrix.

[0118]

[0119] The affine transformation parameters can be obtained, and then the transformation matrix can be constructed to realize the projection of image coordinates to real coordinates.

[0120] S2. Build a target detection model for traffic scenes, use a traffic scene detection network designed with classification and regression tasks separated, and optimize the anchor point design to be suitable for rapidly changing traffic scenes to achieve efficient vehicle and pedestrian detection.

[0121] 21) In this example, CSPDarknet is used as the backbone network to extract low-level and high-level features of the image. CSPDarknet optimizes the gradient information and feature flow of the network through the Cross Stage Partial Networks (CSPNet), thereby improving the efficiency and effect of feature extraction. Assuming that the input image is I, the feature map F is obtained after the backbone network extracts features:

[0122] F=CSPDarknet(I)

[0123] Using the Path Aggregation Network (PANet), the feature pyramid is further enhanced to achieve more effective feature fusion through bottom-up and top-down information flow:

[0124] P i =PANet(F i )

[0125] 22) Anchor-Free center point prediction method is used. Each pixel is predicted to be the center of the target, and the size of the bounding box is directly regressed. The output = {category probability, center point offset (Δx, Δy), bounding box size (w, h).

[0126] Separate the classification and regression tasks into different branches to avoid mutual interference between the two and improve the detection performance. Let C and R be the outputs of the classification and regression branches respectively:

[0127] C=Classification_Head(P i )

[0128] R = Regression_Head (P i )

[0129] 23) SimOTA, a dynamic label assignment strategy based on optimal transmission theory, dynamically assigns labels according to the matching degree between each predicted box and the real target, and optimizes the matching of positive and negative samples during training. The predicted box set D = {d1, d2, ..., d n} and the true target set G = {g1, g2, ..., g m}, and maximize the total matching degree through the optimal transmission matrix T:

[0130]

[0131] T = SimOTA(D, G)

[0132] 24) The loss function consists of classification loss, regression loss and objectness loss. Under the decoupled head structure, the classification and regression losses are calculated separately, classification loss (using Focal Loss) regression loss (GIoU) ​​objectness loss.

[0133]

[0134] Among them, p i is the predicted class probability, α and γ are adjustment parameters.

[0135]

[0136] Where b is the true bounding box, is the predicted bounding box.

[0137]

[0138] Among them, y i is the true label, is the confidence of the predicted object.

[0139] S3. Build a target tracking model for traffic scenarios, optimize the combination of detection and tracking through innovative strategies, make full use of low-confidence detection, and efficient target association strategies to significantly improve tracking accuracy and robustness.

[0140] 31) In this example, in the tth frame, the target detector is used to generate a set of detection boxes:

[0141]

[0142] For high confidence and low confidence detection classification, each detection box d of frame t t,i Contains bounding box coordinates Confidence According to the confidence threshold τ, the detection frame is classified into high confidence and low confidence. By combining high confidence and low confidence detection, it ensures that targets such as vehicles and pedestrians can be continuously and accurately tracked even in high-density traffic and complex environments.

[0143]

[0144]

[0145] 32) High confidence matching, simple association strategy based on IoU, the target tracking model of traffic scenes and the optimization of the Hungarian algorithm enable the target tracking model of traffic scenes to achieve real-time target tracking at high-traffic intersections, meeting the strict real-time requirements of the intelligent transportation system.

[0146]

[0147] A track =Hungarian(M track )

[0148] A track It is the matching relationship between the detection box and the target trajectory obtained by the Hungarian algorithm, representing a matching pair Indicates which detection boxes d t,i And which trajectories A matching relationship is established. track The result is directly used to update the target trajectory set T t .

[0149] Update to new detection box, or remove unmatched objects.

[0150]

[0151] For the successfully matched trajectory, update it to the corresponding detection frame and The position and state of the detection box d t,i The position and status of the target (such as the bounding box position information, category information, etc.).

[0152] 33) Perform IoU matching again and update the tracking status, creating a new target for high confidence tracking New tracking targets are also created for low-confidence detection frames that meet the matching conditions. In complex traffic scenarios such as fast-moving vehicles and frequent occlusions, the target tracking model can maintain stable target tracking through a flexible detection frame association and update mechanism, thereby improving the overall robustness and reliability of the system. Repeat the operations 31) to 33) for the previously unmatched trajectory set, and try to add the low-confidence detection box to the unmatched trajectory.

[0153] S4. A cross-camera ID matching module is constructed based on the perception results. First, the results of a single camera are mapped to a unified global coordinate system. Then, the cross-camera targets are processed by extracting prior information and enhancing features, and the position association is optimized using the Hungarian algorithm to achieve global cross-camera ID matching.

[0154] 41) In this example, S1 obtains the camera calibration parameter P. The coordinates of the center point of the single camera detection frame in the image are (x i ,y i ), the camera calibration parameter P, then the coordinates in the global coordinate system (X i, Y i ) can be calculated by the following affine transformation formula:

[0155]

[0156] 42) In the boundary area of ​​the field of view of each camera, multiple prior regions are preset according to the geometric layout of the traffic intersection and the driving route. These prior regions are used to predict the possible location of the target, reduce the matching search space, and improve the matching efficiency. For the targets divided into the prior regions, the pre-trained Re-Identification (ReID) model TransReID is used to effectively extract the recognizable features. Suppose the target i is in the prior region P j The eigenvector in is f i , the target k is in the prior region R d (d represents 4 cameras with different viewing angles) The feature vector in is f k , then the cosine similarity The calculation is as follows:

[0157]

[0158] 43) Based on the cosine similarity calculated in the previous step, construct the similarity matrix S of the cross-camera targets, where S mn Where n and m are the number of targets detected by the two cameras in the current time frame, and the Hungarian algorithm is used to solve the optimal matching matrix A reid , in order to maximize the total similarity of matching pairs, determine the correspondence between targets across cameras, update the global ID of the target, and achieve consistent tracking across cameras.

[0159] S5. In view of the inaccurate detection, abnormal jitter or data missing of the target under the global perspective obtained in S4, the trajectory smoothing and prediction module can effectively deal with the inaccuracy, abnormal jitter and data missing of target detection in traffic intersection scenes, ensuring the continuous and stable perception and tracking of the multi-camera system in complex dynamic environments.

[0160] 51) For each target i in time frame t, its current position is The position is smoothed using an exponentially weighted average method:

[0161]

[0162] α is a smoothing factor, 0<α<1, which controls the weight of the new position and the historical position. is the current position after smoothing.

[0163] 52) For each target i in time frame t, its current motion direction angle is The angle smoothing also uses exponentially weighted averaging.

[0164]

[0165] 53) The velocity vector of target i at time frame t is It can be calculated by position change:

[0166]

[0167] Where Δt is the time interval.

[0168] The speed smoothing also uses exponentially weighted averaging:

[0169]

[0170] 54) When target i is not detected in time frame t (i.e., data is missing), the current position is estimated based on the smoothed velocity of the previous frame:

[0171]

[0172] S6. In traffic scenes, there are many different visual perception targets, such as license plate recognition, color detection, anomaly detection, lane line detection, equipment and facility detection, and vehicle model detection. For these targets, corresponding intersection camera visual perception models are constructed. These models are developed with lightweight and robustness as the design premise, so that efficient perception can be achieved in complex scenes.

[0173] 61) In terms of lightweighting, by optimizing the model structure and algorithm logic, the model's computational workload and storage requirements are greatly reduced, allowing it to run efficiently in an environment with limited hardware resources. Specifically, the model uses deep separable convolution technology, which decomposes the standard convolution into deep convolution and point-by-point convolution, thereby reducing the number of convolution kernel parameters and effectively compressing the model parameters. At the same time, the model pruning technology is used to remove neurons and connections in the model based on the degree of influence on the results, reducing unnecessary intermediate calculation links, and effectively avoiding performance bottlenecks caused by bloated models.

[0174] 62) In terms of robustness, the present invention incorporates an adaptive illumination adjustment algorithm, which dynamically adjusts the image contrast and brightness based on the brightness histogram information of the image, thereby automatically adapting to changes in image features under different illumination conditions, and significantly improving the accuracy of target recognition in complex illumination scenes. Secondly, an image denoising filtering algorithm is also introduced. This algorithm removes high-frequency noise signals generated by severe weather based on the analysis of image signals of different frequencies, thereby effectively filtering image noise caused by severe weather and ensuring perception accuracy under severe weather conditions. In addition, the present invention designs a target recognition algorithm based on contextual information, which analyzes the relationship between the target's surrounding environment and other targets, and makes reasonable inferences about the obscured target, so that the model can still accurately identify the target when facing complex target obstructions and image confusion caused by traffic congestion. This robustness ensures that the model can maintain efficient perception capabilities in various complex scenarios, effectively making up for the shortcomings of existing technologies in complex traffic scene applications, and providing reliable, accurate and efficient visual perception support for traffic management and monitoring.

[0175] S7. Perform a comprehensive performance evaluation of the detection performance of a single camera and the unified visual perception results of multiple cameras at intersections on the visual analysis method for the multi-camera collaborative perception under the complex traffic environment.

[0176] In this embodiment, the automatic evaluation index used includes AP and its derivative index AP 50 , A.P. 75 , A.P. S , A.P. M , A.P. L It is a key indicator for evaluating the performance of target detection models. The artificial evaluation indicators used are mainly evaluated from the tracking accuracy and target tracking precision.

[0177] 71) The detection performance of a single camera is comprehensively reflected in various detection tasks through the combination of different IoU thresholds and target sizes, providing an important basis for model optimization and application.

[0178] 72) For the unified visual perception of multi-camera intersections, manual evaluation indicators are used: Tracking accuracy: comprehensively considers detection errors (missed detection, false detection) to evaluate the overall tracking performance. Target tracking accuracy: measures the matching accuracy of the bounding box position during the tracking process, reflecting the fine-grained accuracy of the tracking.

[0179] Corresponding to the above steps, Figure 2 This is an example of a fast intersection calibration diagram; Figure 3 A diagram of the framework for target detection and tracking in traffic scenes; Figure 4 Schematic diagram for setting the cross-lens prior region; Figure 5 This is an example of visual analysis for collaborative perception at multi-camera intersections.

[0180] In order to verify the performance of the method of this application, the following experiments were designed.

[0181] This embodiment is compared with other advanced object detection models on the road-side traffic dataset we use. The specific details are as follows:

[0182] (1) YOLOv3, an object detection model based on multi-scale prediction capabilities; (2) EfficientDet, an object detection model using a scalable architecture and compound scaling techniques; (3) PP-YOLOv2, an object detection model based on a more advanced backbone network optimized by YOLO; (4) YOLOv5, a highly flexible and efficient object detection model;

[0183] The specific experimental results are shown in Table 1 and Table 2.

[0184] Table 1 Performance of different object detection models on the road-side traffic dataset (%).

[0185]

[0186]

[0187] AP is the average precision of the comprehensive precision and recall information in target detection. 50 ,AP 75 They are the AP values ​​when the intersection-over-union ratio threshold is 50% and 75%, respectively. S , A.P. M , A.P. L It is the average accuracy of detecting small, medium and large target objects. Experimental results show that our method outperforms other models in the detection performance of road-side traffic datasets and has higher accuracy in vehicle detection capabilities.

[0188] Table 2: Manual evaluation results of the present invention in actual scenarios

[0189] direction Number of tests Actual car passing Accuracy Detection rate north 59 78 75％ 93％ South 71 84 80％ 84％ West 114 124 91％ 91％ East 191 210 91％ 91％

[0190] The table shows the half-hour manual evaluation test of the proposed method at different intersections (north, south, west, east), and the results fully demonstrate its effectiveness and reliability. It is particularly noteworthy that the accuracy and detection rate of the west and east intersections are both as high as 91%. The data strongly shows that our method can still maintain high-precision vehicle detection capabilities under complex traffic flow conditions, which is highly consistent with the actual passing situation, and fully proves the stability and efficiency of the method in different traffic flow scenarios.

[0191] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A visual analysis method for collaborative perception at traffic intersections, characterized in that: The method comprises the following steps: S1, using the intersection fast calibration algorithm, by setting a number of key points at different positions of the traffic intersection, generating an affine transformation matrix, and obtaining camera images at different positions; S2. For the camera image at each position, input the target detection model to obtain the target detection result; S3, the target detection result at each position is input into the target tracking model, and the target tracking result is output; S4, transforming the target tracking result into a unified global coordinate system based on the affine transformation matrix, performing global cross-lens ID matching, and obtaining a cross-camera tracking result; S5, performing smoothing processing on the cross-camera tracking result to obtain a smooth tracking result; S6. Perform visual perception based on the smooth tracking result to obtain a visual perception result.

2. The visual analysis method for collaborative perception at traffic intersections according to claim 1 is characterized in that: The affine transformation matrix is: Among them, a 11 ,a 12 ,a 21 ,a 22 is the linear part parameter, t x ,t y is the parameter of the translation vector.

3. The visual analysis method for collaborative perception at traffic intersections according to claim 1 is characterized in that: The specific steps of S2 are: The camera image at each position is input into the backbone network to obtain the feature map F. The feature map F passes through the multi-scale aggregation module to obtain the fusion feature. The fusion feature is input into the classification and regression branches respectively. The classification and regression branches output the detection box to obtain the target detection result.

4. The visual analysis method for collaborative perception at traffic intersections according to claim 3 is characterized in that: The training process of the target detection model is: Construct a target detection network, based on the prediction box set D = {d1, d2, ..., d n } and the true target set G in the training set = {g1,g2,…,g m }Maximize the transfer matrix, and use the predicted box under the maximized transfer matrix as the current predicted box output by the target detection network. The maximized transfer matrix is ​​specifically: O = SimOTA(D, G) Among them, n is the number of prediction boxes in the prediction box set, m is the number of real targets, O ij represents an n×m matrix, if d i Assigned to g j , then O ij =1, otherwise 0, SimOTA represents the dynamic label assignment strategy, D represents the set of prediction boxes output by the target detection network, d i represents a prediction box, G represents the real target set in the training set, and g i represents a real target, S is also an m×n matrix, where S ij Denotes the predicted box d i and the real target g i The matching score.

5. The visual analysis method for collaborative perception at traffic intersections according to claim 4 is characterized in that: The loss function of the target detection model includes classification loss, regression loss and objectness loss, among which the classification loss is: in, is the classification loss, N pred represents the number of prediction boxes, p i is the predicted class probability, α and γ are adjustment parameters; The regression loss is: in, represents the regression loss, b is the true bounding box, To predict the bounding box; The objectness loss is: in, Represents objectness loss, y i is the true label, is the confidence of the predicted object.

6. The visual analysis method for collaborative perception at traffic intersections according to claim 1 is characterized in that: The specific steps of S3 are: 31) The detection box set of the target detection result is divided into a high confidence detection box set And the low confidence detection box set And input into the target tracking model, from the confidence detection box set Select a current detection box d t,i , based on a set of high confidence detection boxes And the target trajectory set T of the previous frame t-1 Construct the matching matrix and define the t-1 frame, that is, the target trajectory set of the previous frame as N track Represents the number of tracked targets and calculates each trajectory With the current detection box d t,i IoU, get the matching matrix 32) Use the Hungarian algorithm to match the matrix M track Solve the optimal allocation above, match the high confidence detection frame with the target track of the previous frame, and obtain the matching result. In the matching result, if a target track of the previous frame is not matched, it is considered that the track may disappear, and the track is set as an unmatched track. If a detection frame is not matched to the track, it is considered to be a new target, and the detection frame is set as an unmatched detection frame. If a match is found, the matching relationship between the successfully matched detection frame and the target track is: A track =Hungarian(M track ) A track It is the matching relationship between the detection box and the target trajectory obtained by the Hungarian algorithm, representing a matching pair 33) Based on the matching results, the current frame target trajectory set T is obtained t , specifically: For the successfully matched trajectory, update it to the corresponding detection frame and The position and state of the detection box d t,i location and status; For unmatched detection boxes, in the trajectory Create a new target trajectory in For unmatched trajectories, if multiple consecutive frames are not matched, Remove the trajectory from the current frame to obtain the target trajectory set T t , update the time, repeat 31)~33) to obtain a high confidence tracking result; 34) Supplementary matching of low confidence detection boxes, for low confidence detection box sets The operations of 31) to 33) are repeated for the previously unmatched trajectory set to obtain a low-confidence tracking result. The low-confidence tracking result and the high-confidence tracking result are integrated into the target tracking result.

7. The visual analysis method for collaborative perception at traffic intersections according to claim 1 is characterized in that: The specific steps of S4 are: 41) Unifying the cross-camera tracking results into the global coordinate system based on the affine transformation matrix to obtain a unified tracking result; 42) Preset multiple prior regions, extract feature vectors for targets in the prior regions in the unified tracking results, and extract feature vectors for target i in the prior region P. j The eigenvector f i and target k in the prior region P d The eigenvector f k Calculate the cosine similarity of feature vectors; 43) Find the optimal target matching relationship A based on cosine similarity reid ; 44) Based on the target ID corresponding to the optimal target matching relationship, determine whether different IDs under cross-lens conditions are the same target, and obtain the cross-camera tracking result.

8. The visual analysis method for collaborative perception at traffic intersections according to claim 7 is characterized in that: The cosine similarity is: in, Denotes the eigenvector f i and the eigenvector f k The cosine similarity of .

9. The visual analysis method for collaborative perception at traffic intersections according to claim 1, characterized in that: The specific steps of S5 are: For the cross-camera tracking results at time frame t, if there is a target, then: The current position of each target in time frame t is smoothed using the exponential weighted average method to obtain a position smoothing result; The current moving direction angle of each target in time frame t is smoothed using an exponential weighted average method to obtain a moving direction angle smoothing result; The velocity vector of each target in time frame t is smoothed by using the exponential weighted average method to obtain the velocity vector smoothing result. The three smoothing results are used as the smooth tracking results of the target. If there is no target, the current position is estimated based on the velocity vector smoothing result of the previous frame as the position smoothing result.

10. The visual analysis method for collaborative perception at traffic intersections according to claim 9, characterized in that: The estimated current position is: in, is the current location, is the position smoothing result of the previous frame, The velocity vector smoothing result for the previous frame.

Images