Multi-source collaborative intelligent traffic management method

Abstract

The invention relates to a multi-source collaborative intelligent traffic management method, and the method comprises the following steps: S1, constructing an affine transformation equation set, and calibrating an internal and external parameter matrix of a camera; s2, obtaining a target sensing result under each view angle, and carrying out cross-lens matching and optimization to obtain a road end global sensing result; s3, inputting the task cue word, the road end global perception result and the original video stream into a Qwen2-VL model of the cloud to obtain prior information; s4, the end-to-end cloud sensing model outputs a cloud global sensing result by taking the four paths of video data as input; s5, performing position-semantic hybrid alignment to obtain an enhanced global perception result; and S6, optimizing the enhanced global perception result based on traffic rule prior information and historical data of the perception result. Compared with the prior art, the method has the advantages of improving the matching precision of the sensing data at the two ends of the cloud path, enhancing the stability of end-cloud collaboration and the like.

Description

Technical Field

[0001] The present invention relates to the field of smart transportation, and in particular to a multi-source collaborative smart transportation management method. Background Art

[0002] With the acceleration of urbanization and the continuous growth of the number of motor vehicles, traffic congestion and safety issues have become more serious. Traditional traffic management methods often rely on single sensor data, such as traffic cameras or vehicle sensors, which makes it difficult for the system's perception and decision-making efficiency in complex traffic scenarios to meet actual needs. In this context, the core goal of intelligent traffic management methods is to achieve comprehensive monitoring and precise management of the entire road area through efficient collaboration among multiple devices. This task not only has important industrial value, but has also gradually emerged in the fields of computer vision and multimedia computing in recent years, and has made significant progress.

[0003] The intelligent traffic management method needs to solve two major difficulties: (1) efficient and accurate real-time perception, how to process massive traffic data with limited costs and resources, and ensure that the method has sufficient response speed and perception accuracy; (2) the stability and robustness of the method, especially in severe occlusion or bad weather conditions, how to maintain the stability and reliability of perception performance. In response to the first difficulty, many researchers use pruning or distillation strategies to reduce the number of parameters in the network structure. However, the existing methods for processing radar data have increased the cost of the model while bringing performance benefits and reducing the running speed of the model. In response to the second difficulty, although many models improve their perception capabilities in challenging scenarios by expanding training data sets or using data enhancement techniques, the above methods still cannot meet actual application needs when faced with severe occlusion or extreme weather conditions. Summary of the invention

[0004] The purpose of the present invention is to improve the matching accuracy of perception data at both ends of the cloud and road, enhance the stability of end-cloud collaboration, and thereby improve the perception capability in challenging scenarios and improve the efficiency and accuracy of perception results, thereby providing a multi-source collaborative intelligent traffic management method.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A multi-source collaborative intelligent traffic management method, the method comprising the following steps:

[0007] S1. Construct an affine transformation equation group to calibrate the camera's internal and external parameter matrix;

[0008] S2. Obtain the target perception results from each perspective, realize the target coordinate system transformation based on the internal and external parameter matrices, obtain the target detection and tracking results from the global BEV perspective, and perform cross-lens matching and optimization to obtain the road-side global perception results;

[0009] S3, input the task prompt words, road-side global perception results and original video stream into the Qwen2-VL model in the cloud to obtain the prior information of the color, license plate and model of the challenging target in each perspective;

[0010] S4, the end-to-end cloud perception model takes four-channel video data as input, combines the prior information output by Qwen2-VL and the internal and external parameter matrix of S1, and outputs the cloud global perception results through feature extraction and attention mechanism;

[0011] S5, performing position-semantic hybrid alignment on the road-side global perception result and the cloud-side global perception result to obtain an enhanced global perception result;

[0012] S6. Global perception results after optimization and enhancement based on prior information of traffic rules and historical data of perception results.

[0013] Furthermore, the specific steps of S4 are:

[0014] Input the four-channel video data into the convolutional neural network of the cloud perception model for feature extraction, obtain the convolution features and basic depth information of the image, and reduce the number of channels of the convolution features to d. Then, cascade the depth information and the image convolution features with d channels as the input of the subsequent attention mechanism;

[0015] The serialized image features are input into the attention mechanism of the cloud perception model. The encoder part of the attention mechanism uses eight sets of parallel encoding blocks to obtain 8 sets of query features, and the weighted query features are saved in the memory unit as weighted memory features. The cross attention mechanism in the decoder uses the weighted memory features as the current key and value, and finally the perception head is connected to the decoded output features to obtain the global perception result of the cloud.

[0016] Furthermore, a set of query features is:

[0017] QbV t =Qo t ⊙Atten(Σ(Qo t , Cal)

[0018] Among them, Q t Represents the video encoding output of each channel, and Cal represents the internal and external parameter matrix characteristics.

[0019] Furthermore, the encoding output of each video channel is:

[0020]

[0021] in, is the serialized image feature, is the encoded external knowledge feature.

[0022] Furthermore, the weighted memory feature is:

[0023] Mu=Qbev⊙Atten(Qbev)

[0024] Among them, Qbev is 8 groups of query features.

[0025] Furthermore, in S5, if there is a conflict between the road-side global perception result and the cloud-side global perception result, the cloud-side global target perception result is used as the enhanced global perception result.

[0026] Furthermore, the position-semantic hybrid alignment is specifically as follows:

[0027] The matching degree, cosine similarity and L2 distance of the road-side and cloud-side location information are calculated to obtain an optimization function, and the location-semantic hybrid alignment is performed based on the optimization function.

[0028] Furthermore, the optimization function is:

[0029]

[0030] Among them, T p and C p They are used to represent the instance locations in the road-side and cloud-side global perception results, Cos(T s , C s ) and L2(T s , C s ) represent cosine similarity and L2 distance respectively, T s and C s They are used to represent the semantic features extracted from the global perception results of the road side and the cloud side based on the text encoder, T and C represent the comprehensive perception results of the road side and the cloud side in the global coordinate system, and λ iou , cos , L2 is a hyperparameter.

[0031] Furthermore, the cosine similarity is:

[0032]

[0033] Among them, T i and C i Represent the perception results of the i-th instance on the road side and in the cloud side respectively.

[0034] Furthermore, the L2 distance is:

[0035]

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention combines high-precision cloud algorithms driven by large models and relies on the powerful reasoning capabilities of the multimodal large model (Qwen2-VL) to provide accurate external knowledge priors for the processing of challenging instance targets. At the same time, the introduction of memory features and depth information further improves the accuracy of the end-to-end cloud tracking algorithm. In addition, in addition to basic position matching, the location-semantic hybrid alignment of the road-side and cloud-side perception results also uses cosine similarity and L2 distance to jointly constrain the matching process of semantic features, and introduces threshold hyperparameters to finely screen redundant information, further improving the matching accuracy and enhancing the stability of end-cloud collaboration. Finally, by combining traffic rules prior information and historical data of perception results, the present invention achieves the optimization of each target trajectory from a data level, thereby improving the overall accuracy, stability and fluency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the main steps of the present invention;

[0039] Figure 2 This is the framework diagram of the road-side real-time perception algorithm;

[0040] Figure 3 High-precision cloud algorithm architecture diagram driven by large models;

[0041] Figure 4 It is the framework of the end-cloud collaborative fusion module;

[0042] Figure 5 Flowchart for post-processing logic optimization guided by prior knowledge DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0044] The present invention proposes a multi-source collaborative intelligent traffic management method, which takes complex traffic scenes as the research object, uses multi-channel video data as input, combines multi-device collaboration and end-cloud collaboration technology to achieve real-time monitoring, accurate analysis and refined management of traffic scenes. The intelligent traffic management method includes: a road-side real-time perception algorithm, a high-precision cloud algorithm driven by a large model, an end-cloud collaborative fusion module and an efficient post-processing optimization module. The road-side real-time perception algorithm realizes accurate perception and tracking of traffic participants through edge computing, and the high-precision cloud algorithm driven by a large model is used to enhance global perception and deep reasoning capabilities. The end-cloud collaborative fusion module is used to balance efficiency and accuracy to improve the robustness and generalization of the model. The efficient post-processing optimization module is used to further improve the display accuracy and visual effects of the intelligent traffic intersection. Compared with the most advanced existing methods, the present invention shows significant advantages in perception accuracy, response efficiency, stability and adaptability. In particular, when dealing with complex scenes and diversified traffic conditions, it shows efficient real-time and strong robustness, which not only improves travel safety and traffic efficiency, but also provides scientific decision-making support for urban traffic management, and is an important technological breakthrough in promoting the construction of smart cities. The method comprises the following steps, and the corresponding flow chart is as follows: Figure 1 As shown:

[0045] S1. Construct an affine transformation equation group to calibrate the camera's internal and external parameter matrix;

[0046] S2. Obtain the target perception results from each perspective, realize the target coordinate system transformation based on the internal and external parameter matrices, obtain the target detection and tracking results from the global BEV perspective, and perform cross-lens matching and optimization to obtain the road-side global perception results;

[0047] S3, input the task prompt words, road-side global perception results and original video stream into the Qwen2-VL model in the cloud to obtain the prior information of the color, license plate and model of the challenging target in each perspective;

[0048] S4, the end-to-end cloud perception model takes four-channel video data as input, combines the prior information output by Qwen2-VL and the internal and external parameter matrix of S1, and outputs the cloud global perception results through feature extraction and attention mechanism;

[0049] S5, performing position-semantic hybrid alignment on the road-side global perception result and the cloud-side global perception result to obtain an enhanced global perception result;

[0050] S6. Global perception results after optimization and enhancement based on prior information of traffic rules and historical data of perception results.

[0051] The present invention takes multi-channel videos in traffic scenes as input, and utilizes multi-device collaboration and end-cloud collaboration technology to achieve real-time monitoring and refined management of complex traffic scenes. The intelligent traffic management method includes: a road-side real-time perception algorithm, a high-precision cloud-side algorithm driven by a large model, an end-cloud collaborative fusion module, and an efficient post-processing optimization module. The road-side real-time perception algorithm is used to obtain various information of traffic participants in real time, the high-precision cloud-side algorithm driven by a large model is used to enhance global perception and deep reasoning capabilities, the end-cloud collaborative fusion module is used to balance efficiency and accuracy to improve the robustness and generalization of the model, and the efficient post-processing optimization module is used to further improve the display accuracy and visual effects of the smart intersection.

[0052] In the present invention, steps S1 and S2 mainly realize the design and training of the road-side real-time perception algorithm, which mainly includes sensor calibration, perspective unification, image feature extraction, instance detection and tracking, cross-lens ID matching, license plate detection, vehicle color detection, vehicle model detection, abnormal event detection and trajectory smoothing and prediction, and performs joint training and reasoning on the COCO dataset, VeRi dataset and CCPD dataset.

[0053] In S1, the sensor calibration and multi-device perspective unification strategy is based on key points on the road surface and actual distance. By combining perspective transformation and the geometric position relationship of different perspectives, the internal and external parameter matrices of each device are solved, thereby achieving precise alignment and unification of multi-device perspectives.

[0054] Based on the key points of the road surface and the actual distance, combined with the perspective transformation and the geometric position relationship of different perspectives, the internal and external parameter matrices of each device are solved, thereby achieving accurate alignment and unification of the perspectives of multiple devices. Specifically, in order to establish the mapping relationship between 2D and 3D, the affine transformation equations are first constructed through the correspondence between the pixel coordinates of the key points in the image and the physical coordinates in the corresponding orthophoto, and the local optimal solution is obtained as the internal and external parameters of the camera by combining the Gauss-Newton optimization method. In view of the lack of sufficient capture key points in the central area of ​​the road surface, it is necessary to manually deploy several control points in the early stage of calibration to construct an effective mapping relationship. In addition, in order to ensure the calibration accuracy, it is required to select at least three non-collinear key points to determine the transformation matrix, because the six-degree-of-freedom transformation requires at least three pairs of coordinate points to construct enough equations (each pair of coordinate points can generate two independent equations). After obtaining the calibration parameters of each device, through the correspondence between the pixel coordinate system, image coordinate system, camera coordinate system and world coordinate system, combined with the internal and external parameter matrix, the pictures of different perspectives can be unified into the world coordinate system, thereby achieving the unification of perspectives, that is, the global BEV perspective.

[0055] In S2, the road-side real-time perception algorithm takes multi-channel video data as input, and combines target detection (YOLOX), multi-target tracking (ByteTrack), re-identification (ReID), character recognition (OCR), inference acceleration (Tensor-RT), module parallelization and hard decoding technologies to accurately implement various complex visual tasks while ensuring the efficiency of model operation.

[0056] like Figure 2 As shown in the figure, the end-to-end algorithm takes multi-channel video data as input, designs specific detection and tracking strategies, combines local ReID and trajectory optimization methods, and realizes efficient global real-time perception tasks. Specifically, under the computing power configuration of the terminal device and the deployment requirements of real-time operation, the terminal perception algorithm in this method adopts a detection-based multi-target tracking strategy, which mainly includes three stages: single-view target detection, single-view multi-target tracking, and cross-view matching and fusion. In the target detection stage, in order to improve the operation efficiency of the algorithm and the detection accuracy of long-distance small targets, based on the Darknet backbone network, a strategy combining IOU matching and dynamic label allocation is adopted to improve the accuracy of classification and regression. In addition, the end-to-end target center detection and position regression method that eliminates the post-processing process also further improves the operation efficiency of the detection. In the multi-target tracking stage, in addition to the basic ByteTrack tracking algorithm, a high-low confidence matching strategy is designed. By adding logical judgments of position and category to low-confidence instance target matching, it makes full use of low-confidence detection results, thereby effectively improving tracking performance and model robustness. Finally, in order to better perform the global perception task of smart intersections, this method divides specific prior regions (i.e., fan-shaped regions with four corners as the center and a radius of 10 meters) for complex intersection scenes, and sets logical vanishing activation points (i.e., the moment of screen switching within the prior region) to trigger the pre-trained ReID module to perform secondary registration of unmatched instances at the moment of screen switching. Through the combination of the above logic and algorithms, the model significantly improves the accuracy of cross-lens matching and global target tracking without increasing the complexity of the algorithm.

[0057] Steps S3 and S4 mainly implement the design and training of high-precision cloud algorithms driven by large models. Relying on the powerful reasoning ability of the multimodal large model (Qwen2-VL), accurate external knowledge priors are provided for complex target instances, such as color information under strong light, vehicle license plate information under license plate occlusion, etc. In addition, the introduction of memory modules and depth estimation strategies further improves the accuracy of the end-to-end tracking algorithm.

[0058] In S3, the knowledge enhancement strategy driven by a multimodal large model uses the reasoning ability of Qwen2-VL and designs task-specific prompts to provide the model with external knowledge in challenging traffic scenarios, such as color information under strong light, vehicle license plate information under license plate occlusion, and traffic light information at intersections. Figure 3 The specific process of the strategy is demonstrated. In addition to receiving the original video frame data, the cloud algorithm also uses the real-time perception results of the road side as input to preliminarily determine the target vehicle in the video frame. In addition, in order to give full play to the reasoning ability of the visual language large model, task-specific prompts are also designed. For example, for the vehicle color recognition task, the generated prompt is: "Here is a traffic scene map and the pixel position of the vehicle in it. Please output the true color of each numbered instance according to the instance position and the corresponding ID number (if the instance is a non-motor vehicle, it can be ignored)". Finally, by inputting the above video data, relevant perception results and task prompts into the Qwen2-VL model, the corresponding external knowledge such as color, license plate and type is provided for the instance target at each moment. It should be noted that in order to ensure high efficiency and real-time performance, quantitative compression and decompression technology is used in the process of sending and receiving data, and the low-latency and highly reliable MQTT protocol is used to optimize communication to further improve the processing speed and stability of the model.

[0059] Specifically, take the tth frame image in the i-th video Taking the color recognition task as an example, the corresponding road-side real-time perception result is The prompt generated by the vehicle color is Then the above knowledge enhancement strategy can be defined as follows:

[0060]

[0061] in Contains the ID and color information of each instance target in the current image, and VLM is the Qwen2-VL model. Similarly, the above strategy can also be used to obtain the license plate dictionary of each target and traffic light signals

[0062] In S4, the end-to-end video perception algorithm that integrates detection and tracking enhances the model's perception capabilities when facing severe occlusion and small targets at long distances by incorporating memory units and depth estimation strategies. At the same time, the full use of historical data further improves the generalization and robustness of the model when dealing with long-tail problems.

[0063] The overall framework of the end-to-end tracking algorithm is as follows Figure 3 As shown, the specific construction and training steps include:

[0064] 41) Build a training dataset for the end-to-end algorithm. The end-to-end tracking algorithm for four-channel video takes four-channel video data as input and outputs the detection and matching results of each frame. Existing tracking datasets usually only contain single-view video images and cannot provide global information of the entire intersection, so the construction of this dataset is crucial.

[0065] Specifically, the input data is obtained by cutting each video data into 20-second segments of equal length and matching the four data. In terms of data annotation, preliminary labels are generated through the road-side real-time perception algorithm and manually proofread frame by frame. It should be noted that when the dataset was constructed, the acquisition frame rate was 25 frames / second, while the model training used a lower frame rate of 8 frames / second to ensure training efficiency.

[0066] 42) Figure 3 As shown, for the four-channel video data input (For ease of description, we still take the tth frame image of the i-th video For example), we first use a simple convolutional neural network (ResNet) to extract features and obtain the convolution features of the image. Finally, a 1×1 convolution operation is used to reduce the number of feature channels to d to improve the computational efficiency of the subsequent attention mechanism. At the same time, in order to obtain the depth information of the instance target, the classic category depth generation module is used to obtain the depth map of the video frame. And cascade it with the convolution feature to obtain a more informative image feature. Where C′=C+1,

[0067] Next, by serializing the image features and the encoded S3 external knowledge features MLP is a simple three-layer perceptron, which is combined as the input of the attention mechanism to further obtain the global information of the image. It should be noted that the composition of the single encoding and decoding unit of the attention mechanism is consistent with the DETR model. Specifically, in order to integrate the four-way data information at time t, based on the Transformer encoder of DETR, the self-attention mechanism is used to integrate the features of the internal and external parameter matrices of S1. Video encoding output Perform weighted processing to obtain the fused global coding features Where 100 is the number of query instance features, and D is the corresponding feature dimension. The above process can be defined as follows:

[0068] QbV t =Qo t ☉Atten(Σ(Qo t , Cal)

[0069]

[0070] Cal=MLP(C ma )

[0071] Among them, Atten is the self-attention layer, Encoder is the Transformer encoder of the DETR model, and C ma is the camera internal and external parameter matrix obtained by S1. At the same time, in order to solve the performance degradation problem caused by severe occlusion and extremely bad weather, the encoder also uses eight sets of parallel coding blocks to simultaneously process 8 frames of continuous data including the current frame, where the specific structure of each coding block is the same as described above, and then the global query feature Qbev corresponding to the continuous data is obtained. t1 , Qbev t2 , ..., Qbev t8}. For the parallel encoding results, the basic self-attention layer is still used to weight the different encoding results, and the weighted results are saved in the memory unit, that is, Mu = Qbev⊙Atten(Qbev), where Mu is the memory feature corresponding to time t. For the decoder part, similar to the DETR model, 100 instances are still randomly initialized to track different target instances, and the weighted memory features of the encoder are used as the key (Key) and value (Value) of the cross attention in the decoder. Through the above operations, historical data information can be fully utilized, so that the model can combine temporal context features in the reasoning process, thereby improving end-to-end perception and prediction capabilities.

[0072] The step of S5 is to send the road-side and cloud-side perception results into the designed location-semantic hybrid alignment fusion module, while considering the location similarity and semantic similarity, so as to improve the model's tolerance to location errors and take into account the operating efficiency and perception accuracy of the entire management method.

[0073] The end-cloud collaborative fusion module of the location-semantic hybrid alignment in S5. In order to balance the operating efficiency and perception accuracy of the method, the perception results of the road side and the cloud side are sent to the designed location-semantic hybrid alignment fusion module for joint processing. By considering both location similarity and semantic similarity, the model's tolerance to location errors is improved, and the stability of the end-cloud collaboration is enhanced. Specifically, Figure 4 As shown in the figure, after receiving the perception results from the edge and the cloud, in addition to using the location information for instance matching, a text encoder is designed to further extract the semantic features of the instance target, namely, attribute features such as color, vehicle model, and license plate. For the matching of the extracted semantic features, cosine similarity and L2 distance are used as joint constraints to further improve the matching accuracy. The above steps can be defined as:

[0074]

[0075]

[0076] Among them, IoU(T p , C p ) is used to measure the matching degree of position information. AoI and AoU represent the intersection and union of detection boxes respectively. p and C p They represent the instance positions of the perception results of the terminal and the cloud in the global coordinate system respectively; Cos(T s , C s ) and L2(T s , C s ) represent the cosine similarity and L2 distance respectively, which measure the matching degree of semantic features. s and C s Represent the semantic features of the perception results extracted by the terminal and the cloud, respectively, T i and C i Represent the perception results of the i-th instance of the terminal and the cloud respectively. Based on the above measurement method, the optimization function of the final fusion module is defined as:

[0077]

[0078] where λ iou , cos , L2 is a hyperparameter, T and C are used to represent the overall perception results of the terminal and the cloud in the global coordinate system. For the perception results after the instance is successfully matched, the average strategy is used to process the coordinate information of the detection box. When there is a conflict in matching, other information such as category, license plate, and model, the more accurate cloud perception results are used first.

[0079] The steps of S6 are: based on the post-processing logic optimization module guided by knowledge a priori, combining the prior information of traffic rules with the historical data of perception results, deeply exploring the potential laws of the target trajectory, and effectively improving the accuracy and stability of the target trajectory through logical reasoning and optimization at the data level.

[0080] like Figure 5As shown in the figure, in the post-processing logic optimization guided by knowledge prior, by combining the prior information of traffic rules and the historical data of perception results, the problems such as accidental ID jumps, sudden disappearance or appearance of IDs, and short-term data loss caused by network fluctuations in the perception results are successfully dealt with. This method realizes accurate optimization of the target trajectory from the data level. Specifically, the post-processing optimization module relies on the output data of the end-cloud collaborative fusion module, sets id_group to distinguish instance targets of different IDs during the data reading process, and then finds the data loss and repeated sending problems caused by network fluctuations through the frame number and position of adjacent data in different ID groups. Starting from the logical level, the equidistant interpolation algorithm is combined to supplement the missing frames and remove duplicate frames. In addition, in order to adapt to the size of the rendering canvas and improve the smoothness of the rendering picture, the post-processing optimization module also performs appropriate backtracking and extension operations on the appearance and disappearance of instance targets through logical judgment, and combines trajectory smoothing technology to enhance smoothness. Finally, in order to ensure stability during long-term operation, a timed cache cleaning mechanism is designed. It selectively clears specific instance targets by judging whether a certain identification group has changed within a certain period of time, avoiding system performance degradation caused by cache backlog.

[0081] The present invention uses multi-channel videos in traffic scenes as input, and utilizes multi-device collaboration and end-cloud collaboration technology to achieve real-time monitoring and refined management of complex traffic scenes. The smart traffic method includes: a road-side real-time perception algorithm, a high-precision cloud-side algorithm driven by a large model, an end-cloud collaborative fusion module, and an efficient post-processing optimization module. The road-side real-time perception algorithm is used to obtain various information of traffic participants in real time, the high-precision cloud-side algorithm driven by a large model is used to enhance global perception and deep reasoning capabilities, the end-cloud collaborative fusion module is used to balance efficiency and accuracy to improve the robustness and generalization of the model, and the efficient post-processing optimization module is used to further improve the display accuracy and visual effects of the smart intersection.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] 1. The present invention proposes a multi-source collaborative intelligent traffic management method, which can use multi-channel videos in traffic scenes as input, and utilize multi-device collaboration and end-cloud collaboration technology to achieve real-time monitoring and refined management of complex traffic scenes.

[0084] 2. The real-time perception algorithm designed in the present invention can realize functions such as sensor calibration, unification of multi-channel perspectives, image feature extraction, instance detection and tracking, cross-lens ID matching, license plate detection, vehicle color detection, vehicle model detection, trajectory smoothing and prediction, and abnormal event detection.

[0085] Third, this invention combines high-precision cloud algorithms driven by large models and relies on the powerful reasoning ability of the multimodal large model (Qwen2-VL) to provide accurate external knowledge priors for the processing of challenging instance targets. In addition, the introduction of memory modules and depth estimation also further improves the accuracy of the end-to-end tracking algorithm.

[0086] 4. Based on the end-cloud collaborative fusion strategy, the present invention performs hybrid alignment of the position and semantics of the perception results of the road-side and cloud-side algorithms, and sets threshold hyperparameters to filter out redundant information, thereby achieving the goal of balancing efficiency and accuracy, and improving the robustness and generalization of the model.

[0087] 5. The present invention utilizes a post-processing logic optimization module based on knowledge prior guidance, and optimizes the trajectory of each target from the data level by combining the prior information of traffic rules and the historical data of perception results, thereby improving the visual analysis effect and interactive experience of the management method.

[0088] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A multi-source collaborative intelligent traffic management method, characterized in that: The method comprises the following steps: S1. Construct an affine transformation equation group to calibrate the camera's internal and external parameter matrix; S2. Obtain the target perception results from each perspective, realize the target coordinate system transformation based on the internal and external parameter matrices, obtain the target detection and tracking results from the global BEV perspective, and perform cross-lens matching and optimization to obtain the road-side global perception results; S3, input the task prompt words, road-side global perception results and original video stream into the Qwen2-VL model in the cloud to obtain the prior information of the color, license plate and model of the challenging target in each perspective; S4, the end-to-end cloud perception model takes four-channel video data as input, combines the prior information output by Qwen2-VL and the internal and external parameter matrix of S1, and outputs the cloud global perception results through feature extraction and attention mechanism; S5, performing position-semantic hybrid alignment on the road-side global perception result and the cloud-side global perception result to obtain an enhanced global perception result; S6. Global perception results after optimization and enhancement based on prior information of traffic rules and historical data of perception results.

2. A multi-source collaborative intelligent traffic management method according to claim 1, characterized in that: The specific steps of S4 are: Input the four-channel video data into the convolutional neural network of the cloud perception model for feature extraction, obtain the convolution features and basic depth information of the image, and reduce the number of channels of the convolution features to d. Then, cascade the depth information and the image convolution features with d channels as the input of the subsequent attention mechanism; The serialized image features are input into the attention mechanism of the cloud perception model. The encoder part of the attention mechanism uses eight sets of parallel encoding blocks to obtain 8 sets of query features, and the weighted query features are saved in the memory unit as weighted memory features. The cross attention mechanism in the decoder uses the weighted memory features as the current key and value, and finally the perception head is connected to the decoded output features to obtain the global perception result of the cloud.

3. A multi-source collaborative intelligent traffic management method according to claim 2, characterized in that: A set of query features is: Qbev t =Qo t ⊙Atten(∑(Qo t ,Cal)) Among them, Q t Represents the video encoding output of each channel, and Cal represents the internal and external parameter matrix characteristics.

4. The multi-source collaborative intelligent traffic management method according to claim 3 is characterized in that: The video encoding output of each channel is: in, is the serialized image feature, is the encoded external knowledge feature.

5. The multi-source collaborative intelligent traffic management method according to claim 4 is characterized in that: The weighted memory feature is: Mu=Qbev⊙Atten(Qbev) Among them, Qbev is 8 groups of query features.

6. The multi-source collaborative intelligent traffic management method according to claim 1 is characterized in that: In S5, if the road-side global perception result conflicts with the cloud-side global perception result, the cloud-side global target perception result is used as the enhanced global perception result.

7. A multi-source collaborative intelligent traffic management method according to claim 6, characterized in that: The position-semantic hybrid alignment is specifically as follows: The matching degree, cosine similarity and L2 distance of the road-side and cloud-side location information are calculated to obtain an optimization function, and the location-semantic hybrid alignment is performed based on the optimization function.

8. The multi-source collaborative intelligent traffic management method according to claim 7 is characterized in that: The optimization function is: Among them, T p and C p They are used to represent the instance locations in the road-side and cloud-side global perception results, Cos(T s , C s ) and L2(T s , C s ) represent cosine similarity and L2 distance respectively, T s and C s They are used to represent the semantic features extracted from the global perception results of the road side and the cloud side based on the text encoder, T and C represent the comprehensive perception results of the road side and the cloud side in the global coordinate system, and λ iou , cos , L2 is a hyperparameter.

9. The multi-source collaborative intelligent traffic management method according to claim 8 is characterized in that: The cosine similarity is: Among them, T i and C i Represent the perception results of the i-th instance on the road side and in the cloud side respectively.

10. A multi-source collaborative intelligent traffic management method according to claim 9, characterized in that: The L2 distance is:

Images