Lightweight traffic detection method based on audio-video fusion

Abstract

The light-weight traffic detection method based on audio-video fusion comprises the following steps: step 1, modeling the synchronous video frame and audio segment of the same traffic scene to obtain visual detection results containing candidate boundary boxes, categories and visual detection confidence, and audio probability representing whether vehicles exist in the scene; step 2, receiving the visual detection confidence and the audio probability, mapping them to a unified Logit decision scale and performing weighted fusion, and then forming a final vehicle detection set according to a fusion threshold; step 3, receiving the final vehicle detection set obtained in step 2, and reusing the audio probability representing whether vehicles exist in the scene to enhance track association, track confidence and track life cycle, and output a vehicle track set. The application utilizes audio scene priori to perform weighted correction on detection confidence or dynamic adjustment on detection threshold, and has the characteristics of small calculation amount in the fusion process, high modularization degree, strong real-time performance and strong adaptability to complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of traffic monitoring technology, specifically relating to a lightweight traffic detection method based on audio and video fusion. Background Technology

[0002] Early road traffic monitoring systems relied on video surveillance as their core, using cameras to capture images of road scenes and combining them with traditional image processing algorithms to complete vehicle target detection and traffic flow statistics.

[0003] With the rapid iteration and development of deep learning technology, convolutional neural networks have been gradually applied to traffic visual perception tasks, which has greatly improved the overall performance of tasks such as accurate vehicle detection and traffic anomaly event recognition.

[0004] To further improve the operational stability of traffic monitoring systems in complex scenarios such as nighttime, rain, fog, and backlighting, researchers have begun to introduce various types of sensor information to assist pure visual perception. Among these auxiliary sensing methods, acoustic sensors have gradually become a research hotspot due to their advantages such as low cost, low power consumption, and lack of limitation by lighting conditions. During vehicle operation, acoustic signals such as engine noise, tire-road friction noise, and aerodynamic noise are continuously generated, containing rich information about vehicle type, driving status, and traffic situation characteristics.

[0005] Thanks to continuous breakthroughs in multimodal perception technology, the research field is gradually exploring cross-modal fusion methods for visual and audio information to build a traffic perception system with stronger anti-interference capabilities and better robustness. At the same time, the technological maturity of deep learning in audio feature extraction and scene recognition provides solid theoretical and technical support for traffic acoustic signal analysis and noise event classification.

[0006] Mainstream existing technologies include: Object detection: Video-only (baseline method): This method uses only a video detection model to detect vehicle targets in each frame of the image and filters candidate detection boxes based on the detection confidence output by the video model.

[0007] The video detection model is used to detect vehicle targets in each frame of the image. The candidate detection boxes are filtered directly based on the detection confidence output by the model to obtain the final vehicle detection result.

[0008] WSF (Weighted Score Fusion): While keeping the target location and category information output by the video detection model unchanged, the scene-level vehicle existence probability output by the audio model is introduced, and the video detection confidence and audio prior are weighted and fused at the score level. Finally, the target-level screening is completed based on the fused score.

[0009] TGF (Gated Threshold Fusion): Based on the video detection results, it combines audio priors with the visual detection score of the current frame to construct gating coefficients, and dynamically adjusts the detection threshold according to the gating results, thereby realizing the gating enhancement of audio information on the visual detection screening process.

[0010] Target tracking: DeepSORT is a multi-target tracking method based on the joint association of appearance features and motion information. By introducing target appearance features for cross-frame matching, it has certain advantages in target occlusion and interactive scenarios.

[0011] ByteTrack: A multi-target tracking method based on hierarchical association of detection confidence. By fully utilizing information from low-confidence detection boxes, it improves target recall and is one of the more representative tracking-by-detection methods currently available.

[0012] OCSORT: A multi-target tracking method based on motion state modeling optimization. It improves target tracking stability by refining motion prediction and association strategies, and has good adaptability to short-term changes in target motion.

[0013] In existing technologies, Xiwen Li et al. disclosed a joint audio-visual idling vehicle detection method in the paper "Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies," published at the 2025 IEEE / CVF Winter Conference on Applications of Computer Vision Workshops (IEEE document number 10972572, DOI 10.1109 / WACVW65960.2025.00098). This method extracts video and audio features separately, maps the features of both modalities to a joint feature space through a bidirectional attention mechanism, and detects vehicle position and state based on the fused features, classifying vehicles into three categories: driving, idling, and engine off.

[0014] However, this existing technology primarily targets idling vehicle state detection. It fuses audio and video information end-to-end at the feature layer, but does not disclose a scheme to weight and correct video detection confidence using scene-level vehicle presence probabilities output by the audio model while maintaining the video detection bounding box position and category information. Furthermore, it does not disclose a scheme to construct gating coefficients based on audio priors and the current frame's visual detection score, and to dynamically adjust the vehicle candidate bounding box selection threshold based on these gating coefficients. Therefore, this method struggles to directly address the problem of vehicle candidate bounding boxes being mistakenly deleted due to low detection confidence at fixed thresholds under conditions of low light, occlusion, long distance, and small targets. In addition, this method requires jointly labeled audio and video data to train the end-to-end fusion network. When the audio is affected by non-vehicle noise, multiple vehicle sound sources, or environmental reverberation, erroneous audio features may further affect vehicle position and state prediction. This method also lacks an independent multi-object tracking module and cannot directly solve multi-object tracking problems such as vehicle occlusion, cross-frame identity association, low-confidence bounding box association, and identity switching. Summary of the Invention

[0015] To overcome the shortcomings of the existing technologies, the present invention aims to provide a lightweight traffic detection method based on audio-video fusion. This method has the advantages of using audio scene priors to weight and correct the detection confidence or dynamically adjust the detection threshold without changing the position and category information of the video candidate detection boxes. Furthermore, the fusion process has the characteristics of low computational load, high modularity, real-time performance, and strong adaptability to complex scenarios.

[0016] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A lightweight traffic detection method based on audio and video fusion includes the following steps; Step 1: Model synchronized video frames and audio segments of the same traffic scene using an audio-visual perception model to obtain visual detection results including candidate bounding boxes, categories and visual detection confidence, as well as audio probabilities representing the presence of vehicles in the scene, providing input for the decision layer fusion in Step 2; Step 2: The audio-visual fusion target detection method based on Logit space is used to receive the visual detection confidence and audio probability, map the two to a unified Logit decision scale and perform weighted fusion, and then form the final vehicle detection set according to the fusion threshold, providing target-level detection input for Step 3; Step 3: Multi-target tracking enhancement based on audio prior is used to receive the final vehicle detection set obtained in Step 2, and reuse the audio probability representing the presence of vehicles in the scene obtained in Step 1 to enhance trajectory association, trajectory confidence and trajectory lifecycle, and finally output the vehicle trajectory set.

[0017] The above steps are executed in the order of "perceptual modeling - detection fusion - temporal tracking": Step 1 generates basic multimodal observations; Step 2 completes confidence correction and target selection without changing the position and category of visual candidate boxes; Step 3 performs cross-frame association based on the detection results of Step 2; wherein, the audio scene prior of Step 1 is applied to both Step 2 and Step 3.

[0018] The audio-visual perception model includes a visual vehicle target detection model and an audio vehicle activity detection model. Step 1 specifically involves: Step 11): Using a visual vehicle target detection model, multi-scale visual features are extracted from the current video frame, and the location, category, and visual detection confidence of the target candidate box are output. Step 12): Using an audio vehicle activity detection model, acoustic features are extracted from the road environment audio clips corresponding to the current video moment, and the scene-level vehicle presence probability is output.

[0019] Steps 11) and 12) process spatial target information and scene event information respectively. The two are aligned on the time axis and independent of each other. The target-level result of step 11) and the scene-level probability of step 12) are input into step 2. The audio vehicle activity detection model does not directly modify the position and category of the visual candidate box.

[0020] Step 11) specifically involves: (1) Based on public transportation scenario datasets such as COCO, UA-DETRAC, and BDD100K, and combined with traffic target images collected in actual road environments, the original data is supplemented and expanded, and the target category set required for training is defined as follows: The above English categories correspond to pedestrians, bicycles, cars, motorcycles, buses, trucks, and tricycles, for a total of K=7 categories.

[0021] Based on this, the above public datasets are automatically filtered to remove images and their corresponding annotation files that do not contain any target category, so as to reduce the interference of irrelevant samples on model training; To address the issues of sample redundancy and scene mismatch after initial screening, the COCO dataset was downsampled, the UA-DETRAC dataset was sampled every other frame, and the BDD100K dataset was downsampled as a whole. After merging with the self-collected data, the training set, test set, and validation set were divided into three sets in an 8:1:1 ratio. The self-collected data is uniformly labeled, and the filtered public data is relabeled and the labels are aligned. Finally, a visual traffic target dataset with consistent category semantics and uniform label format is obtained for use in training, verification and testing of visual detection models. (2) Based on the YOLOv8 series models, the Backbone-Neck-Head detection framework was adopted, and its network structure was optimized for traffic monitoring scenarios. A visual vehicle target detection model yolo_track for vehicle detection and subsequent multi-target tracking was constructed: Backbone is responsible for extracting multi-scale visual features. It adopts a relatively high channel dimension configuration, and the number of output channels of the feature map at each stage is set to {32,64,128,256,512} to enhance the model's ability to express features of vehicle appearance details, scale changes and occlusion. The CSPlayer_2Conv structure is introduced by incorporating the characteristics of the YOLO detection head. The CSPlayer_2Conv structure divides the feature map into two parts, performs convolution transformation and residual connection on each part, and then fuses them in a subsequent stage. The Neck is used to fuse feature information from different levels, while the Head is used to complete target classification and bounding box regression tasks. Overall, it adopts the FPN-PAN structure and three-scale detection head commonly used in YOLOv8. Through bidirectional feature propagation from top to bottom and bottom to top, it achieves full fusion of multi-scale features and enhances the detection capability of small-scale and long-distance vehicle targets.

[0022] The visual detection model yolo_track is built based on the YOLOv8 series models. The YOLOv8 series models, arranged sequentially according to the data flow direction, include a Backbone, a Neck, and a Head. The Backbone is the main feature extraction network, the Neck is a multi-scale feature fusion network, and the Head is the target classification and bounding box regression network. The model features three-scale detection, ample feature reuse, and ease of edge deployment, and is used for vehicle detection, providing target-level observations for subsequent multi-target tracking. In Backbone, the input traffic image is downsampled stepwise and multi-scale feature extraction is performed. The number of output channels for the feature map at each stage is set to {32,64,128,256,512} to enhance the ability to represent vehicle appearance details, scale changes and occlusion status. The C2f module in the original YOLOv8 Backbone and Neck is replaced with the CSPrayer_2Conv module. The CSPrayer_2Conv consists of feature splitting, convolutional transformation, DarknetBottleneck residual processing, feature concatenation and terminal convolution, which allows some features to be directly retained and other features to be deeply transformed and then fused, thereby reducing redundant computation and maintaining the local feature expressiveness. The Neck uses an FPN-PAN structure, which fuses different scale features from the Backbone output through a bidirectional path from top to bottom and bottom to top. The Head performs class prediction and bounding box regression at three scales respectively, and finally outputs the candidate bounding box, class confidence and corresponding scale information for each video frame, and uses this output as the visual detection result of step 21).

[0023] Step 12) specifically involves: The audio vehicle activity detection model uses YamNet as the base network and combines it with the constructed traffic scene audio dataset for task adaptation and retraining. The audio task is simplified into an event-level binary classification of "car present" and "car absent" to represent the overall existence state of vehicle events in the scene. (1) Perform unified preprocessing on the continuous road environment audio collected synchronously with the video stream, convert the original audio into a single-channel waveform with a sampling rate of 16 kHz, and trim it into 2 s audio segments according to the sliding time window; fill zeros for segments less than 2 s, and cut the corresponding time window for segments longer than 2 s to form the standardized audio input in step 12). (2) The audio training set is constructed by using public datasets as the main data source and self-collected data as the supplementary data source. The public data includes traffic-related sounds in UrbanSound8K and VGGSound, and is uniformly mapped to two categories: "with cars" and "without cars". The self-collected road audio is used to supplement the silent, weak traffic sounds, wind noise and real road background sounds, so that the public data and the actual deployment scenario complement each other. (3) The YamNet audio model includes waveform preprocessing, Mel spectrum feature transformation, YamNet feature extraction network composed of depthwise separable convolutional units, global pooling layer, and binary classification fully connected layer. For a batch of B audio waveforms, the batch input is split into B single-channel waveforms according to the sample dimension. The acoustic feature vectors are obtained one by one using the YamNet feature extraction network with shared parameters. Then, the feature vectors are stacked according to the batch dimension to form a feature matrix and output B groups of "car / no car" probabilities through fully connected layer and Softmax layer. This realizes the feature extraction and unified classification of one by one of multiple audio waveforms in a batch.

[0024] Step 2 specifically involves: Step 21): Model the audio and video detection results to organize the visual detection results output in Step 11) and the audio detection results output in Step 12) at the same time t. Construct a vehicle candidate set B(t) from the visual detection results and obtain the smoothed audio vehicle existence probability from the audio detection results after smoothing and hysteresis processing. _a(t); Step 22): An audio-video fusion algorithm based on Logit (logistic probability) space is used to receive B(t) from step 21) and _a(t), the visual confidence of each candidate box is corrected on a unified decision scale to obtain the fusion confidence s_i^f(t); Step 23): Set the fusion confidence threshold. Construct the final vehicle detection set.

[0025] Step 23) is used to compare the fusion confidence obtained in step 22) with the fusion threshold τ_f, retain the candidate boxes that meet the conditions and construct the final vehicle detection set B_f(t), whose output is used as the tracking input in step 31).

[0026] Step 21) Complete the time alignment and structured modeling of multimodal inputs; Step 22) Complete the cross-modal confidence fusion; Step 23) Complete the target-level threshold decision. The three are connected in sequence.

[0027] Step 21) specifically involves: (1) An improved model based on YOLOv8 was used as the video-side detection model. The model was retrained for road monitoring scenarios, and the detection categories were redefined into 7 categories to construct a vehicle candidate set. It includes bounding box location and confidence information; The seven target categories are person, bicycle, car, motorcycle, bus, truck, and tricycle, which correspond to pedestrians, bicycles, cars, motorcycles, buses, trucks, and tricycles, respectively. The vehicle-related category set C_veh consists of the six categories other than person, and the subsequent audio vehicle prior is only used for the vehicle-related categories. At any moment The visual model generates a fixed number of candidate predictions for the input image, where the input image is set to... In this case, its original output is represented as: Y(t) is obtained by forward inference of the current video frame by the yolo_track model in step 11); when the input size is 3×640×640, the three detection scales correspond to 80×80, 40×40 and 20×20 spatial locations, and the total number of candidates is 80×80+40×40+20×20=8400, so Y(t)∈R^(8400×(4+K)); Where t represents the current video time, K represents the total number of target categories, and in this embodiment K=7; each row of Y(t) contains 4 bounding box regression parameters and K category confidences, and 8400 is the total number of all spatial locations at the three detection scales.

[0028] Furthermore, the first The candidate predictions are denoted as: in: This represents the position and scale parameters of the candidate bounding box; while: This indicates the confidence vector (K=7) representing the category to which the candidate box belongs, where Corresponding to the Class confidence level.

[0029] Where i∈{1,…,8400} is the candidate box number, and k∈{1,…,K} is the category number; =[ [] represents the center coordinates, width, and height of the i-th candidate box; This indicates the confidence level that the candidate box belongs to the k-th class; K is the total number of classes.

[0030] Determine the most likely category and corresponding category confidence for each candidate box, and use the maximum a posteriori category selection method, i.e., for the th... Each candidate box defines its predicted category as follows: And define the corresponding category confidence level as: The above represent the category prediction results and confidence levels of the candidate bounding box, respectively. Based on this, the candidate predictions are further organized into a set of detection boxes: To maintain consistency with the goals of subsequent audio-visual fusion modules, from... Filter vehicle-related categories to construct a candidate vehicle set: in This represents the predicted category corresponding to the maximum confidence of the i-th candidate box. D(t) represents the visual detection confidence score for this category; D(t) represents the set of all candidate detection boxes. Let D(t) represent the set of vehicle-related categories, and let B(t) represent the set of vehicle candidates selected from D(t).

[0031] (2) In the time dimension, the audio detection module uses a sliding time window to analyze continuous environmental audio. Let the audio segment corresponding to time t be... This yields the audio probability output in the time dimension. , Furthermore, a smoothing and hysteresis mechanism is introduced to obtain the smooth audio probability. and their corresponding states This is considered a stable estimate of the vehicle's activity status over a period of time, and serves as important scenario-level prior information for subsequent audio-visual fusion at the decision-making level.

[0032] Where a(t) represents the 2 s audio segment corresponding to time t; = This represents the probability of the original vehicle's existence as output in step 12); α∈(0,1) is the time smoothing coefficient. The probability of a vehicle existing after smoothing; ∈{0,1} represents a hysteresis state; θ_on and θ_off are the state-on and state-off thresholds, respectively, and θ_on>θ_off. The smoothing mechanism is used to suppress single-probability fluctuations, while the hysteresis mechanism is used to prevent the vehicle's state from frequently switching near the threshold.

[0033] Step 22) specifically involves: (1) Introducing at the decision-making level As a globally consistent prior information, it contributes to the confidence level of visual detection. Perform correction; Use the smoothed audio probability obtained in step 21) Visual confidence of the i-th vehicle candidate box Without changing the candidate box and its categories Confidence levels are only adjusted at the decision-making level; (2) Map the video detection confidence and audio scene probability from the probability space to the Logit space, and construct a weighted linear fusion model for the th... The visual confidence level of each vehicle candidate detection bounding box The Logit representation is defined as: Audio scene-level vehicle presence probability The Logit is represented as: Construct the following weighted linear fusion model: in The fusion weights for audio information are used to adjust the strength of the influence of audio scene priors on the visual detection results. When =0, the fusion model degenerates into pure visual detection; as As the value increases, the role of audio detection results in fusion decision-making gradually strengthens; in, Visual confidence Logit value, To smooth audio probability Logit value, The weighted fusion Logit value is the audio fusion weight, where λ≥0 represents the audio fusion weight, and log represents the natural logarithm. To avoid numerical overflow caused by probabilities being 0 or 1, the input probability can be limited to [ε, 1] during actual calculation. Within ε], ε is a pre-defined small positive number; (3) Merge Logit values By mapping back to the probability space using the Sigmoid function, we obtain the output of step 22) – the fusion confidence score. ), and pass it to step 23) for detection box filtering; .

[0034] Step 23) specifically involves: Set fusion confidence threshold Construct the final vehicle detection set: in, ∈(0,1) is the fusion confidence threshold; Represents the final set of vehicle detections at time t; , These represent the position and scale, category, and fusion confidence of the retained candidate boxes, respectively. Only keep ≥ The vehicle candidate boxes are used as input for multi-target tracking in step 31); Filter the test results at the target level: when At lower levels, audio Logit suppresses low-confidence visual candidate boxes, reducing false detections in car-free scenarios; when When the level is high, audio priors improve the scores of vehicle candidate boxes that are realistic but have low visual confidence, reducing the risk of missed detections in low-light, occluded, and long-distance scenes.

[0035] Step 3 specifically involves: Step 31): Multi-target tracking input, used to receive the final vehicle detection set output from step 23). Complete trajectory prediction, detection-trajectory association and state update to form the current trajectory set T(t); Step 32): Trajectory confidence modeling is used to form visual trajectory confidence based on the association detection results, and it is fused with the smoothed audio probability from step 21) in the Logit space to obtain trajectory fusion confidence; Step 33): The trajectory lifecycle is dynamically adjusted according to the audio scene prior, which is used to change the maximum number of consecutive lost frames of the trajectory based on the smooth audio probability, so that the trajectory preservation strategy matches the current vehicle activity state.

[0036] Step 31) Establish a basic trajectory; Step 32) Evaluate and correct the reliability of the trajectory; Step 33) Control the trajectory to retain or terminate based on the scenario prior. Steps 32) and 33) are both executed based on the trajectory state of Step 31) and jointly output a stable vehicle trajectory.

[0037] Step 31) specifically involves: Using the TBD method, the target position is first predicted using state estimation methods such as Kalman filtering. Then, the association cost is constructed by combining IOU. Finally, the Hungarian algorithm or its variants are used to match the trajectory with the detection and update the trajectory state. The vehicle target is continuously associated over time to form a trajectory set. ; The TBD stands for Tracking-by-Detection, i.e., detection-based multi-target tracking; IOU stands for Intersection over Union. Step 31) Take the result from step 23) As a detection observation, Kalman prediction is performed on the existing trajectory, and an association cost is constructed based on the IOU between the predicted bounding box and the detection bounding box, appearance features, or detection confidence. Then, the matching is completed using the Hungarian algorithm. DeepSORT, ByteTrack, or OCSORT can all be used as specific trackers for this step.

[0038] in, Indicates the first The state of a trajectory includes the target's spatial location, motion state, and trajectory confidence information.

[0039] in, Let be the number of valid trajectories at time t. The state of the k-th trajectory can include the target position, velocity, trajectory age, number of consecutive lost frames, and trajectory confidence; T(t) is the set of all valid trajectories.

[0040] Step 32) specifically involves: Let the first The trajectory at time The confidence level of the visual trajectory is Mapped to the Logit space, and combined with the mapping of audio scene probability in the Logit space under a unified Logit decision scale, a trajectory-level fusion model is constructed. Finally, the confidence of the fused trajectory is mapped back to the probability space through Sigmoid. in, ∈[0,1] represents the confidence level of the visual trajectory obtained by the k-th trajectory based on the current and historical association detection; Set its Logit value; The trajectory Logit after fusion with audio Logit; λ represents the confidence level of trajectory fusion after Sigmoid mapping; λ has the same meaning as the audio fusion weight in step 22) or can be set separately; the trajectory fusion confidence level is used for subsequent trajectory confirmation, maintenance and termination decisions; The Logit representation of visual trajectory confidence is: Trajectory-level fusion model: Mapping back to probability space: .

[0041] Step 33) specifically involves: Maximum number of consecutive frames allowed in the trajectory , Based on the number of frames to be lost tolerable. For audio adjustment coefficients, when A higher value indicates continuous vehicle activity in the scene, and the system will appropriately extend the trajectory retention time to improve trajectory continuity under occlusion conditions; when When the confidence level is low, the system will terminate the trajectory with low confidence level more quickly, thereby reducing the accumulation of false trajectories and false alarms; Where L(t) is the maximum number of consecutive frames allowed at time t, L_0 is the basic frame loss tolerance without audio adjustment, and β≥0 is the audio adjustment coefficient. The smoothed vehicle presence probability obtained in step 21) is denoted as L(t). In actual execution, L(t) can be rounded and limited to preset upper and lower limits; the trajectory retention time is extended when the vehicle activity probability is high, and shortened when the vehicle activity probability is low.

[0042] The beneficial effects of this invention are: This invention, while maintaining the position and category of visual candidate boxes, fuses audio scene priors with visual detection confidence in the Logit space. When real vehicles generate low visual confidence due to occlusion, low light, or long-distance imaging, a higher probability of vehicle presence in the audio field can improve the fused confidence, thereby reducing the possibility of candidate boxes being mistakenly deleted by a fixed threshold and lowering the risk of missed vehicle detection.

[0043] The audio branch of this invention does not depend on scene lighting and forms a stable vehicle activity prior through time smoothing and hysteresis state machine; the visual branch maintains the vehicle spatial positioning capability through three-scale feature extraction and FPN-PAN feature fusion. The two complement each other at the decision layer, which helps to improve the environmental adaptability and comprehensive robustness in complex traffic scenarios such as night, backlight, rain and fog.

[0044] This invention further applies the smooth audio vehicle presence probability to trajectory confidence fusion and trajectory lifecycle adjustment: when the vehicle activity probability is high, the maximum number of consecutive lost frames is extended so that the trajectory can be preserved during short-term occlusion; when the vehicle activity probability is low, the termination of low-confidence trajectories is accelerated, thereby taking into account trajectory continuity, identity consistency and false trajectory suppression.

[0045] This invention adopts a decision layer fusion method in the target detection backend, without re-regressing the candidate box position and category, but only adding scalar operations such as probability smoothing, Logit transformation, weighting and Sigmoid mapping; the audio model adopts a depthwise separable convolutional structure and can be modularly combined with existing DeepSORT, ByteTrack or OCSORT trackers, so the overall computational overhead is small and it is easy to deploy in embedded terminals and roadside edge devices. Attached Figure Description

[0046] Figure 1 This is a flowchart of the audio-video fusion algorithm of the present invention.

[0047] Figure 2 This is the network structure for the video model of the present invention.

[0048] Figure 3 This is the network structure of the audio model of the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings.

[0050] like Figure 1 As shown, this invention operates sequentially through a video branch, an audio branch, an audio-video fusion branch, and a multi-target tracking branch. The video input is processed by a visual target detection model to obtain vehicle candidate detection boxes {( , The audio input is processed by a vehicle activity classification model, time smoothing, and a hysteresis state machine to obtain the scene-level vehicle presence probability. The two are fused at the decision level in the Logit space, and the final detection set is obtained through Sigmoid mapping and threshold filtering. ; Input online multi-target tracker, and simultaneously It is used for trajectory confidence fusion and trajectory lifecycle adaptation, and finally outputs a set of vehicle trajectories.

[0051] A lightweight traffic detection method based on audio and video fusion includes the following steps; Step 1: Audio and Video Perception Model Design Step 11): Visual vehicle target detection model (1) Based on the existing public transportation scenario dataset and combined with traffic target images collected in the actual road environment, the original data is supplemented and expanded, and the target category set required for training is defined as follows: Based on this, public datasets are automatically filtered to remove images and their corresponding annotation files that do not contain any of the aforementioned target categories, thereby reducing the interference of irrelevant samples on model training. After initial screening, the three types of public datasets exhibit issues such as sample redundancy and scene mismatch. The COCO dataset is downsampled, the UA-DETRAC dataset is sampled every other frame, and the BDD100K dataset is downsampled overall; the jointly self-collected dataset is then divided into training, testing, and validation sets in an 8:1:1 ratio.

[0052] After determining the data scale, the self-collected data is uniformly labeled, and the selected public dataset is relabeled and the labels are aligned to solve the inconsistency in labeling granularity and semantic definition between different datasets.

[0053] (2) Based on the YOLOv8 series models, the Backbone-Neck-Head detection framework was adopted, and its network structure was optimized for traffic monitoring scenarios. A visual detection model yolo_track for vehicle detection and subsequent multi-target tracking was constructed: Backbone is responsible for extracting multi-scale visual features. It adopts a relatively high channel dimension configuration, and the number of output channels of the feature map at each stage is set to {32,64,128,256,512} to enhance the model's ability to express features of vehicle appearance details, scale changes and occlusion. The C2f module in the Backbone and Neck structures of the original YOLOv8 was replaced with a CSPlayer_2Conv structure, which is more adapted to the characteristics of the YOLO detection head. This module is based on the CSP (CrossStage Partial) design concept, dividing the feature map into two parts, performing convolutional transformations and residual connections on each part separately, and then fusing them in subsequent stages. This reduces computational redundancy while maintaining strong feature representation capabilities. By introducing Darknet Bottleneck, the model maintains the advantages of cross-stage feature splitting and fusion while enhancing the stable extraction capability of local features, thus helping to improve the consistency of vehicle target detection in complex traffic environments. The Neck is used to fuse feature information from different levels, while the Head is used to complete object classification and bounding box regression tasks. It adopts the FPN-PAN structure and three-scale detection head commonly used in YOLOv8. Through bidirectional feature propagation from top to bottom and bottom to top, it achieves full fusion of multi-scale features, enhancing the detection capability for small-scale and long-distance vehicle targets. At the same time, it retains the three-scale detection mechanism of YOLOv8 and sets the number of detection categories to correspond to the target categories in the constructed dataset, making the model more focused on the vehicle target detection task, reducing the additional overhead of multi-class classification, and thus further improving detection efficiency.

[0054] Step 12): Audio vehicle activity detection model YamNet was selected as the audio feature extraction and modeling network, and task adaptation and retraining were performed in conjunction with the constructed traffic scene audio dataset. The audio task was simplified into an event-level binary classification of "car present" and "car absent" to represent the overall existence state of vehicle events in the scene.

[0055] (1) In terms of audio dataset construction, the audio input format and data source are designed in a unified manner. The audio model uses single-channel audio with a sampling rate of 16kHz as input, and the audio segments are trimmed or padded with zeros to a fixed length of 2s.

[0056] (2) Regarding data sources, an audio dataset was constructed using a strategy of primarily public datasets and supplemented by private datasets to improve the model's generalization and detection stability in traffic scenarios. UrbanSound8K and VGGSound, two major public audio datasets, were selected, and traffic-related sounds were filtered and uniformly classified into two categories: those with vehicles and those without. In addition, private road audio was collected on-site to supplement quiet and weak traffic sound samples, thereby enhancing the model's real-world scene detection performance.

[0057] (3) In terms of building the audio model, we refer to the YamNet audio classification model proposed by Google as the basic network structure, modularly encapsulate the YamNet feature extraction network, design a feature extraction layer for batch audio input, and realize the feature extraction of multiple audio waveforms in a batch while keeping the original acoustic feature modeling capability unchanged.

[0058] Step 2: Logit space-based audio-video fusion target detection method: Step 21) Modeling audio and video detection results (1) An improved model based on YOLOv8 was used as the video-side detection model. The model was retrained for road monitoring scenarios, and the detection categories were redefined into 7 categories to construct a vehicle candidate set. It includes bounding box location and confidence information.

[0059] At any moment The visual model generates a fixed number of candidate predictions for the input image, where the input image is set to... In this case, its original output can be represented as: Here, 8400 represents the number of candidate predictions, which is determined by the network structure and the size of the multi-scale feature map; each row corresponds to one candidate prediction, which includes 4 bounding box regressions and 7 class confidences.

[0060] Furthermore, the first The candidate predictions are denoted as: in: This represents the position and scale parameters of the candidate bounding box; while: This indicates the confidence vector (K=7) representing the category to which the candidate box belongs, where Corresponding to the Class confidence level.

[0061] Determine the most likely category and corresponding category confidence for each candidate box, using the maximum a posteriori (MAP) class selection method. That is, for the... Each candidate box defines its predicted category as follows: And define the corresponding category confidence level as: The above represent the category prediction results and confidence levels of the candidate bounding box, respectively. Based on this, the candidate predictions are further organized into a set of detection boxes: To maintain consistency with the goals of subsequent audio-visual fusion modules, from... Filter vehicle-related categories to construct a candidate vehicle set: (2) In the time dimension, the audio detection module uses a sliding time window to analyze continuous environmental audio. Let the audio segment corresponding to time t be... This yields the audio probability output in the time dimension. ; Furthermore, a smoothing and hysteresis mechanism is introduced to obtain the smooth audio probability. and their corresponding states This is considered a stable estimate of the vehicle's activity status over a period of time, and serves as important scenario-level prior information for subsequent audio-visual fusion at the decision-making level.

[0062] Step 22) Logit space-based audio-video fusion algorithm; (1) Introducing at the decision-making level As a globally consistent prior information, it contributes to the confidence level of visual detection. Perform corrections.

[0063] (2) Map the video detection confidence and audio scene probability from the probability space to the Logit space to construct a weighted linear fusion model. For the , The visual confidence level of each vehicle candidate detection bounding box The Logit representation is defined as: Audio scene-level vehicle presence probability The Logit is represented as: Construct the following weighted linear fusion model: in The fusion weights for audio information are used to adjust the strength of the influence of audio scene priors on the visual detection results. When =0, the fusion model degenerates into pure visual detection; as As the number of [something] increases, the role of audio detection results in fusion decision-making gradually strengthens.

[0064] (3) The fused Logit value is mapped back to the probability space through the Sigmoid function to obtain the final fusion confidence, which serves as the basis for subsequent detection box screening and suppression.

[0065] Step 23): Set the fusion confidence threshold. Construct the final vehicle detection set: At the target level, the detection results are filtered. When the audio module determines that the probability of vehicle activity in the current scene is low, the vehicle candidate boxes with low confidence in the visual detection will be further suppressed. When the audio detection results strongly indicate the presence of vehicle activity, the confidence of the visual detection results will be enhanced overall, thereby reducing the risk of missed detections.

[0066] Step 3: Enhanced multi-target tracking based on audio priors: Step 31) Multi-target tracking input The TBD method first predicts the target position using state estimation methods such as Kalman filtering, then constructs an association cost using methods like IOU, and finally completes the matching between the trajectory and the detection using the Hungarian algorithm or its variants, updating the trajectory state. Continuous association of the vehicle target over time forms a trajectory set. .

[0067] in, Indicates the first The state of a trajectory includes information such as the target's spatial location, motion state, and trajectory confidence.

[0068] Step 32): Trajectory confidence modeling: Let the first The trajectory at time The confidence level of the visual trajectory is The trajectory is mapped to the Logit space and combined with the mapping of audio scene probabilities in the Logit space under a unified Logit decision scale to construct a trajectory-level fusion model. Finally, the confidence of the fused trajectory is mapped back to the probability space through a Sigmoid function to enhance the temporal stability of the trajectory.

[0069] The Logit representation of visual trajectory confidence is: Trajectory-level fusion model: Mapping back to probability space: Step 33): The trajectory lifecycle is dynamically adjusted according to the audio scene prior. Maximum number of consecutive frames allowed in the trajectory , Based on the number of frames to be lost tolerable. For audio adjustment coefficients, when A higher value indicates continuous vehicle activity in the scene, and the system will appropriately extend the trajectory retention time to improve trajectory continuity under occlusion conditions; when At lower confidence levels, the system will terminate low-confidence trajectories more quickly, thereby reducing the accumulation of false trajectories and false alarms. For example... Figure 2 As shown, the input to the visual vehicle target detection model is a 3×640×640 video image. The backbone consists of convolutional layers, a CSPlayer_2Conv module, and an SPPF module, where SPPF is a fast spatial pyramid pooling module. The neck forms an FPN-PAN bidirectional feature fusion path through upsampling, feature concatenation, and the CSPlayer_2Conv module, where FPN-PAN is a feature pyramid network and a path aggregation network. The head outputs class confidence and bounding box regression values ​​at three scales. This structure fully integrates shallow detail features with deep semantic features and outputs three-scale candidate predictions to step 21).

[0070] like Figure 3 As shown, the audio vehicle activity detection model first performs fixed-length cropping or zero-padding on the original audio, then converts the waveform into Mel-spectral features. The YamNet backbone extracts acoustic features through multi-level deep separable convolutions, and after global pooling and binary classification fully connected layers, the probabilities of "car present" and "no car present" are output by the Softmax normalized exponential function. During batch input, each audio sample shares the same YamNet parameters and features are extracted one by one, then stacked and classified uniformly.

[0071] Figure 2 The visual model outputs target-level location, category, and confidence. Figure 3 The audio model outputs the scene-level vehicle presence probability, and the two are in Figure 1 The Logit fusion module shown integrates these elements and together provides detection information and scene priors for online multi-target tracking.

[0072] In the visual vehicle target detection model, relevant traffic targets were collected, focusing on vehicles such as tricycles that are missing in the category system of public datasets, as well as samples of two-wheeled traffic participants.

[0073] The audio task is simplified into an event-level binary classification, distinguishing only between "with a car" and "without a car" scenarios, thus reducing task complexity.

[0074] Mapping video detection confidence and audio scene probability from probability space to Logit space effectively mitigates fusion bias caused by nonlinearity in probability space.

[0075] The probability of vehicle presence at the smooth scene level output by the audio module is extracted and used as a globally consistent prior constraint, which is then incorporated into the trajectory-level decision-making process to improve the rationality of the decision.

[0076] Applications of this invention: Monitoring and Maintenance of Smart City Traffic Arteries: In the construction and maintenance scenarios of smart city traffic arteries, construction areas are prone to complex visual interference due to factors such as road occupancy and temporary barriers. Coupled with the effects of low-visibility environments such as nighttime, rain, and fog, traditional vehicle detection methods are prone to missed detections and false detections, leading to a significant decline in detection performance. Based on the vehicle detection and tracking technology proposed in this paper, the limitations of low-visibility environments can be effectively overcome, accurately identifying and tracking passing vehicles in real time. This dynamically identifies safety hazards such as vehicles illegally entering or stopping within the construction area. Once an anomaly is detected, an alarm is immediately triggered and reported to the monitoring management platform, providing comprehensive protection for the safety of construction personnel. Simultaneously, it ensures the continuity and stability of traffic monitoring during construction, reducing safety accidents and traffic congestion caused by detection failures.

[0077] Enforcement of Speeding Violations on Key Road Sections: Speeding on key road sections (such as highways, urban expressways, and accident-prone areas) is a major cause of traffic accidents. Low-visibility environments such as nighttime, rain, fog, and strong sunlight / backlighting can easily lead to inaccurate vehicle tracking and significant instantaneous speed measurement errors by traditional speed measuring equipment, affecting the fairness and reliability of enforcement. Utilizing high-precision vehicle detection and tracking capabilities, this system can clearly capture vehicle trajectories and instantaneous speeds in various low-visibility environments. This effectively compensates for the performance shortcomings of traditional speed measuring equipment, significantly improving instantaneous speed measurement accuracy and violation identification accuracy, ensuring precise enforcement of speeding violations, creating an effective deterrent, further regulating road traffic order, and reducing the traffic accident rate.

[0078] Emergency Early Warning for Remote Mountain Roads: Remote mountain roads generally suffer from complex road conditions, poor lighting, and variable weather. Low-visibility environments such as nighttime, rain, snow, and heavy fog are frequent. Traditional vehicle detection methods are prone to failure, failing to capture vehicle movement status in a timely manner and making it difficult to provide early warnings of potential collisions, falls, and other accidents, potentially leading to serious personal and property damage. By applying vehicle detection and tracking technology, vehicle trajectories, speeds, and abnormal behaviors such as passing and lane obstruction can be reliably captured even in low-visibility environments. Combined with comprehensive analysis of road condition information, timely audible and visual warning signals can be issued to remind drivers to pay attention to driving safety. Simultaneously, abnormal information is reported to emergency management departments, providing data support for rapid response to emergencies, effectively avoiding safety risks caused by detection failures, and ensuring safe passage on remote mountain roads.

[0079] Management of Closed Roads in Industrial Parks: Vehicle management in closed roads of industrial parks, logistics parks, and campuses is characterized by enclosed environments, complex vehicle types, and diverse traffic needs. Furthermore, insufficient lighting and tree obstructions in some areas can create localized low-visibility environments, making it difficult for traditional management methods to accurately identify and track vehicles. This can lead to problems such as unauthorized entry of external vehicles and haphazard parking of internal vehicles, impacting park management efficiency and security. By leveraging vehicle detection and tracking technology, various types of vehicles (including motor vehicles and non-motor vehicles) within the park can be accurately identified even in low-visibility environments, and their movement trajectories can be tracked in real time. This effectively avoids the performance degradation of traditional detection methods in complex environments, enabling dynamic, end-to-end control of vehicles within the park. This improves the precision and reliability of closed road management, ensuring the safety of personnel and property within the park.

Claims

1. A lightweight traffic detection method based on audio-video fusion, characterized in that, Includes the following steps; Step 1: Model synchronized video frames and audio segments of the same traffic scene using an audio-visual perception model to obtain visual detection results including candidate bounding boxes, categories and visual detection confidence, as well as audio probability representing the presence of vehicles in the scene. Step 2: The Logit space-based audio-video fusion target detection method receives the visual detection confidence and audio probability, maps the two to a unified Logit decision scale and performs weighted fusion, and then forms the final vehicle detection set based on the fusion threshold; Step 3: Multi-target tracking enhancement based on audio prior. Receive the final vehicle detection set obtained in Step 2, and reuse the audio probability representing the presence or absence of vehicles in the scene obtained in Step 1 to enhance trajectory association, trajectory confidence, and trajectory lifecycle, and finally output the vehicle trajectory set.

2. The lightweight traffic detection method based on audio-video fusion according to claim 1, characterized in that, The audio-visual perception model includes a visual vehicle target detection model and an audio vehicle activity detection model. Step 1 specifically involves: Step 11): Using a visual vehicle target detection model, multi-scale visual features are extracted from the current video frame, and the location, category, and visual detection confidence of the target candidate bounding box are output. Step 12): Using an audio vehicle activity detection model, acoustic features are extracted from the road environment audio clips corresponding to the current video moment, and scene-level vehicle presence probability is output.

3. The lightweight traffic detection method based on audio-visual fusion according to claim 2, characterized in that, Step 11) specifically involves: (1) Based on the public transportation scene datasets of COCO, UA-DETRAC, and BDD100K, and combined with traffic target images collected in actual road environments, the original data is supplemented and expanded, and the target category set required for training is defined as follows: The above English categories correspond to pedestrians, bicycles, cars, motorcycles, buses, trucks, and tricycles, respectively, for a total of K=7 categories; Based on this, the aforementioned public datasets are automatically filtered; To address the issues of sample redundancy and scene mismatch after initial screening, the COCO dataset was downsampled, the UA-DETRAC dataset was sampled every other frame, and the BDD100K dataset was downsampled as a whole. After being merged with the self-collected data, the dataset was divided into training, testing, and validation sets. The self-collected data is uniformly labeled, and the filtered public data is relabeled and the labels are aligned. Finally, a visual traffic target dataset with consistent category semantics and uniform label format is obtained for use in training, verification and testing of visual detection models. (2) Based on the YOLOv8 series models, and using the Backbone-Neck-Head detection framework, a visual vehicle target detection model yolo_track is constructed for vehicle detection and subsequent multi-target tracking: Backbone is responsible for extracting multi-scale visual features, using a relatively high channel dimension configuration; The CSPlayer_2Conv structure is introduced by incorporating the characteristics of the YOLO detection head. The CSPlayer_2Conv structure divides the feature map into two parts, performs convolution transformation and residual connection on each part, and then fuses them in a subsequent stage. The Neck is used to fuse feature information from different levels, while the Head is used to complete target classification and bounding box regression tasks. Through bidirectional feature propagation from top to bottom and bottom to top, it achieves full fusion of multi-scale features and enhances the detection capability for small-scale and long-distance vehicle targets.

4. The lightweight traffic detection method based on audio-visual fusion according to claim 3, characterized in that, Step 12) specifically involves: The audio vehicle activity detection model uses YamNet as the base network and combines it with the constructed traffic scene audio dataset for task adaptation and retraining. The audio task is simplified into an event-level binary classification of "car present" and "car absent" to represent the overall existence state of vehicle events in the scene. (1) Perform unified preprocessing on the continuous road environment audio collected synchronously with the video stream, convert the original audio into a single-channel waveform with a sampling rate of 16 kHz, and trim it into 2 s audio segments according to the sliding time window; fill zeros for segments less than 2 s, and cut the corresponding time window for segments longer than 2 s to form the standardized audio input in step 12). (2) The audio training set is constructed by using public datasets as the main data source and self-collected data as the supplementary data source. The public data includes traffic-related sounds from UrbanSound8K and VGGSound, and is uniformly mapped to two categories: "with cars" and "without cars". The self-collected road audio is used to supplement the silent, weak traffic sounds, wind noise and real road background sounds, so that the public data and the actual deployment scenario complement each other. (3) The YamNet audio model includes waveform preprocessing, Mel spectrum feature transformation, YamNet feature extraction network composed of depthwise separable convolutional units, global pooling layer and binary classification fully connected layer. For a batch of B audio waveforms, the batch input is split into B single-channel waveforms according to the sample dimension. The YamNet feature extraction network with shared parameters is used to obtain acoustic feature vectors one by one. Then, the feature matrix is ​​stacked according to the batch dimension and outputs the probability of "car / no car" in B groups through the fully connected layer and Softmax layer. In this way, feature extraction and unified classification of multiple audio waveforms in a batch are achieved.

5. The lightweight traffic detection method based on audio-visual fusion according to claim 4, characterized in that, Step 2 specifically involves: Step 21): Model the audio and video detection results to organize the visual detection results output in Step 11) and the audio detection results output in Step 12) at the same time t. Construct a vehicle candidate set B(t) from the visual detection results and obtain the smoothed audio vehicle existence probability from the audio detection results after smoothing and hysteresis processing. _a(t); Step 22): A Logit-space-based audio-video fusion algorithm is used to receive B(t) from step 21) and _a(t), the visual confidence of each candidate box is corrected on a unified decision scale to obtain the fusion confidence s_i^f(t); Step 23): Set the fusion confidence threshold. Construct the final vehicle detection set.

6. The lightweight traffic detection method based on audio-visual fusion according to claim 5, characterized in that, Step 21) specifically involves: (1) An improved model based on YOLOv8 was used as the video-side detection model. The model was retrained for road monitoring scenarios, and the detection categories were redefined into 7 categories to construct a vehicle candidate set. It includes bounding box location and confidence information; The seven target categories are person, bicycle, car, motorcycle, bus, truck, and tricycle, which correspond to pedestrians, bicycles, cars, motorcycles, buses, trucks, and tricycles, respectively. The vehicle-related category set C_veh consists of the six categories other than person, and the subsequent audio vehicle prior is only used for the vehicle-related categories. At any moment The visual model generates a fixed number of candidate predictions for the input image, where the input image is set to... In this case, its original output is represented as: The Y(t) is obtained by forward inference of the current video frame by the yolo_track model in step 11); Where t represents the current video moment, and K represents the total number of target categories; the th The candidate predictions are denoted as: in: This represents the position and scale parameters of the candidate bounding box; while: This represents the confidence vector indicating which category the candidate box belongs to, where Corresponding to the Class confidence; Where i∈{1,…,8400} is the candidate box number, and k∈{1,…,K} is the category number; =[ [] represents the center coordinates, width, and height of the i-th candidate box; This indicates the confidence level that the candidate box belongs to the k-th class; K is the total number of classes. Determine the most likely category and corresponding category confidence for each candidate box, and use the maximum a posteriori category selection method, i.e., for the th... Each candidate box defines its predicted category as follows: And define the corresponding category confidence level as: The above represent the category prediction results and confidence levels of the candidate bounding boxes, respectively; based on this, the candidate predictions are further organized into a set of detection boxes: To maintain consistency with the goals of subsequent audio-visual fusion modules, from... Filter vehicle-related categories to construct a candidate vehicle set: in This represents the predicted category corresponding to the maximum confidence of the i-th candidate box. D(t) represents the visual detection confidence score for this category; D(t) represents the set of all candidate detection boxes. Let D(t) represent the set of vehicle-related categories, and let B(t) represent the set of candidate vehicles selected from D(t). (2) In the time dimension, the audio detection module uses a sliding time window to analyze continuous environmental audio. Let the audio segment corresponding to time t be... This yields the audio probability output in the time dimension. ; Furthermore, a smoothing and hysteresis mechanism is introduced to obtain the smooth audio probability. and their corresponding states This is considered a stable estimate of the vehicle's activity status over a period of time and serves as important scenario-level prior information for subsequent audio-visual fusion at the decision-making level. Where a(t) represents the 2s audio segment corresponding to time t; = This represents the probability of the original vehicle's existence as output in step 12); α∈(0,1) is the time smoothing coefficient. The probability of a vehicle existing after smoothing; ∈{0,1} represents a hysteresis state; θ_on and θ_off are the state opening and closing thresholds, respectively, and θ_on>θ_off; the smoothing mechanism is used to suppress single probability fluctuations, and the hysteresis mechanism is used to avoid frequent state switching of the vehicle near the threshold; 。 7. The lightweight traffic detection method based on audio-visual fusion according to claim 6, characterized in that, Step 22) specifically involves: (1) Introducing at the decision-making level As a globally consistent prior information, it contributes to the confidence level of visual detection. Perform correction; Use the smoothed audio probability obtained in step 21) Visual confidence of the i-th vehicle candidate box Without changing the candidate box and its categories Confidence levels are only adjusted at the decision-making level; (2) Map the video detection confidence and audio scene probability from the probability space to the Logit space, and construct a weighted linear fusion model for the th... The visual confidence level of each vehicle candidate detection bounding box. The Logit representation is defined as: Audio scene-level vehicle presence probability The Logit is represented as: Construct the following weighted linear fusion model: in The fusion weights for audio information are used to adjust the strength of the influence of audio scene priors on the visual detection results. When =0, the fusion model degenerates into pure visual detection; as As the value increases, the role of audio detection results in fusion decision-making gradually strengthens; in, Visual confidence Logit value, To smooth audio probability Logit value, The weighted fusion Logit value is the result of the two components, where λ≥0 represents the audio fusion weight, and log represents the natural logarithm. In actual calculations, the input probability is restricted to [ε, 1]. Within ε], ε is a pre-defined small positive number; (3) Merge Logit values By mapping back to the probability space using the Sigmoid function, we obtain the output of step 22) – the fusion confidence score. And pass it to step 23) for detection box filtering; 。 8. The lightweight traffic detection method based on audio-visual fusion according to claim 7, characterized in that, Step 23) specifically involves: Set fusion confidence threshold Construct the final vehicle detection set: in, ∈(0,1) is the fusion confidence threshold; Represents the final set of vehicle detections at time t; , These represent the position and scale, category, and fusion confidence of the retained candidate boxes, respectively. Only keep ≥ The vehicle candidate boxes are used as input for multi-target tracking in step 31); Filter the test results at the target level: when At lower levels, audio Logit suppresses low-confidence visual candidate boxes, reducing false detections in car-free scenarios; when When the level is high, audio priors improve the scores of vehicle candidate boxes that are realistic but have low visual confidence, reducing the risk of missed detections in low-light, occluded, and long-distance scenes.

9. The lightweight traffic detection method based on audio-visual fusion according to claim 8, characterized in that, Step 3 specifically involves: Step 31): Multi-target tracking input, used to receive the final vehicle detection set output from step 23). Complete trajectory prediction, detection-trajectory association and state update to form the current trajectory set T(t); Step 32): Trajectory confidence modeling is used to form visual trajectory confidence based on the association detection results, and it is fused with the smoothed audio probability from step 21) in the Logit space to obtain trajectory fusion confidence; Step 33): The trajectory lifecycle is dynamically adjusted according to the audio scene prior, which is used to change the maximum number of consecutive lost frames of the trajectory based on the smooth audio probability, so that the trajectory preservation strategy matches the current vehicle activity state.

10. The lightweight traffic detection method based on audio-visual fusion according to claim 9, characterized in that, Step 31) specifically involves: Using the TBD method, the target position is first predicted using state estimation methods such as Kalman filtering. Then, the association cost is constructed by combining IOU. Finally, the Hungarian algorithm or its variants are used to match the trajectory with the detection and update the trajectory state. The vehicle target is continuously associated over time to form a trajectory set. ; The TBD stands for Tracking-by-Detection, which is multi-target tracking based on detection; IOU stands for Intersectionover Union. in, Indicates the first The state of a trajectory includes the target's spatial location, motion state, and trajectory confidence information; in, Let be the number of valid trajectories at time t. Let T(t) be the state of the k-th trajectory, including target position, velocity, trajectory age, number of consecutive lost frames, and trajectory confidence; T(t) is the set of all valid trajectories. Step 32) specifically involves: Let the first The trajectory at time The confidence level of the visual trajectory is Mapped to the Logit space, and combined with the mapping of audio scene probability in the Logit space under a unified Logit decision scale, a trajectory-level fusion model is constructed. Finally, the confidence of the fused trajectory is mapped back to the probability space through Sigmoid. in, ∈[0,1] represents the confidence level of the visual trajectory obtained by the k-th trajectory based on the current and historical association detection; Set its Logit value; The trajectory Logit after fusion with audio Logit; λ represents the confidence level of trajectory fusion after Sigmoid mapping; λ has the same meaning as the audio fusion weight in step 22) or can be set separately; the trajectory fusion confidence level is used for subsequent trajectory confirmation, maintenance and termination decisions; The Logit representation of the confidence score for visual trajectories is as follows: Trajectory-level fusion model: Mapping back to probability space: ； Step 33) specifically involves: Maximum number of consecutive frames allowed in the trajectory , Based on the number of frames to be lost tolerable. This is the audio adjustment coefficient; when When the value is high, it indicates that there is continuous vehicle activity in the scene. The system will appropriately extend the trajectory retention time to improve the trajectory continuity under occlusion conditions. when When the confidence level is low, the system will terminate the trajectory with low confidence level more quickly, thereby reducing the accumulation of false trajectories and false alarms; Where L(t) is the maximum number of consecutive frames allowed at time t, L_0 is the basic frame loss tolerance without audio adjustment, and β≥0 is the audio adjustment coefficient. The smoothed vehicle presence probability obtained in step 21) is used; in actual execution, L(t) can be rounded and limited to the preset upper and lower limits; when the vehicle activity probability is high, the trajectory retention time is extended, and when the vehicle activity probability is low, the trajectory retention time is shortened.

Images