Vehicle target detection system and method based on multiple unmanned aerial vehicle video streams

Abstract

The invention discloses a vehicle target detection system and method based on multi-channel unmanned aerial vehicle video streams, and the system comprises a multi-channel video frame obtaining module which obtains the addresses of the multi-channel unmanned aerial vehicle video streams from a streaming media server, and obtains the respective video frame from each video stream; the improved algorithm network module is used for reading and detecting each path of video by starting different threads, and a partial convolutional layer, a bidirectional feature pyramid network and a focusing modulation network module are used for an algorithm network, so that an algorithm model supports lightweight processing; the data set training model module is used for carrying out model training by utilizing the public data set and carrying out enhancement operation on the data; and the target detection and recognition module is used for performing real-time recognition and detection on each frame of image of the multiple paths of video streams by utilizing a trained target detection algorithm, and drawing the detected vehicle target information on the current frame of image for marking. According to the invention, the detection precision and stability of the to-be-detected target under the high-altitude view angle can be effectively improved.

Description

Technical Field

[0001] The present invention relates to computer vision and machine learning technologies, and in particular to a vehicle target detection system and method based on multi-channel unmanned aerial vehicle video streams. Background Art

[0002] With the deepening of urbanization, the problems faced by traffic management are becoming increasingly complex. Especially in the context of a surge in vehicles, urban traffic congestion, violations and frequent accidents have become common problems. Traditional traffic management methods usually rely on ground cameras and manual monitoring, which have shortcomings such as limited camera monitoring coverage, still relying on manual participation in real-time monitoring, and low efficiency in handling emergencies. In addition, traditional ground camera systems lack intelligent analysis functions, making it difficult to process and predict complex traffic data, and cannot predict potential traffic jams or traffic accidents.

[0003] With the advancement of electronic technology and aerospace technology, a new trend of using drones as auxiliary tools for traffic management has emerged. Due to the advantages of drones such as flexibility and wide field of view, they have been widely used in agriculture, surveying and mapping, power inspection and other fields. With the continuous upgrading of drone software and hardware technology and the enhancement of information processing capabilities, it has become possible to use drone aerial video / images to quickly or in real time detect and identify stationary, slow or medium-high speed moving targets such as pedestrians, non-motor vehicles and motor vehicles on the road, and to conduct effective supervision.

[0004] For example, the invention application with application number CN202210388211.1 and name “A DeepSort YOLOv5-based Pedestrian Multi-target Tracking Method from UAV Perspective” discloses a target detector using the YOLOv5 network algorithm and the DeepSort multi-target tracking algorithm, which comprehensively utilizes Kalman filtering, Hungarian algorithm, etc. to detect and track normal-scale targets with obvious features. However, for complex scenes in drone aerial images, such as targets that are too small, densely distributed, and severely occluded from a high-altitude perspective, the detection accuracy of the YOLOv5 network algorithm is insufficient, resulting in a low recognition accuracy of the detection results, and therefore it cannot meet the current traffic management needs for road target detection and recognition. Similarly, the invention application with application number CN202311732901.5 and titled “A method for tracking multiple pedestrians from a drone perspective” discloses preprocessing of a drone-perspective image dataset containing pedestrians, and training the designed pedestrian detection and tracking network to obtain a pedestrian detection and tracking model by designing and optimizing the loss function of a pedestrian detection and tracking network from a drone perspective. The technology for real-time detection of new drone-perspective image sequences to track multiple pedestrians can, to a certain extent, solve the problem of locking pedestrian targets through cascade matching and Kalman filtering when the background of pedestrian targets is complex or obscured, thereby achieving accurate detection and continuous tracking of pedestrians, and outputting the location information and motion trajectory of pedestrians. However, in current actual complex application scenarios, the application of this method to detect multi-scale targets has exceeded the processing capability of the pedestrian multi-target tracking method.

[0005] As for the existing technologies for traffic control of motor vehicles, in addition to the above-mentioned technology of using ground cameras assisted by manual monitoring, there are also technical solutions based on video monitoring based on drones equipped with recording equipment. However, most drones currently use a single video stream, and there is also a lack of technical applications equipped with intelligent analysis algorithms for vehicle target detection. Therefore, it is still difficult to achieve the requirements of multi-scale and small target recognition accuracy when dealing with complex traffic environments.

[0006] For example, the invention application with application number CN201910601359.7 and titled "A vehicle detection method for drone images based on lightweight neural networks" discloses a technology that uses a neural network drone to detect vehicle targets to achieve high recognition accuracy and speed. By designing an anchor frame of adaptive size and using the network structure of the lightweight neural network ShuffleNet v2 to extract the target vehicle image features in the drone-collected image, the accuracy and real-time requirements of the low-power processor in the drone for the detection task are achieved. The application of this method can achieve better detection results for simple vehicle targets. For another example, the invention application with application number CN202310940455.0 and titled "Road vehicle trajectory extraction method based on drone monocular camera" discloses a technology that uses a drone monocular camera to shoot driving video data of vehicles on mountain roads, and uses a computer vision processing method to extract vehicle trajectories to obtain high-definition vehicle trajectories. For another example, the invention application with application number CN202110665507.9 and titled “Vehicle Target Detection Method for UAV Perspective Images Based on Deep Learning” obtains the feature map of the feature extraction module by constructing a feature extraction module, constructs a lightweight feature fusion module based on the feature map of the feature extraction module, obtains the multi-scale target feature map, and then constructs a multi-scale feature detection module. Then, the above-mentioned feature extraction module, lightweight feature fusion module and multi-scale feature detection module are used to construct a UAV perspective vehicle target rapid detection network, and the network is trained on the data set, thereby realizing the detection of vehicle targets. It not only improves the adaptability to the characteristics of vehicle targets in UAV perspective images, but also effectively improves the detection efficiency of vehicle targets by deep learning networks, and to a certain extent solves the contradiction between the large scale change of vehicle targets and the limited computational complexity of airborne embedded platforms in the detection of vehicle targets in airborne embedded UAV perspective images.

[0007] However, the above-mentioned existing technology for vehicle target detection based on drone monitoring video still has the following defects and deficiencies: First, the current drone monitoring method still mostly uses single-channel video. Since single-channel monitoring can only be shot from a single angle, facing complex traffic scenes, such as vehicle occlusion, targets of different scales (such as small vehicles, long-distance vehicles), its detection ability is very limited, so it is difficult to handle multi-angle and multi-scale scene changes, especially when the target vehicle to be detected is blocked by objects such as buildings and trees, its detection effect and recognition accuracy will be greatly reduced, so it cannot meet the needs of high-precision and multi-angle traffic supervision. Second, single-channel video monitoring still has the problem of insufficient system stability in complex traffic environments. For example, single-channel video is easily disturbed and affected by external environmental changes (such as strong winds, rainfall, strong light, etc.), resulting in unstable or interrupted video signals, which in turn affects the continuity and accuracy of vehicle recognition. Third, drone monitoring methods usually simply capture and transmit videos and images, and still lack support for intelligent analysis algorithms and applications of multi-channel targets to be detected, such as real-time intelligent analysis functions such as vehicle target detection, vehicle tracking, and license plate recognition in multi-channel drone video streams. These intelligent analyses and applications are crucial for today's complex road traffic supervision and illegal behavior monitoring. Summary of the invention

[0008] In view of this, the main purpose of the present invention is to provide a vehicle target detection system and method based on multi-channel UAV video streams to overcome the shortcomings of the above-mentioned prior art, while ensuring the lightweight of the algorithm model, effectively improving the detection accuracy and stability in complex scenarios such as the vehicle targets to be detected from a high-altitude perspective are too small, have large scale differences, are densely distributed, and are severely occluded.

[0009] Another object of the present invention is to further improve the reliability, monitoring coverage and intelligent application analysis level of the monitoring system by combining multi-channel video monitoring and intelligent analysis algorithms.

[0010] Another object of the present invention is to integrate the vehicle target detection system and the vehicle target tracking algorithm so that the target detection system can be easily deployed in a small server or edge computing device to provide technical support for road traffic supervision and monitoring of illegal activities.

[0011] To achieve the above object, the technical solution of the present invention is as follows:

[0012] The vehicle target detection system based on multi-channel UAV video streams includes:

[0013] A multi-channel video frame acquisition module is used to obtain the addresses of multiple drone video streams from a streaming media server and obtain respective video frames from each of the video streams;

[0014] The improved algorithm network module is used to realize the reading, detection and processing of each video channel and the streaming of the processed video by opening different threads, and the algorithm network uses some convolutional layers, bidirectional feature pyramid networks and focus modulation network modules to enable the algorithm model to support lightweight processing;

[0015] The data set training model module is used to perform model training using a public data set, and divide the data set into a training set, a validation set, and a test set; on this basis, the data in the data set is flipped, rotated, scaled, and enhanced by adding noise, and the recognition accuracy of the data set training model is further improved by expanding the training data set;

[0016] The target detection and recognition module uses a pre-trained target detection algorithm to perform real-time recognition and detection on each frame of multiple video streams, draws the detected vehicle target information on the current frame image for marking, and then re-synthesizes the recognized and processed image frames into a video.

[0017] In addition, it also includes a multi-channel UAV video image acquisition module, which is used to collect video signals through an image acquisition card mounted on the UAV, capture image data of vehicle targets, and then automatically connect the video signals to the wireless communication network in real time, and push the obtained image data to the streaming media server.

[0018] The multi-channel video frame acquisition module further includes a video streaming media parsing submodule and a video frame extraction submodule.

[0019] Preferably, it also includes a multi-channel video stream push module for pushing the video stream re-synthesized from each channel of image frames identified and processed by the target detection and recognition module to a designated streaming media server in real time.

[0020] A vehicle target detection method based on multi-channel UAV video streams includes the following steps:

[0021] The steps of reading multiple channels of image data from a streaming media server and obtaining corresponding video frames of each channel;

[0022] The step of establishing an improved algorithm network and training the improved algorithm network model using a data set training model;

[0023] The step of sending the video stream of the vehicle target to be detected to the trained and improved algorithm network model for detection and identification, and outputting the vehicle target information obtained by the detection to the current frame image.

[0024] Wherein: before the step of reading multiple channels of image data from the streaming media server to obtain corresponding video frames of each channel, the step further includes:

[0025] The step of collecting road vehicle image data by using multiple drones, and pushing the collected vehicle image data / video stream to a streaming media server.

[0026] The step of reading multiple channels of image data from the streaming media server to obtain corresponding video frames of each channel includes:

[0027] Perform video streaming analysis on multiple image data channels and obtain multiple drone video streaming addresses from the streaming server;

[0028] Steps for extracting individual video frames from multiple drone video streams.

[0029] The step of establishing the improved algorithm network comprises:

[0030] Steps to carry out partial bottleneck module design;

[0031] Steps for designing dual-channel characteristic structures;

[0032] The step of replacing the spatial pyramid pooling structure module design with a focused modulation network module.

[0033] Wherein: the process of training the improved algorithm network model using the data set training model includes:

[0034] The step of converting the format of the labeled data in the existing dataset to make it consistent with the input format of the YOLOv8 model;

[0035] The steps of dividing the data set into a training set, a validation set and a test set, and classifying or merging the categories contained in the data set;

[0036] The data in the data set is enhanced and the data set is expanded for training.

[0037] Wherein: the step of sending the target vehicle video stream to be detected into the trained and improved algorithm network model for detection and identification specifically includes:

[0038] Another trained and improved algorithm network model processing thread is started, and the algorithm network model extracts a frame image from the cache queue according to the generated Stream_ID;

[0039] Use the trained target detection algorithm network to identify and detect each frame of the multi-channel drone video stream;

[0040] When a vehicle target to be detected is detected, the vehicle detection system draws the output vehicle target information on the current frame image.

[0041] The vehicle target detection system and method based on multi-channel UAV video streams of the present invention have the following beneficial effects:

[0042] 1) By reading multiple video streams recorded by drones, the video frames of each video stream are obtained, and the video frames are sent to the pre-trained improved YOLOv8 algorithm model for detection and recognition, so as to realize vehicle target detection and processing of multiple video streams, and finally send them to the streaming media service platform for real-time display.

[0043] 2) The improved YOLOv8 algorithm is applied to open different threads to realize the reading, detection and processing of each video in the system, as well as the streaming of processed videos. In addition, the YOLOv8s network algorithm uses some convolutional layers, bidirectional feature pyramid networks, and focal modulation networks (FMN) modules to replace the fast spatial pyramid pooling structure (SPPF, Spatial Pyramid Pooling Fast) module. While ensuring the lightweight of the algorithm model, it greatly improves the accuracy of target detection of small targets, dense vehicles, and targets in complex backgrounds.

[0044] 3) The vehicle target detection system and method of multi-channel UAV video streams of the present invention can further improve the reliability, coverage and intelligent analysis level of monitoring by combining multi-channel video monitoring and intelligent analysis algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a functional block diagram of a vehicle target detection system based on multi-channel UAV video streams according to an embodiment of the present invention;

[0046] Figure 2 The present invention is a flowchart of a vehicle target detection method based on multi-channel UAV video streams according to an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the improved YOLOv8 network structure used in an embodiment of the present invention;

[0048] Figure 3A , Figure 3B , Figure 3C for Figure 3 Schematic diagram of the local network structure content.

[0049] Figure 4A , Figure 4B Schematic diagrams of vehicle target detection effects of four-channel UAV video streams according to embodiments of the present invention;

[0050] Figure 5A , Figure 5B They are schematic diagrams of dense small target detection results of vehicle target detection in one of the drone video streams according to an embodiment of the present invention;

[0051] Figure 6A to Figure 6D They are all enlarged views of the effect of dense small target detection results of vehicle target detection in any drone video stream of an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

[0053] Figure 1 The present invention is a functional block diagram of a vehicle target detection system based on multi-channel UAV video streams according to an embodiment of the present invention.

[0054] like Figure 1 As shown, the vehicle target detection system mainly includes: a multi-channel UAV video image acquisition module, a multi-channel video frame acquisition module, an improved algorithm network module, a data set training model module, a target detection and recognition module and a multi-channel video stream processing push module. Among them:

[0055] The multi-channel UAV video image acquisition module collects video signals through the image acquisition card mounted on the UAV, captures the image data of the vehicle target, and then automatically connects the video signal to the wireless communication network in real time, and pushes the obtained image data to the streaming media server. On the streaming media server, the image / video data containing the vehicle target can be received, transmitted and displayed in real time.

[0056] The multi-channel video frame acquisition module is used to obtain the addresses of the multi-channel drone video streams from the streaming media server and obtain respective video frames from each of the video streams. The multi-channel video frame acquisition module further includes a video streaming media parsing submodule and a video frame extraction submodule.

[0057] Preferably, a video stream caching submodule may also be included.

[0058] The video streaming parsing submodule is used to obtain the addresses of multiple drone video streams from the streaming server. The video frame extraction submodule is used to convert the input video stream address through a hash algorithm to generate a unique ID for each video; and to open a thread for each Stream_ID to obtain video frames from each video stream through tools such as Opencv.

[0059] Preferably, each video frame is stored in a queue Q, that is, the video stream is stored in a video stream cache submodule. Specifically, the capacity of the queue Q can be preset to 1024, and each queue Q is used for secure data transmission between different threads, so that the secure data can be subsequently transmitted to the thread for model processing.

[0060] The improved algorithm network module solves the problems of low-altitude drone data sets with small number of pixels, dense distribution, and uneven categories of targets to be tested, as well as the problems of missed detection, false detection, and low precision of the original YOLOv8 network algorithm. The present invention designs and improves the original YOLOv8 network algorithm to solve the above technical problems. In addition, by performing video processing on multiple drone video streams, the algorithm network is lightweighted in the embodiment of the present invention to achieve good results in order to solve the problem of high consumption of required computing resources. Please refer to Figure 1 to Figure 3 , the specific improvement measures are as follows:

[0061] 1) For the bottleneck part of the existing YOLOv8 network, the partial convolution layer (PConv) is used to replace the ordinary convolution layer in the bottleneck structure of the cross stage partial feature fusion module (C2f) to avoid redundant gradient calculations and balance the feature extraction capabilities. For the C2f module, please refer to Figure 3 .

[0062] 2) A bidirectional feature pyramid network (BiFPN) is used to replace the path aggregation feature pyramid network (PAFPN) structure of the original YOLOv8 to increase the feature fusion capability of the network. Similarly, a partial convolutional layer (PConv) is used to improve the feature extraction module (not shown).

[0063] 3) Use the Focal Modulation Networks (FMN) module to replace the Spatial Pyramid Pooling Fast (SPPF) structure to enhance the model's target recognition ability and robustness while maintaining or improving accuracy.

[0064] In an embodiment of the present invention, the dataset training model module uses the VisDrone 2019 dataset training model. The VisDrone 2019 dataset is an existing public dataset focusing on drone / unmanned aerial vehicle (UAV) image analysis, mainly used for computer vision tasks such as target detection, tracking and density estimation. VisDrone 2019 provides labeled data, including target categories, bounding boxes and other information. In order to make it consistent with the input format of the YOLOv8 model, the data is uniformly converted into the YOLO format in the present invention.

[0065] The YOLOv8 model is trained using the public dataset VisDrone 2019 dataset. The dataset is further divided into training set, validation set and test set. The dataset contains 10 categories, of which categories 0 and 1 are pedestrian types, and the rest are vehicle types. Considering that the application scenario does not require detailed distinction between categories, categories 2 to 9 are merged into one large category and unified as vehicle types. At the same time, on this basis, the data is enhanced by flipping, rotating, scaling, adding noise, etc. By expanding the training dataset, the recognition accuracy of the training model can be improved.

[0066] The data set training model module communicates with the target detection and recognition module and is data-connected. In this embodiment, when the target detection and recognition module is started, another thread is automatically started, and video frame images are extracted from each video cache queue through the trained training model in the data set training model module, and each frame image is recognized and detected using the pre-trained target detection algorithm. In this detection process, multiple targets in the image can be recognized at the same time, and key information such as the category, location coordinates, and confidence level of each target can be returned.

[0067] When a target vehicle is detected, the target detection system will draw the above information on the current frame image, usually marking the location of the target vehicle with a rectangular box, and displaying the category name and confidence level next to the rectangular box. At the same time, the image frames that have been identified and processed will be re-synthesized into a video stream and transmitted to the designated streaming server in real time to support remote viewing and real-time monitoring.

[0068] The above processing flow not only ensures the real-time nature of the video stream, but also incorporates the results of intelligent analysis during the transmission process, thereby enabling the entire system to have efficient target detection and monitoring capabilities.

[0069] The multi-channel video stream push module is used to mark the tracking results in each frame of video, start the FFmpeg process, use the FFmpeg tool to encode each frame of video, and bind its input and output streams to the pipeline. The encoded video contains the object's tracking mark and other processing information, and then pushes it to the streaming media server platform according to the streaming media address to realize the release and playback of real-time video streams. Figure 4A , Figure 4B , which is a schematic diagram of the vehicle target detection effect of four-channel UAV video streams in an embodiment of the present invention.

[0070] Figure 2 The present invention is a flowchart of a vehicle target detection method based on multi-channel UAV video streams according to an embodiment of the present invention.

[0071] like Figure 2As shown, the vehicle target detection method based on multi-channel UAV video streams mainly includes the following steps:

[0072] Step 20: Collect road vehicle image data through multiple drones, and push the collected vehicle image data / video stream to a streaming media server.

[0073] The specific steps include:

[0074] Step 201: Control a drone to take off, and use the camera and image acquisition card carried by the drone to collect image data of road vehicles. The image data may include images, videos, and audio.

[0075] Step 202: The image data collected by multiple drones are connected to the communication network by wireless communication through the communication antennas of the drones, and the respective image data are pushed to the streaming media server for storage, and can also be transmitted and displayed in real time.

[0076] Here, the streaming media server usually uses standard protocols such as Real-Time Streaming Protocol (RSTP) and Real-Time Messaging Protocol (RTMP) to transmit image / video data.

[0077] Step 21: Read multiple channels of image data from the streaming media server to obtain multiple channels of video frames. Specifically, the steps include:

[0078] Step 211: Perform video streaming analysis on the multi-channel image data, and obtain the multi-channel drone video stream addresses from the streaming server.

[0079] Step 212: Extracting individual video frames from multiple drone video streams. Use a hash algorithm to convert the input video stream address and generate a unique Stream_ID for each video stream. Open a thread for each ID and use a video capture tool to obtain video frames from each video stream.

[0080] Preferably, the method further includes step 213: caching each of the acquired video frames, which is as follows: storing each extracted video frame in a queue Q, assuming that the capacity of the queue Q is 1024, and the queue Q is used for secure data transmission between different threads, so that the secure data can be subsequently transmitted to the thread for model processing.

[0081] Step 22: Steps for establishing an improved YOLO algorithm network module.

[0082] The embodiment of the present invention mainly reduces redundant calculations by reconstructing the network structure of YOLOv8, reduces the amount of network calculation and parameter quantity, thereby ensuring real-time and fast processing of multiple video streams; by improving the feature fusion capability of the algorithm network, the detection accuracy and recognition accuracy are improved, thereby improving the detection capability of small targets and targets to be detected in complex backgrounds. The present invention mainly improves the existing YOLOv8 algorithm network as follows: 1) The Bottleneck of the backbone network and the feature extraction network is lightweight processed, and only convolution calculations are performed on part of the feature layer, while retaining the original network hierarchy, thereby reducing the amount of network calculation and improving the detection speed. 2) A dual-channel feature extraction network that integrates a lightweight attention mechanism is introduced to increase the feature fusion capability of the network, and lightweight processing is performed on the feature extraction module, thereby improving the detection accuracy without affecting the network reasoning speed. 3) In the drone target detection scenario, due to the rapid change of the drone's flight altitude, the size of the target to be detected is quite different, and the same object in the images collected at different times often presents different sizes, and the multi-layer pooling operation of the original SPPF module makes it easy to ignore the feature information of small targets. Therefore, the improved network of the present invention introduces a focal modulation network (FMN) module to replace SPPF, thereby improving the multi-scale feature extraction capability of the network. Currently, YOLOv11 has been released, which is superior to YOLOv8 in detection accuracy and processing speed, and v11 is similar to v8 in engineering implementation. The YOLO algorithm network module of the present invention is backward compatible, and with slight improvements, it also supports the use of the YOLOv11 algorithm to implement it. The process of establishing the improved YOLO algorithm network module specifically includes the following steps:

[0083] Step 221: Perform partial bottleneck module design.

[0084] The bottleneck part uses a partial convolution layer (PConv, Partial Convolution) to replace the ordinary convolution layer in the CSPDarknet bottleneck structure to avoid redundant gradient calculations and balance feature extraction capabilities. The specific process is: 1) PConv is a fast convolution structure. Given that the previous method of reducing floating-point calculations cannot effectively reduce operator delays, the operator only performs convolution operations in some input channels CP, and the remaining channels remain unchanged. 2) The floating-point calculation amount of PConv is lower than that of ordinary convolution, but higher than that of DWConv and GConv. At the same time, by reducing redundant calculations and memory access, PConv can extract spatial features more effectively. 3) PConv is used to lightweight improve the cross-stage partial feature fusion (C2f, Cross Stage Partial Feature Fusion Module) module. The extended feature extraction function of C2f is retained in the CSPLayer_PConv module, and the partial convolution bottleneck (Partial Bottleneck) is used to replace the original network structure. 4) The bottleneck structure consists of two Pconv Model modules, the first module is used to compress the features, and the second module is used to map the features to high dimensions and fuse them with the input. 5) The PconvModel module contains some convolutional layers, only performs convolution calculations on some features, concatenates the uncalculated parts with the calculated parts, uses batch normalization to speed up network convergence, and uses SiLU (sigmoid linear unit) activation function to improve the optimization effect of deep models.

[0085] Step 222: Perform dual-channel characteristic structure design.

[0086] In an embodiment of the present invention, an improved BiFPN is used to replace PAFPN to increase the generalization ability of shallow network spatial features. The specific process is: 1) BiFPN increases the fusion of low semantic features and high semantic features by repeated bidirectional cross-scale connections. 2) The weighted feature fusion mechanism is used to increase the focus of fusion to distinguish the importance of different features. 3) The BiFPN feature extraction module is replaced by a partial bottleneck. 4) Due to the characteristics of small and clustered targets of drones to be detected, a single weighted feature fusion cannot significantly improve spatial information learning. Therefore, a channel attention squeeze and excitation (SE) module is added to BiFPN during the feature fusion process, aiming to allow the network to learn more small target spatial information during the fusion process through attention training.

[0087] Step 223: Replace the SPPF module design with a focal modulation network (FMN) module.

[0088] The traditional SPPF module enhances the model's ability to detect multi-scale targets by pooling feature maps at different scales. Although SPPF performs well in multi-scale fusion, it may encounter performance bottlenecks when dealing with complex scenes and small targets. In an embodiment of the present invention, the FMN module is applied to dynamically adjust the weights of the feature map by introducing a modulation mechanism, so as to better focus on important areas and reduce attention to unimportant image areas. Therefore, the use of this technical feature can accurately capture the details of the target, thereby improving detection accuracy.

[0089] The focus modulation network module is mainly implemented through the following process:

[0090] Step 2231: Combine the focus modulation process with contextual information and use deep convolutional layer stacking to encode visual contextual information of different ranges.

[0091] Step 2232: Perform a gated aggregation process to selectively aggregate context information into the modulator of each query token through a gating mechanism.

[0092] Step 2233: The process of element-by-element affine transformation, injecting the aggregated modulator into each query token through affine transformation.

[0093] Step 23: Use the improved YOLO algorithm network to continuously improve the recognition rate and detection accuracy of multi-size, small-target vehicles to be detected in complex scenes. Please refer to Figure 3 .

[0094] In this embodiment, this step focuses on the recognition and detection problems in scenarios where the vehicle target to be tested is blocked or overlapped.

[0095] The main method to solve the dense occlusion problem in target detection is the repulsion loss (RL) method. Repulsion Loss is a loss function used to solve the occlusion problem between dense targets in target detection. Its design idea is to introduce additional constraints so that different detection boxes can repel each other, thereby avoiding overlap and tight aggregation.

[0096] The RL module mainly includes three functional modules. The role of Repulsion loss in the embodiment of the present invention is to require the predicted box P to be close to (attract) its real target T, while also requiring P to be away from (repel) other real boxes (such as B) next to T. The repulsion loss function L is expressed by the following formula:

[0097] L=L Attr +α*L RepGT +β*L RepBox

[0098] Where: the parameter L Attr The subscript Attr stands for Attraction Loss, which represents the attraction loss and is used to attract the predicted boxes (Pred Box) to their corresponding real boxes (GT Box). The parameter L RepGT RepGT in the above is Repulsion to Ground Truth Loss, which represents the rejection loss of the prediction box to other real boxes, preventing the prediction boxes of different targets from being aggregated together. RepBox RepBox stands for Repulsion to other Predicted Boxes Loss, which represents the repulsion loss between prediction boxes to prevent multiple prediction boxes of the same target from being too aggregated. The α and β are both hyperparameters used to balance the weights of different effects. Preferably, when the hyperparameters α and β are both 0.5, the detection performance is optimal.

[0099] The physical meaning of the above formula is explained as follows:

[0100] The L Attr The module is used to make the predicted box and its target box as close as possible. Attr The module uses the regression loss in general target detection. The method of the present invention supports the use of Euclidean distance, Smooth L1 Distance or I O U distance. However, in order to be comparable with other algorithms, Smooth is used in this embodiment. L1 distance.

[0101]

[0102] Where: P∈Pos + , that is, all positive samples. The positive samples are the set of detection boxes P divided according to the set IoU threshold.

[0103] For each detection box P, match a ground truth box with the maximum IoU value. P It is the predicted box obtained by regression shift from the detection box P.

[0104] Smooth L1 The distance expression is as follows:

[0105]

[0106] Among them: Parameter Represents the prediction box coordinates; v = (v x , v y , v w , v h ) represents the real target box; Represents each value in the prediction box; v i Represents each value in the real target box; x, y, w, h represent the upper left corner coordinates and width and height respectively.

[0107]

[0108] From the above formula, we can conclude that B p and The upper left corner coordinates and width and height (x, y, w, h) are smoothed separately L1 Calculate and then accumulate the sum. The goal of optimization is to shorten the distance between the two so that the predicted box and its target box are as close as possible.

[0109] Here, the L RepGT , in the embodiment of the present invention, plays the role of making the prediction box and the surrounding target box G as far away as possible. Its expression is as follows:

[0110]

[0111] in: σ is a hyperparameter ∈ [0, 1), which is a smoothing parameter that controls the sensitivity of the loss function to outliers; IOG is an indicator to measure the degree of overlap between the predicted box and the surrounding objects.

[0112] The surrounding target box G here is the target box with the largest IoU except for the matched target box, which can be used express:

[0113]

[0114] Right now It is the Ground_truth-B with the largest IoU distance with the prediction box P among the remaining Ground_truths except the Ground_truth-A that matches the prediction box P. Among them,

[0115]

[0116] σ is a hyperparameter ∈ [0, 1), which is a smoothing parameter that controls the sensitivity of the loss function to outliers; B is the prediction box; G is the target box of the surrounding object; area() is the area function.

[0117] Here, the parameter L RepBox , which makes the prediction box P i and the surrounding prediction box P j As far away as possible, P i and P j Different prediction boxes are assigned respectively, and the distance between them is I O U distance, defined as follows:

[0118]

[0119] here, is the i-th prediction box; is the jth prediction box; I[] represents the identity function, and ε is a non-zero constant to avoid making the denominator zero. It can be found that when the prediction box P i and other prediction boxes P around it j The larger the IoU, the larger the loss generated. Therefore, it can effectively prevent two prediction boxes from being filtered out by NMS due to being too close, thereby reducing missed detections. From the optimization goal, it can be seen that the prediction boxes corresponding to different targets will eventually be separated, avoiding missed recalls caused by merging prediction boxes in NMS post-processing.

[0120] refer to Figure 3 , Figure 3A to Figure 3C In this embodiment, the specific solution process for the problem that the vehicle to be detected is a small-sized target is as follows:

[0121] Here, the YOLOv8 described is based on Anchor-free, which means that its detection mechanism does not rely on predefined Anchor boxes, which makes clustering adjustment or manual optimization of Anchor parameters in traditional Anchor-based methods ineffective for improving the detection performance of small targets. This is reflected in the solution of the present invention through the following points:

[0122] (1) In the VisDrone dataset, the target size distribution has a significant characteristic: extremely small targets with a pixel area less than 16×16 account for 12.05% of the total targets in the dataset, and small targets with a pixel area less than 32×32 account for as much as 44.7%. This data distribution feature shows that the proportion of small targets in the VisDrone dataset is very high, and this data characteristic provides a good foundation for the detection effect of small targets.

[0123] (2) The above datasets were enhanced by flipping, rotating, scaling, adding noise, etc. By expanding the training dataset, the recognition accuracy of the trained model was improved, especially the small target recognition capability was further enhanced.

[0124] (3) Multi-scale feature fusion

[0125] Most of the objects in the images taken by drones are small targets, and their feature information mainly exists in the shallow network. As the number of network layers increases, the feature information of small targets will be lost, resulting in a decrease in the detection accuracy of small targets. In the network, in order to improve the detection ability of small targets, a new model output layer P2 is added. The P2 layer has fewer convolutions and a larger feature map size, so that it can retain more small target features. In order to further improve the detection performance of the network, the present invention combines the idea of ​​Bi-FPN, optimizes the feature fusion module of the original network, and increases the connection channels between different scales, so that the feature information can flow across levels, enhancing the network's detection ability for targets of different scales.

[0126] Figure 3A The figure shows the feature fusion method in YOLOv8. Although the small target detection layer P2 is added, it is not directly connected to the output layer, but is input into the P3 layer and then output. In the feature extraction process, the features of small targets are easily interfered by larger targets. The deepening of the network layer also means the loss of feature information of small targets, which makes the original network unable to meet the actual requirements for small target detection.

[0127] Figure 3B The figure shows a Bi-FPN network, which has jump connections between the input nodes and output nodes of the same layer. Therefore, under the same scale, the jump connection can better extract and transmit feature information. Therefore, this design adds jump connections in the shallow network to retain more shallow features. At the same time, it optimizes the network structure, reduces the number of network parameters and calculations, and improves the feature fusion structure in view of the limited computing resources of the drone platform.

[0128] The original YOLOv8 model has three detection heads of 80×80, 40×40, and 20×20 that retain feature layer information of different scales. The improved detection heads are 160×160, 80×80, and 40×40. The larger scale can better retain the relevant feature information of small targets, expand the model resolution, and enhance the perception ability of small targets. Figure 3C shown.

[0129] like Figure 3As shown in the figure, Conv stands for convolution module, the full name is convolution; C2f_Pconv, C2f module is CSPLayer with 2Conv, Pconv is partial convolution, and the design uses PConv to make lightweight improvements on the C2f module; BiFPN stands for Bi-directional Feature Pyramid Network; Upsmaple stands for upsampling.

[0130] Here, the main workflow of the focal modulation network (FMN) module includes global context extraction and local dynamic adjustment.

[0131] First, the global context extraction process is described as follows: the input feature map undergoes a dimensionality reduction operation (such as pooling or linear projection) to generate a global feature vector; the global feature vector captures the semantic information or global context of the entire feature map, which is used to guide the dynamic adjustment of local weights; this process is similar to the extraction of global information in the attention mechanism, but FocalModulation does not directly calculate the attention between pixels, but operates on global information through lightweight integration.

[0132] Second, the local dynamic adjustment process is described as follows: a. The features of the local area are extracted through the local context perception module to extract fine-grained information (such as convolution operation). b. The global context features interact with the local features through the dynamic adjustment module to generate dynamic weights. c. The dynamic weights are used to adjust the importance of local features, that is, to enhance or suppress the response of a specific area through dot multiplication or weighted operations.

[0133] In the embodiment of the present invention, after introducing the Focal Modulation Networks module to replace the SPPF module, training and evaluation were performed on VisDrone. The experimental results show that the YOLOv8 network algorithm model with the Focal Modulation Networks module has significantly improved accuracy, especially in small target detection and complex scenes.

[0134] Specifically, 1) the recognition accuracy is effectively improved. Compared with the traditional SPPF module, the combination of the YOLOv8 algorithm network and the Focal Modulation Networks module has increased the mAP (mean average precision) by about 3 to 5 percentage points. 2) In terms of computational efficiency, although the Focal Modulation Networks module is introduced, due to its efficient attention mechanism, the computational overhead has not increased significantly, so when using the model, the real-time detection capability can be maintained at a high level. As shown in Table 1:

[0135] Table 1:

[0136] Model mAP(%) <![CDATA[I O In(%)]]> Average inference time (ms) YOLOV8+SPPF 48.5 50.8 34 YOLOV8+FMN module 51.6 53.9 35

[0137] Step 24: The step of training the improved algorithm network model using the dataset training model.

[0138] In the embodiment of the present invention, the VisDrone 2019 dataset is specifically used to train the improved algorithm network model. The process includes the following steps:

[0139] Step 241: Make the labeled data provided by the VisDrone 2019, including information such as target category and bounding box, consistent with the input format of the YOLOv8 model, and convert the data set into the YOLO format.

[0140] Step 242: The data set is divided into a training set, a validation set, and a test set, with the ratio being 8:1:1.

[0141] Step 243: Assume that the data set contains 10 categories, of which categories 0 and 1 are pedestrian types, and the rest are vehicle types. Considering that the application scenario does not require detailed distinction between categories, categories 2 to 9 are merged into one large category, unified as vehicle type.

[0142] Step 244: Based on the above steps, the data in the data set is flipped, rotated, scaled, and enhanced by adding noise, and the recognition accuracy of the trained model is improved by expanding the training data set.

[0143] Step 245: Take 100 rounds of training on the Alibaba Cloud server as an example, set the learning rate to 0.01, the batch size to 16, and set the image size to 1280 during training to improve detection accuracy.

[0144] Step 246: The step of evaluating the trained YOLOv8 model using the test set.

[0145] Here, it should be particularly noted that in the above embodiment of the present invention, the execution order from step 20 to step 24 is only one implementation method. Among them, the steps 23 to 24 can be executed simultaneously with the steps 20 to 22, or can be executed before the steps 20 to 22. Finally, technical preparations are made for executing the step 25 to perform vehicle target detection.

[0146] Compared with other existing target detection and recognition technologies, the above-mentioned technology used in the embodiments of the present invention has obvious advantages:

[0147] (1) The present invention supports real-time detection of multiple video streams. The different perspectives of multiple videos can partially solve the problem of vehicle occlusion. For example, multiple drones can simultaneously shoot the same target from different angles to obtain comprehensive perspectives and details. (2) Drones with different resolutions meet the multi-size detection characteristics, improve the detection accuracy of small targets, support data fusion of multiple drone video streams in post-processing, and use the improved algorithm provided by the present invention for real-time recognition, thereby greatly improving the reliability and accuracy of the detection and recognition results. (3) When the above-mentioned output end data is used to display (draw) the video stream in real time, the shape obtained by the multiple drone video streams can be drawn, making it easier to identify and distinguish the target vehicles to be detected.

[0148] Step 25: Send the video stream of the vehicle target to be detected into the trained and improved algorithm network model for detection and identification. Specifically:

[0149] Step 251: start another processing thread of the algorithm network model after training and improvement, and the network model extracts the frame image from the cache queue according to the generated Stream_ID.

[0150] Step 252: Use the trained target detection algorithm network to identify and detect each frame of the multi-channel drone video stream.

[0151] Step 253: When the vehicle target to be detected is detected, the vehicle detection system draws the output vehicle target information on the current frame image. The vehicle target information is usually displayed in the form of a rectangular frame marking the position of the target, and displaying the category name and confidence level next to the frame.

[0152] Furthermore, it also includes:

[0153] Step 254: According to the actual vehicle target detection scenario, modify the confidence level, category and other parameters to achieve the best vehicle target detection effect.

[0154] Step 26: Pushing the multi-channel drone video streams containing the above detection results to other servers and other receiving ends.

[0155] In this embodiment, the specific implementation process is as follows:

[0156] Step 261: After marking the tracking result in each frame of the multi-channel drone video stream, start the FFmpeg process to push the video stream.

[0157] Step 262: Encode the video using the FFmpeg tool. The encoding setting parameters are as follows: encode the original video stream (BGR format) into H.264 format; set the video resolution to 1280x720 or other resolutions, and the frame rate to 15fps; use the ultra-fast preset for encoding optimization; output the streaming media format to FLV to the specified RTMP server. Finally, set the disable duration and file size limit, and set the debugging level to debug. This process is a prior art, and the actual application does not limit the use of specific video encoding, media transmission, and video decoding technologies.

[0158] Step 263: The encoded video contains the object tracking mark and other processing information, and is then pushed to the streaming server platform to achieve the publishing and playback of real-time video streams. Figure 5A , Figure 5B shown. Figure 6A to Figure 6D This is an enlarged view of the dense small target detection results for vehicle target detection in any UAV video stream.

[0159] The vehicle target detection system of the present invention supports processing multiple streams at the same time, and the user can flexibly add or close any data stream. For example, when streams A, B, and C are pushed at the same time, the user can add video streams D, E, etc., or choose to close any one or more of A, B, and C.

[0160] The vehicle target detection system of the present invention adopts real-time synchronous processing of multi-channel video streams, so that the system can synchronously process video streams from multi-channel drones, thereby realizing monitoring of a larger range of traffic environments, especially in complex traffic environments or large-scale traffic networks. By using the improved YOLOv8s algorithm model, the detection speed is improved by reducing the parameter amount of YOLOv8s, thereby meeting the demand for rapid response in traffic environments; and by enhancing the semantic fusion between different feature layers of the network and optimizing multi-scale visual tasks, the system significantly improves the ability to detect small targets, dense targets, and even targets in complex environmental backgrounds. By using the output of target detection frame information and multi-channel streaming media push, the vehicle target detection system of the present invention can directly superimpose the vehicle target frame information detected in each video stream of the multi-channel drone video stream into the original video stream, and push the processed multi-channel video stream to the streaming media service platform for real-time display. By using intelligent monitoring and analysis technology, the system can accurately identify vehicle targets. Combined with vehicle trajectory prediction and other technologies, it can effectively identify speeding, illegal parking, wrong-way driving and other illegal behaviors, and then count and analyze traffic flow and congestion trends, helping traffic management departments to quickly grasp road conditions and traffic flow information, and providing strong support for high-precision monitoring and intelligent decision-making in complex traffic environments. In addition, this system also provides a convenient real-time monitoring interface for traffic management departments, which is conducive to the realization of intelligent traffic supervision.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. The vehicle target detection system based on multi-channel UAV video streams is characterized by: include: A multi-channel video frame acquisition module is used to obtain the addresses of multiple drone video streams from a streaming media server and obtain respective video frames from each of the video streams; The improved algorithm network module is used to realize the reading, detection and processing of each video channel and the streaming of the processed video by opening different threads, and the algorithm network uses some convolutional layers, bidirectional feature pyramid networks and focus modulation network modules to enable the algorithm model to support lightweight processing; The data set training model module is used to perform model training using a public data set, and divide the data set into a training set, a validation set, and a test set; on this basis, the data in the data set is flipped, rotated, scaled, and enhanced by adding noise, and the recognition accuracy of the data set training model is further improved by expanding the training data set; The target detection and recognition module uses a pre-trained target detection algorithm to perform real-time recognition and detection on each frame of multiple video streams, draws the detected vehicle target information on the current frame image for marking, and then re-synthesizes the recognized and processed image frames into a video.

2. The vehicle target detection system based on multi-channel UAV video streams according to claim 1 is characterized in that: It also includes a multi-channel UAV video image acquisition module, which is used to collect video signals through an image acquisition card carried on the UAV, capture image data of vehicle targets, and then automatically connect the video signals to the wireless communication network in real time, and push the obtained image data to the streaming media server.

3. The vehicle target detection system based on multi-channel UAV video streams according to claim 1 is characterized in that: The multi-channel video frame acquisition module further includes a video streaming media parsing submodule and a video frame extraction submodule.

4. The vehicle target detection system based on multi-channel UAV video streams according to claim 1 is characterized in that: It also includes a multi-channel video stream push module, which is used to push the video stream re-synthesized by each image frame identified and processed by the target detection and recognition module to a designated streaming media server in real time.

5. A vehicle target detection method based on multi-channel UAV video streams, characterized in that: The steps include: The steps of reading multiple channels of image data from a streaming media server and obtaining corresponding video frames of each channel; The step of establishing an improved algorithm network and training the improved algorithm network model using a data set training model; The step of sending the video stream of the vehicle target to be detected to the trained and improved algorithm network model for detection and identification, and outputting the vehicle target information obtained by the detection to the current frame image.

6. The vehicle target detection method based on multi-channel UAV video stream according to claim 5 is characterized in that: Before the step of reading multiple channels of image data from the streaming media server to obtain corresponding video frames of each channel, the method further includes: The step of collecting road vehicle image data by using multiple drones, and pushing the collected vehicle image data / video stream to a streaming media server.

7. The vehicle target detection method based on multi-channel UAV video stream according to claim 5 is characterized in that: The step of reading multiple channels of image data from the streaming media server to obtain corresponding video frames of each channel includes: A1. Perform video streaming analysis on multiple image data channels and obtain multiple drone video streaming addresses from the streaming server; A2. Steps for extracting individual video frames from multiple drone video streams.

8. The vehicle target detection method based on multi-channel UAV video streams according to claim 5 is characterized in that: The step of establishing the improved algorithm network comprises: Steps to carry out partial bottleneck module design; Steps for designing dual-channel characteristic structures; The step of replacing the spatial pyramid pooling structure module design with a focused modulation network module.

9. The vehicle target detection method based on multi-channel UAV video streams according to claim 5 is characterized in that: The process of training the improved algorithm network model using the data set training model includes: The step of converting the format of the labeled data in the existing dataset to make it consistent with the input format of the YOLOv8 model; The steps of dividing the data set into a training set, a validation set and a test set, and classifying or merging the categories contained in the data set; The data in the data set is enhanced and the data set is expanded for training.

10. The vehicle target detection method based on multi-channel UAV video streams according to claim 5 is characterized in that: The step of sending the target vehicle video stream to be detected into the trained and improved algorithm network model for detection and identification specifically includes: Another trained and improved algorithm network model processing thread is started, and the algorithm network model extracts a frame image from the cache queue according to the generated Stream_ID; Use the trained target detection algorithm network to identify and detect each frame of the multi-channel drone video stream; When a vehicle target to be detected is detected, the vehicle detection system draws the output vehicle target information on the current frame image.

Images