A real-time detection method and device for intelligent transportation

Abstract

The present invention discloses a real-time detection method and device for intelligent transportation, the method comprising: obtaining a road or aerial image collected in real time, inputting a detection model established and trained in advance, and obtaining the position, size and category of a target to be detected; the detection model comprises a backbone layer, an FPN layer, three detection heads with the same structure, and an NMS network; wherein the backbone layer and the FPN layer are used to downsample the collected image, extract features, and obtain a class graph and a feature graph; the detection head is used to convert the class graph and the feature graph into sparse features by generating a class graph network, and then respectively input the class graph and the feature graph into a detection frame regression network and a classification network after sparse convolution, and obtain the position, size and category of the target to be detected; the NMS network is used to perform non-maximum suppression on the position, size and category of the target to be detected obtained by the three detection heads, and obtain the final position, size and category of the target to be detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent traffic real-time detection, and in particular relates to a real-time detection method and device for intelligent traffic. Background Art

[0002] In recent years, in the field of intelligent transportation, imaging technology has been widely used with the popularization of the industry and the advancement of computer vision technology. The neural network-based method of general target detection has achieved remarkable success on public data sets and improved the accuracy of target detection. For example, road section cameras combined with target detection can be used in road condition detection, autonomous driving, violation detection, vehicle tracking and other fields. Current methods mainly rely on cloud computing, which pursues higher accuracy by transmitting images back to the cloud server for reasoning. However, cloud computing requires transmitting images back to the terminal server, which requires a lot of communication bandwidth resources, especially when processing large amounts of high-definition video data. In addition, the latency of cloud computing may also become a problem because the data needs to be transmitted to the cloud first, then calculated and returned, which increases the processing time. To solve these problems, some new technologies and methods are being studied. One method is edge computing, which can perform calculations directly on the edge device where the data is generated, reducing the need for data transmission. This method can greatly reduce the use of communication bandwidth and increase processing speed. Therefore, edge target detectors need to achieve fast reasoning and low latency in resource-constrained traffic edge units.

[0003] Object detection classifies and locates objects through bounding boxes, which can be divided into two-step and one-step object detection. The two-step method includes candidate box generation and object classification, and the representative algorithms are RCNN and Mask RCNN. One-step methods such as YOLO and SSD directly sample at different locations in the image and use CNN for classification and regression. Improvement attempts include inserting a coordinate attention module (MCS-YOLO) and a deeper network structure (YOLO-V7).

[0004] Anchor-free detection methods, such as FCOS, avoid complex anchor calculations and improve computational speed. The latest anchor-free method DETR uses the attention mechanism of Transformer and performs well. For remote sensing detection, some improvements integrate the coordinate attention mechanism (YOLO-V5) and Transformer (RAST-YOLO). Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and to propose a real-time detection method and device for intelligent transportation.

[0006] In order to achieve the above object, the present invention proposes a real-time detection method for intelligent transportation, comprising:

[0007] Obtain road or aerial images collected in real time, input the pre-established and trained detection model, and obtain the location, size and category of the target to be detected;

[0008] The detection model includes a backbone layer, an FPN layer, three detection heads with the same structure, and an NMS network; wherein,

[0009] The backbone layer and the FPN layer are used to downsample the collected images, extract features, and obtain class graphs and feature graphs;

[0010] The detection head is used to convert the class graph and feature graph into sparse features by generating a class graph network, and then input them into the detection frame regression network and the classification network respectively after sparse convolution to obtain the position, size and category of the target to be detected;

[0011] The NMS network is used to perform non-maximum suppression on the position, size and category of the target to be detected obtained by the three detection heads respectively, so as to obtain the final position, size and category of the target to be detected.

[0012] Preferably, the input of the backbone layer is a road or aerial image of size (H*W*3), and the FPN layer includes three layers, each of which outputs features of size (H / n, W / n, channel), where H and W are the height and width of the aerial image, respectively, 3 represents RGB, and n is the downsampling multiple.

[0013] Preferably, the detection head includes: generating a class graph network and two branches, the first branch is a sparse convolutional network and a detection box regression network connected in series, and the second branch is a sparse convolutional network and a category network connected in series, wherein the sparse convolutional network structures of the two branches are the same.

[0014] Preferably, the sparse convolutional network is used to operate on the foreground area using a sparse mask.

[0015] Preferably, the class graph generation network is used to obtain a corresponding class graph according to an input feature graph, and the value in the class graph is between 0 and 1, representing a meaningful area ratio in the region.

[0016] Preferably, the method further comprises a step of training the detection model; comprising:

[0017] Create a training set, input the detection model, adjust the parameters, and obtain a trained detection model when the training requirements are met.

[0018] Preferably, in the training of the detection model, the network loss function Loss total for:

[0019] Loss total =Loss map +Loss cls+Loss reg

[0020] Loss map =α×Loss CM +(1-α)×Loss BCM

[0021] Among them, Loss map is the loss of the graph, Loss cls is the classification loss, Loss reg is the regression loss, Loss CM and Loss BCM denote the class graph loss and binary class graph loss respectively, and α is a manual parameter.

[0022] On the other hand, the present invention provides a real-time detection device for intelligent transportation, the device comprising:

[0023] The acquisition and detection module is used to obtain the road or aerial images collected in real time, input the pre-established and trained detection model, and obtain the location, size and category of the target to be detected;

[0024] The detection model includes a backbone layer, an FPN layer, three detection heads with the same structure, and an NMS network; wherein,

[0025] The backbone layer and the FPN layer are used to downsample the collected images, extract features, and obtain class graphs and feature graphs;

[0026] The detection head is used to convert the class graph and feature graph into sparse features by generating a class graph network, and then input them into the detection frame regression network and the classification network respectively after sparse convolution to obtain the position, size and category of the target to be detected;

[0027] The NMS network is used to perform non-maximum suppression on the position, size and category of the target to be detected obtained by the three detection heads respectively, so as to obtain the final position, size and category of the target to be detected.

[0028] Compared with the prior art, the advantages of the present invention are:

[0029] The SCCMDet method proposed in the present invention focuses on extracting the entire meaningful area from the image, supervises the selection of sparse features through real class mapping, improves the reasoning speed, and introduces the generative class graph network (GCMN) and the corresponding loss algorithm to better generate meaningful sparse features and improve the detector performance, thereby improving the performance while reducing the computational cost. The lightweight design and edge computing method make the device of the present invention suitable for platforms with limited resources but requiring real-time detectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall framework of the detection model of the present invention;

[0031] Figure 2 It is the process of generating a real class diagram;

[0032] Figure 3 It is the generated graph network structure in the detection model;

[0033] Figure 4 It is a schematic diagram of the specific method of sparse convolution. DETAILED DESCRIPTION

[0034] Existing technologies for object detection classify and locate objects through bounding boxes, which can be divided into two-step and one-step object detection. The two-step method includes candidate box generation and object classification, and representative algorithms include RCNN and Mask RCNN. One-step methods such as YOLO and SSD directly sample at different locations in the image and use CNN for classification and regression. Improvement attempts include inserting a coordinate attention module (MCS-YOLO) and a deeper network structure (YOLO-V7).

[0035] Anchor-free detection methods, such as FCOS, avoid complex anchor calculations and improve computational speed. The latest anchor-free method DETR uses the attention mechanism of Transformer and performs well. For remote sensing detection, some improvements integrate the coordinate attention mechanism (YOLO-V5) and Transformer (RAST-YOLO).

[0036] Sparse convolution is a promising alternative method that limits computation by operating convolution on sparsely sampled regions or channels. The SCCMDet method proposed in this paper focuses on extracting the entire meaningful region from the image, supervising the selection of sparse features through true class mapping, and improving the inference speed. At the same time, the Generative Class Graph Network (GCMN) and the corresponding loss algorithm are introduced to better generate meaningful sparse features and improve the detector performance. The research results show that the SCCMDet method designed by this invention improves performance while reducing computational cost.

[0037] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0038] Example 1

[0039] Embodiment 1 of the present invention proposes a real-time detection method for intelligent transportation. The specific steps are: obtaining a road or aerial image collected in real time, inputting a pre-established and trained detection model, and obtaining the location, size and category of the target to be detected. The collected images include aerial images collected by drones, road images collected in real time by cars, road images collected in real time by cameras deployed on the road side, etc. The following is an example of an aerial image of a drone.

[0040] Overall framework:

[0041] like Figure 1 As shown in Figure 1, the detection model includes a backbone layer, an FPN layer, three detection heads with the same structure, and an NMS network;

[0042] The backbone layer and FPN layer are used to downsample the aerial images or road images, extract features, and obtain class graphs and feature graphs;

[0043] The detection head is used to convert the class graph and feature graph into sparse features by generating a class graph network, and then input them into the detection box regression network and classification network respectively after sparse convolution to obtain the position, size and category of the target to be detected;

[0044] The NMS network is used to perform non-maximum suppression on the position, size and category of the target to be detected obtained by the three detection heads respectively, so as to obtain the final position, size and category of the target to be detected.

[0045] The base detector of this network is YOLOv8. In each FPN layer, an adaptive sparse convolutional network based on class graph is used to replace the detection head.

[0046] In SCCM, the input is the features from FPN. The input features are transformed into sparse features through GCMN, which represent redundant meaningful regions. The purpose of Generative Class Graph Network (GCMN) is to extract meaningful regions from the input features under the supervision of the real class graph.

[0047] The class map determines the sparse features, and it is important to extract reasonable class maps from the input features. The pixels in the object are defined as meaningful areas, and the value of the true class map is defined as the proportion of the target in each small block. According to this definition, we can generate a true class map from the training dataset. The size of the true class map varies according to the feature maps from different layers. For example, the size of the true class map in the P5 layer is different from that in the P4 layer. And the number of actual class maps depends on the number of detection heads. Under the supervision of the true class map, the model can be trained to generate more reasonable class maps and sparse features to improve the accuracy of the sparse convolution-based detector. After the sparse convolution generates sparse features, it replaces the orin convolution of YOLOv8, which can accelerate the drone remote sensing airborne detector.

[0048] Class Diagram:

[0049] The pixels in the object are defined as meaningful pixels, and the ground truth map is defined as a map containing the target ratio at each position. The size of the original input image is (H*W*3). After extracting features through the backbone layer and the FPN layer, the size of the original input image is (H*W*3), which is converted from three FPN layers into three features of size (H / n, W / n, channel). n is the downsampling multiple of the feature map relative to the original image. Each value of the feature map represents an acceptable field. For example, the value in (1,1,:) represents the upper left corner area of ​​the original image, and the value in (1,W / n,:) represents the upper left corner area of ​​the original image. This value is most affected by the (1:H / n,1:W / n,3) area. Of course, this value is also affected by other areas, and the closer the distance, the greater the impact.

[0050] In order to select more meaningful regions from the feature map, the class map should be the same size as the feature map. Figure 1 Similarly, the value of the class map also depends on the receptive field. Considering that the receptive field obtained by the deep convolutional network is large and the influence of distance is different, we choose the area of ​​the original image: ((p-1)H / n:pH / n, (q-1)H / n:qW / n,3) as the receptive field of the class map (p,q). Therefore, the value of the true class map at position (p,q) is the ratio of the meaningful pixel area. The numerical formula for calculating the true class map is as follows:

[0051]

[0052] RFM(p,q)=If Meaningful(i),for i in RF(p,q) (2)

[0053]

[0054]

[0055] Where RF represents the receptive field, RFM represents the meaningful pixels in RF, RCM is the real image, and (p,q) represents the location of the class image.

[0056] like Figure 2 As shown in the figure, the feature is downsampled 8 times by the backbone and FPN layers. In the sample patch (6,7), there are a total of 8*8 pixels and 20 objects. As a result, the position (6,7) of the real image is 20 / 64. After calculating each position of the real class image and each layer of FPNS, the real image generation process is completed.

[0057] A network that generates a category map:

[0058] After obtaining the real class diagram, the Generative Class Graph Network (GCMN) is designed to generate the class diagram under the supervision of the real class diagram. Figure 3 , the real class map of the FPN layer and the feature map of the same layer are the input of GCMN. After some convolutional layers, the feature (H*W*C) map will become a class map (H*W*1). Where C represents the channel, which is 3 here, representing RGB.

[0059] The value of the class graph is between 0 and 1, representing the meaningful area ratio in this area. In addition, a binary class graph is set that depends on the learnable parameter l. The formula is as follows:

[0060]

[0061]

[0062] CM(p,q)=Binary(CM(p,q)) (7)

[0063] Where T is the threshold of binarized class graph, and its value is between 0.1 and 0.5. l is a learnable parameter, Binary() is a binary function, and CM is a class graph. Then the real class graph is used to supervise the class graph generated from the feature graph. This method introduces the focal loss between BCM and RCM as the loss function. So a loss is designed to adapt to the situation. The formula of the loss function is as follows:

[0064] Loss CM =∑(CM (p,q) -RCM (p,q) ) 2 (8)

[0065]

[0066]

[0067] Loss map =α×Loss CM +(1-α)×Loss BCM (11)

[0068] Among them, Loss CM and Loss BCM Denote the loss of class graph and binary class graph respectively. map is the total loss of the class graph and α is a manual parameter.

[0069] Loss total =Loss map +Loss cls +Loss reg (12)

[0070] Sparse Convolutional Networks:

[0071] Most existing detection heads for UAV images perform convolution on the overall feature map. This makes full use of background information and is beneficial for the detection of small objects. However, it will incur significant computational cost, which is not suitable for resource-constrained UAV platforms. The foreground area only occupies a small part of the image. Therefore, traditional convolution wastes a lot of computation on useless information.

[0072] Traditional convolution treats each position equally, but wastes computation in sparse data. Sparse convolution reduces computational cost by abandoning computation on irrelevant positions. Submanifold sparse convolutional networks activate output only at valid positions. Various methods prune unimportant parts for efficient reasoning, such as SACT and QueryDet. To adapt to the foreground changes of drone images, some methods use additional networks to learn foreground areas or design adaptive multi-layer masks. Figure 4 As shown, it is a schematic diagram of the specific method of sparse convolution.

[0073] Sparse convolution learns to operate on foreground regions by using sparse masks. It can reduce unnecessary computations in background regions and effectively speed up the inference stage of various visual tasks.

[0074] Given an input feature s, its input channel is c in , the output channel is c out , the data dimension is d, and the size is For sparse convolution, the input features are restricted to p in , where p in Represents the valid area containing the foreground. The output feature is p out At position p, the sparse convolution is expressed as:

[0075]

[0076] p k =p+k represents the position around p, where k is the offset distance to p, and k enumerates all discrete positions in the kernel space X. p is a subset of X, omitting the empty positions in X as follows:

[0077] X p ={k|p+k∈P in ,k∈X}

[0078] If p out Contains p in The union of all the surrounding extended positions covered by X, then this process can be expressed as:

[0079] P n ={p+k|k∈X}

[0080] In this case, the formula represents a regular sparse convolution. However, this method expands the locations containing the foreground region in subsequent layers and increases the computational cost. in =p out This form of sparse convolution is called submanifold sparse convolution. Convolution is performed when the center of the convolution kernel is in the foreground area. This design reduces the amount of calculation and is commonly used. However, this method discards basic information from the background, thereby reducing the model's capabilities.

[0081] Training process:

[0082] 1) Dataset

[0083] We adopt VisDrone for training and evaluation in our experiments. VisDrone is designed for drone platforms and consists of 7,019 aerial images of 10 categories, including pedestrians, people, cars, vans, buses, trucks, motorcycles, bicycles, tricycles, and canopies of tricycles.

[0084] 2) Evaluation Metrics

[0085] We use mean average precision (mAP) as the accuracy evaluation index and FPS as the efficiency evaluation index. If we define TP as the positive samples classified as True, FP as the negative samples classified as True, TN as the negative samples classified as False, and FN as the negative samples classified as False. The precision and recall are calculated as:

[0086]

[0087]

[0088] Precision refers to the accuracy of our prediction. Recall is the accuracy of positive samples. Fps refers to the number of image frames processed by the model per second, which is used to measure the efficiency of processing image data. It can be calculated as:

[0089]

[0090] 3) Implementation details

[0091] We implemented our method based on PyTorch and the spconv toolkit. All models were trained on an NVIDIA 4090 GPU and tested on an NVIDIA 3090 GPU. For VisDrone, the batch size was set to 8 and the initial learning rate was 0.01. Loss map In is set to 0.6.

[0092] Example 2

[0093] Embodiment 2 of the present invention proposes a real-time detection device for intelligent transportation, which is implemented based on the method of embodiment 1. The device includes:

[0094] The acquisition and detection module is used to obtain the road or aerial images collected in real time, input the pre-established and trained detection model, and obtain the location, size and category of the target to be detected;

[0095] The detection model includes a backbone layer, an FPN layer, three detection heads with the same structure, and an NMS network; wherein,

[0096] The backbone layer and the FPN layer are used to downsample the collected images, extract features, and obtain class graphs and feature graphs;

[0097] The detection head is used to convert the class graph and feature graph into sparse features by generating a class graph network, and then input them into the detection frame regression network and the classification network respectively after sparse convolution to obtain the position, size and category of the target to be detected;

[0098] The NMS network is used to perform non-maximum suppression on the position, size and category of the target to be detected obtained by the three detection heads respectively, so as to obtain the final position, size and category of the target to be detected.

[0099] Comparative experiment:

[0100] 1 Benchmark and Comparative Experiments

[0101] To evaluate the performance of the model, we use 8 methods as benchmarks: Cascade RCNN, Retinanet, YOLOV5, YOLOV8, YOLOX, DETR, QueryDet, and CEASC (Retinanet). The results are as follows:

[0102] Table 1 Comparison between baseline methods and SCCMDet

[0103]

[0104] 2 Ablation Experiment

[0105] 1)GCMN CSQ and AMM

[0106] To demonstrate the usefulness of GCMN, we compared GCMN with CSQ and AMM. CSQ and AMM are proposed in the paper. They are both used to generate masks and select meaningful regions in the feature map to feed into the sparse convolutional layer. We have observed that the accuracy of Yolov8 detector is higher than Retinanet because Yolov8 is more powerful. Compared with AMM method, GCMN has almost the same inference speed but outperforms AMM method in accuracy. AMM method is a simple method that can generate masks without the need for true labels. As a SOTA method, CAESC mainly relies on the context-enhanced (CE) mode to improve accuracy. In contrast, CSQ method has the highest accuracy on Yolov8 detector and the second accuracy on Retinanet detector, and the slowest inference speed. CSQ method is fully convolutional at p5 layer and needs to be inferred sequentially. In summary, GCMN simply utilizes the ground truth in the form of labels and has improved in accuracy and inference speed.

[0107] Table 2 GCMN, CSQ and AMM

[0108]

[0109]

[0110] in conclusion:

[0111] This paper mainly studies how to improve the selection ability of meaningful parts in sparse convolution to speed up the reasoning speed. A sparse convolutional network based on class mapping for intelligent traffic real-time detection (SCCMDet) is proposed to solve the selection problem of sparse convolution. In terms of data preprocessing, SCCMDet obtains the real class mapping from the ground truth as a label to supervise the selection. In addition, a loss function is designed for the generative class graph network (GCMN) to explore the importance of features for generating binary class graphs and filter out more meaningful features. Comparative experiments show that this method is suitable for platforms with limited resources but requiring real-time detectors.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A real-time detection method for intelligent transportation, comprising: Obtain road or aerial images collected in real time, input the pre-established and trained detection model, and obtain the location, size and category of the target to be detected; The detection model includes a backbone layer, an FPN layer, three detection heads with the same structure, and an NMS network; wherein, The backbone layer and the FPN layer are used to downsample the collected images, extract features, and obtain class graphs and feature graphs; The detection head is used to convert the class graph and feature graph into sparse features by generating a class graph network, and then input them into the detection frame regression network and the classification network respectively after sparse convolution to obtain the position, size and category of the target to be detected; The NMS network is used to perform non-maximum suppression on the position, size and category of the target to be detected obtained by the three detection heads respectively, so as to obtain the final position, size and category of the target to be detected; The detection head includes: generating a class graph network and two branches, the first branch is a sparse convolutional network and a detection box regression network connected in series, and the second branch is a sparse convolutional network and a category network connected in series, wherein the sparse convolutional network structures of the two branches are the same; The generated class graph network is used to obtain a corresponding binary class graph according to the input feature graph, and the value in the binary class graph is between 0 and 1, representing a meaningful area ratio in the region; In the training of the detection model, the class graph generated from the feature graph is supervised by using the real class graph, and the calculation of the real class graph RCM satisfies the following formula: RFM(p,q)=IfMeaningful(i),for i in RF(p,q) Among them, (p,q) represents the location of the class map, RFM represents the meaningful pixels in RF, RF represents the receptive field, H and W are the height and width of the aerial image, respectively, and n is the downsampling multiple; Loss function for generating graph-like networks total for: Loss total =Loss map +Loss cls +Loss reg Loss map =α×Loss CM +(1-a)×Loss BCM Among them, Loss map is the loss of the graph, Loss cls is the classification loss, Loss reg is the regression loss, Loss CM and Loss BCM denote the class graph loss and binary class graph loss respectively, and α is a manual parameter.

2. The real-time detection method for intelligent transportation according to claim 1, characterized in that: The input of the backbone layer is a road or aerial image of size (H*W*3). The FPN layer includes three layers, each of which outputs features of size (H / n, W / n, channel), where H and W are the height and width of the aerial image, respectively, 3 represents RGB, and n is the downsampling multiple.

3. The real-time detection method for intelligent transportation according to claim 1, characterized in that: The sparse convolutional network is used to operate on the foreground area using a sparse mask.

4. The real-time detection method for intelligent transportation according to claim 1, characterized in that: The method also includes a step of training a detection model; including: Create a training set, input the detection model, adjust the parameters, and obtain a trained detection model when the training requirements are met.

5. A device for real-time detection of intelligent transportation based on claim 1, characterized in that: The device comprises: The acquisition and detection module is used to obtain the road or aerial images collected in real time, input the pre-established and trained detection model, and obtain the location, size and category of the target to be detected; The detection model includes a backbone layer, an FPN layer, three detection heads with the same structure, and an NMS network; wherein, The backbone layer and the FPN layer are used to downsample the collected images, extract features, and obtain class graphs and feature graphs; The detection head is used to convert the class graph and feature graph into sparse features by generating a class graph network, and then input them into the detection frame regression network and the classification network respectively after sparse convolution to obtain the position, size and category of the target to be detected; The NMS network is used to perform non-maximum suppression on the position, size and category of the target to be detected obtained by the three detection heads respectively, so as to obtain the final position, size and category of the target to be detected.

Images