Two-stage efficient target detection method in traffic scene

Abstract

The application relates to a two-stage efficient target detection method in a traffic scene, target detection network construction and training and reasoning; the target detection network construction comprises the following parts: a backbone network, a feature pyramid structure is used to fuse features of each stage of a VoVNetV2 network to obtain high-resolution features; a candidate region generation network, a CenterNet detection head is used to respectively predict candidate region center positions, sizes and positioning deviations, so that all candidate regions are obtained; a spatial attention module, according to features extracted by the backbone network and features in the candidate region generation network, attention operation is performed to obtain new features; and an RoI prediction network, features of the candidate regions in the features output by the spatial attention module are extracted, and then subsequent network conversion is performed to obtain target categories and positions. Through the two-stage target detection paradigm, the advantages of the CenterNet can be retained, the detection accuracy of the original CenterNet is greatly improved, and high real-time performance is achieved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and specifically to a method for target detection in traffic scenarios using a target detection network with high detection accuracy and high inference speed. Background Technology

[0002] In traffic scenarios such as autonomous driving, object detection technology is crucial. In real-world scenarios, high real-time performance and high detection accuracy are required to ensure rapid response and accurate identification of the surrounding environment. Although deep learning technology has significantly improved the performance of object detectors, the increasing complexity of model structures leads to a continuous increase in model size and computational cost, thus limiting the inference speed of object detection. This makes many existing object detectors unable to meet the real-time requirements of practical scenarios.

[0003] Current object detection methods generally require large model parameters and computational loads to achieve high detection accuracy, which severely limits their application in edge computing devices. Furthermore, the high inference time prevents their application in latency-sensitive autonomous driving. Therefore, finding a balance between speed and accuracy in object detection, and providing a high-speed, high-accuracy method, is a key research focus. To achieve this goal, it is necessary to continue exploring more efficient model structures, optimization algorithms, and fast inference techniques to better adapt to various complex traffic scenarios and ensure driving safety. Summary of the Invention

[0004] The purpose of this invention is to provide a method for target detection in traffic scenarios using a target detection network with high detection accuracy and high inference speed, in order to solve the problems of low accuracy and low real-time performance in existing target detection methods.

[0005] The technical solution adopted by this invention to achieve the above objectives is: a two-stage efficient target detection method in traffic scenarios, comprising constructing and training a target detection network, and performing target detection based on the trained target detection network; the construction of the target detection network includes the following steps:

[0006] Phase 1:

[0007] The backbone network fuses features from each stage of the VoVNetV2 network through a feature pyramid structure to obtain high-resolution semantic features.

[0008] The candidate region generation network uses the CenterNet detector head to predict the center position, size, and localization deviation of each candidate region, thereby obtaining all candidate regions.

[0009] The spatial attention module performs attention operations to obtain new features based on the semantic features extracted from the backbone network and the features in the candidate region generation network.

[0010] Phase Two:

[0011] The RoI prediction network extracts features at the candidate box positions from the features output by the spatial attention module, expands the obtained features into column vectors, and then passes them through two fully connected layers to obtain the target's category and location.

[0012] The backbone network includes a VoVNetV2 network and a feature fusion network; the feature fusion network has a pyramid structure and is used to obtain high-resolution features by upsampling low-resolution features and fusing features across scales.

[0013] The VoVNetV2 network improves the spatial resolution by 1 time through upsampling. The upsampled features are then fused with corresponding features of the same resolution in the VoVNetV2 network using a feature fusion network. During fusion, the features in the VoVNetV2 network are aligned with the number of channels by a 1×1 convolution.

[0014] The candidate region generation network extracts high-resolution features from the backbone network and performs feature transformation through convolutional layers with kernels of 3×3 and 1×1 to obtain features for candidate region prediction. These features are then used by the CenterNet detector head to detect the locations of all candidate regions.

[0015] The CenterNet detection head contains three branches, which are used to predict the center position of the candidate region, the size of the candidate region, and the localization deviation, respectively. All three branches output feature maps through independent 3×3 and 1×1 convolutional layers.

[0016] The spatial attention module performs the following steps:

[0017] F SAM =θ1(F BASE )·Sigmoid(θ2(F RPN ))

[0018] Among them, F BASE The backbone network extracts features, where θ1 and θ2 represent linear transformations using 1×1 convolutions. The sigmoid function maps the results to the range 0-1 to obtain a spatial attention feature map. This feature map is then compared with the linearly transformed F... BASE Element-wise multiplication yields the spatially attention-adjusted feature F. SAM ;

[0019] The features F in the candidate region generation network RPNAfter linear transformation and function mapping, and F SAM The merger yields a new feature F. RCNNS .

[0020] The RoI prediction network extracts the feature F output by the spatial attention module through RoI Align. RCNNS The features at the candidate region locations are obtained, the features are expanded into column vectors, and then passed through two fully connected layers to obtain the target category and location.

[0021] A two-stage high-efficiency target detection system for traffic scenarios includes a target detection model module and a target detection module;

[0022] The target detection model module includes:

[0023] The backbone network fuses features from each stage of the VoVNetV2 network through a feature pyramid structure to obtain high-resolution semantic features.

[0024] The candidate region generation network uses the CenterNet detector head to predict the center position, size, and localization deviation of each candidate region, thereby obtaining all candidate regions.

[0025] The spatial attention module performs attention operations to obtain new features based on the features extracted by the backbone network and the features in the candidate region generation network.

[0026] The RoI prediction network is used to extract features at the candidate box positions from the features output by the spatial attention module. The obtained features are expanded into column vectors, and then passed through two fully connected layers to obtain the target's category and location.

[0027] The target detection module is used to input real-time images into the target detection model module to obtain the identified target category and location.

[0028] Firstly, a training method for an object detection network is provided, including:

[0029] Obtain a set of images with annotation information, where the annotation information includes the bounding box of the object in each image and its category.

[0030] The image set is input into the proposed network for iterative training until the convergence condition of the proposed network is met, resulting in the trained target detection network.

[0031] In each iteration, the input image sequentially passes through four parts: the backbone network, the CenterNet detection head, the spatial attention module, and the RoI prediction branch. The CenterNet detection head inputs the candidate region locations of the target into the RoI prediction branch and calculates the CenterNet loss function based on the target's true location in the annotation information. The target location and category predicted by the RoI prediction branch, along with the true target location and category in the annotation information, are used to calculate the RCNN loss function. Finally, the model parameters of the detection model are adjusted based on the above two loss functions and the backpropagation algorithm, completing one iteration.

[0032] Secondly, a target detection network is provided, including:

[0033] A novel backbone network is proposed. After the image to be detected is input into the backbone network, initial features are obtained. These features are then processed by the CenterNet detection head to obtain the candidate region location of the target. On the other hand, they are processed by the spatial attention module to obtain the feature map needed by the RoI prediction branch. Finally, the RoI prediction branch integrates the spatial attention features and the candidate region location of the target to obtain the final target location and category.

[0034] The present invention has the following beneficial effects and advantages:

[0035] 1. This invention extends the single-stage CenterNet to a two-stage architecture. The first stage uses CenterNet as a Region Proposal Network (RPN), while the second stage employs a RoI prediction network similar to Faster R-CNN to further refine and classify the candidate boxes predicted in the first stage. This extension retains the advantages of CenterNet's anchor-free architecture while significantly improving the detection accuracy of the original CenterNet.

[0036] 2. This invention uses a backbone network based on VoVNetV2, which has the advantages of being lightweight, fast, and highly accurate. This backbone network is very suitable for current mainstream AI computing platforms.

[0037] 3. This invention enhances the response of foreground targets in feature maps and improves feature quality through a Spatial Attention Module (SAM). In the model, the RPN and RoI prediction networks each use two feature maps, which are then correlated using the Spatial Attention Module. This invention has demonstrated excellent performance in practical applications, exhibiting high accuracy and low latency when deployed on current mainstream AI computing platforms. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the overall network structure of the object detection network CenterNet-RCNNS proposed in this invention;

[0040] Figure 2 This is a schematic diagram of the backbone network portion of the detection network proposed in this invention;

[0041] Figure 3 This is a schematic diagram of the structure of the CenterNet detection head in the detection network proposed in this invention;

[0042] Figure 4 This is a schematic diagram of the spatial attention module in the detection network proposed in this invention;

[0043] Figure 5 This is a schematic diagram of the detection results of a specific embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0045] Please see Figure 1 This invention provides a novel object detection network, CenterNet-RCNNS, which mainly consists of four parts: a backbone network, a candidate region generation network, an attention module, and an RoI prediction network. Each part is described in detail below:

[0046] 1. Backbone Network

[0047] Please see Figure 2The backbone network is designed based on VoVNetV2. The high-resolution features extracted by the original VoVNetV2 network are located in the shallow layers of the network, resulting in weak expressive power and difficulty in effectively representing the target. While the low-resolution features in the deeper layers have strong expressive power, their downsampling factor is too large to accurately represent the target's center position. To extract high-resolution and expressive features, this invention adds a feature fusion network to VoVNetV2. Its network structure is basically the same as the feature pyramid network. This network obtains more expressive high-resolution features by upsampling low-resolution features and fusing features across scales. For a low-resolution feature map, its spatial resolution is first increased by a factor of 1 through upsampling (nearest neighbor upsampling is used here). The upsampled features are then fused with corresponding features of the same resolution in the VoVNetV2 network. During fusion, the features in the VoVNetV2 network need to undergo a 1×1 convolution to align the number of feature channels. The simplest fusion method chosen here is element-wise addition. Furthermore, to improve the fusion effect, the element-wise added features also need to undergo a 3×3 convolution. The above process will be repeated until the highest resolution fused feature is obtained, which will be used as the input feature for the detection part of CenterNet-RCNNS.

[0048] 2. Candidate Region Generation Network

[0049] In the candidate region generation network part, the high-resolution features F extracted from the backbone network are first... BASE After feature transformation through convolutional layers with kernels of 3×3 and 1×1, the feature F used for candidate box prediction is obtained. RPN This feature allows the CenterNet detector head to detect the locations of all candidate boxes. The network structure of the CenterNet detector head is as follows: Figure 3 As shown, the detection head contains three branches, predicting the target center position, target size, and localization deviation, respectively. Each branch outputs a final feature map through independent 3×3 and 1×1 convolutional layers. Unlike the original CenterNet's target center detection branch, which needs to detect the center position of each target class separately (outputting a size of C×H×W), the RPN's target center detection branch only needs to detect objects in the foreground of the image (outputting a size of 1×H×W). Post-processing the prediction results from these three branches yields the candidate bounding box positions in the input image.

[0050] 3. Spatial Attention Module

[0051] To improve the target localization and classification capabilities of RoI prediction networks, this invention designs a spatial attention module (SAM) to address the F... BASE Reweighted, the structure of this module is as follows Figure 4 As shown. Because network branches are generated in the candidate regions, feature F RPN It was used for foreground object detection, therefore F RPN It can be used to distinguish foreground and background features; with this in mind, the present invention reintegrates F BASE The feature distribution yields the feature F SAM This makes the foreground position features more prominent. The specific calculation process is shown in formula (1):

[0052] F SAM =θ1(F BASE )·Sigmoid(θ2(F RPN )) (1)

[0053] Where θ1 and θ2 both represent linear transformations, this invention simply uses a 1×1 convolution to represent this. The sigmoid function maps the result to the range of 0 and 1, obtaining a spatial attention feature map. This feature map is then compared with the linearly transformed F... BASE Element-wise multiplication yields the spatially attention-adjusted feature F. SAM In addition, F RPN It also contains a wealth of information that helps in target detection. To make full use of this, F RPN and F SAM The merger yields a new feature F. RCNNS .

[0054] 4. RoI Prediction Network

[0055] F RCNNS As input features to the RoI prediction network, the network first extracts F using RoI Align. RCNNS The features at the candidate box locations generated by the candidate region generation network are expanded into column vectors and then passed through two fully connected layers to obtain the target category and relative offset. The loss function of the RoI prediction network is defined as Equation (2):

[0056]

[0057] Here, i represents the index of the candidate box, and c i Let t represent the category vector of the i-th candidate box predicted by the RoI prediction network. i This represents the position offset of the i-th candidate box predicted by the network. and These represent the corresponding true results. Classification loss L cls Using cross-entropy loss, regression loss L reg Smooth L1 loss is used. This means that when the candidate box is not the background, the output is 1, which makes the regression loss L... regIt is only included in the calculation when the candidate box is not the background. i} and {t i The RoI prediction network uses classification and regression layers to make predictions, and α is used to weight the two losses.

[0058] For bounding box regression, this invention uses formulas (3)-(6) to encode the coordinates.

[0059] t x =(x gt -x p ) / x p (3)

[0060] t y =(y gt -y p ) / y p (4)

[0061] t w =log(w gt / w p (5)

[0062] t h =log(h gt / h p (6)

[0063] Here, x, y, w, and h represent the center coordinates of the target detection box and its length and width, respectively. gt and w p These represent the widths of the ground truth bounding box and the candidate bounding box, respectively (x, y, and h are similar). This detection box encoding method ensures that the two-stage coordinate regression is based on the prediction results of the first stage. That is, the RoI prediction network performs regression based on the candidate bounding boxes predicted by the candidate region generation network, reducing the difficulty of regressing the detection boxes and improving the accuracy of localization.

[0064] Finally, the overall training loss of CenterNet-RCNNS is defined by Equation (7), where β is used to balance the two loss functions.

[0065] L total =L CenterNet +βL RCNN (7)

[0066] L total The overall training loss, where L CenterNet This is the loss function for the standard CenterNet network.

[0067] 5. Experimental Environment and Configuration

[0068] The experimental environment for this example was built on a PC configured with an Intel Core i7-7700K CPU and an NVIDIA RTX 2080 GPU, running the Ubuntu operating system. The PyTorch deep learning framework and CUDA 11.1 were used for the experiments. The training dataset was the large-scale traffic scene dataset BDD100K, employing various data augmentation techniques such as random folding, random color changes, random scaling, and random cropping. The default image resolution input to the network during training was 512×512. The SGD optimizer was used during training, with an initial learning rate of 0.01. Training was conducted for 80k epochs, with the learning rate reduced to 0.001 and 0.0001 at epochs 50k and 70k, respectively. The batch size was set to 32. After NMS in the candidate region generation network, a maximum of 2000 candidate boxes were retained, and the IOU threshold was set to 0.7. During testing, the input images were uniformly scaled to 352×640, without any data augmentation. The RPN part retained a maximum of 256 candidate boxes, and the IoU threshold for NMS in the RoI prediction network was set to 0.9. The number of feature channels output by the feature fusion branch of the backbone network was set to 64, and the F... RPN The number of feature channels is set to 64, the number of intermediate features in the two fully connected layers of the RoI prediction network is set to 512, the output space of RoI Align is the pooled features of size 5×5, β is set to 1, and α is set to 1.

[0069] Under this experimental environment and parameter configuration, the detection accuracy (mAP) of the finally trained detection model on the BDD100K dataset was [value missing]. 75 The performance rating is 18.4, the number of parameters is 6.3M, the computational cost is 10.0G Flops, and the inference latency is 9.7ms. Figure 5 The actual detection results of the model are shown.

[0070] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A two-stage efficient target detection method in traffic scenarios, characterized in that, Construct and train an object detection network, and perform object detection based on the trained object detection network; the construction of the object detection network includes the following steps: Phase 1: The backbone network fuses features from each stage of the VoVNetV2 network through a feature pyramid structure to obtain high-resolution semantic features. A candidate region generation network uses a CenterNet detector to predict the center position, size, and localization error of each candidate region, thereby obtaining all candidate regions. The spatial attention module performs attention operations to obtain new features based on the semantic features extracted from the backbone network and the features in the candidate region generation network. Phase Two: The RoI prediction network extracts features at the candidate box positions from the features output by the spatial attention module, expands the obtained features into column vectors, and then passes them through two fully connected layers to obtain the target's category and location. The spatial attention module performs the following steps: ； in, This represents the features extracted by the backbone network. , Both represent linear transformations, using 1×1 convolution; This represents the activation function used to map the result to the range 0, 1, to obtain a spatial attention feature map; this feature map is then compared with the linearly transformed... Element-wise multiplication yields the spatially attention-adjusted features. ; Features in the candidate region generation network After linear transformation and function mapping, and Merging yields new features .

2. The two-stage efficient target detection method in a traffic scenario according to claim 1, characterized in that, The backbone network includes a VoVNetV2 network and a feature fusion network; the feature fusion network has a pyramid structure and is used to obtain high-resolution features by upsampling low-resolution features and fusing features across scales. The VoVNetV2 network improves the spatial resolution by 1 time through upsampling. The upsampled features are then fused with corresponding features of the same resolution in the VoVNetV2 network using a feature fusion network. During fusion, the features in the VoVNetV2 network are aligned with the number of channels by a 1×1 convolution.

3. The two-stage efficient target detection method in a traffic scenario according to claim 1, characterized in that, The candidate region generation network extracts high-resolution features from the backbone network and performs feature transformation through convolutional layers with kernels of 3×3 and 1×1 to obtain features for candidate region prediction. These features are then used by the CenterNet detector head to detect the locations of all candidate regions.

4. The two-stage efficient target detection method in a traffic scenario according to claim 3, characterized in that, The CenterNet detection head contains three branches, which are used to predict the center position of the candidate region, the size of the candidate region, and the localization deviation, respectively. All three branches output feature maps through independent 3×3 and 1×1 convolutional layers.

5. The two-stage efficient target detection method in a traffic scenario according to claim 1, characterized in that, The RoI prediction network extracts features from the spatial attention module output via RoI Align. The features at the candidate region locations are obtained, the features are expanded into column vectors, and then passed through two fully connected layers to obtain the target category and location.

6. A two-stage efficient target detection system for traffic scenarios, the system being used to implement the two-stage efficient target detection method for traffic scenarios as described in any one of claims 1-5, characterized in that, Includes an object detection model module and an object detection module; The target detection model module includes: The backbone network fuses features from each stage of the VoVNetV2 network using a feature pyramid structure to obtain high-resolution semantic features; A candidate region generation network uses a CenterNet detector to predict the center position, size, and localization error of each candidate region, thereby obtaining all candidate regions. The spatial attention module performs attention operations to obtain new features based on the features extracted by the backbone network and the features in the candidate region generation network. The RoI prediction network is used to extract features at the candidate box positions from the features output by the spatial attention module. The obtained features are expanded into column vectors, and then passed through two fully connected layers to obtain the target's category and location. The target detection module is used to input real-time images into the target detection model module to obtain the identified target category and location.

Images