Road traffic target detection method fusing cnn and deep learning morpher

Abstract

The present application belongs to the technical field of computer vision, and specifically relates to a target detection method combining CNN and a deep learning transformer. The method comprises the following steps: constructing a combined model of CNN and a deep learning transformer, compressing parameters of the combined model, assigning labels, and designing a loss function. The present application combines CNN and a deep learning transformer, uses CNN to enhance local features, and uses the deep learning transformer to establish global context links to enhance the cascade network and capture global information of a feature map. A feature bridge is used to promote the flow of feature information between CNN and the deep learning transformer. Shared sublayers are designed to share weight parameters between adjacent convolution layers or cross convolution layers, reduce the number of convolution weight parameters, and improve the prediction speed of target detection. Optimal transmission allocation solves the label assignment problem and improves the detection accuracy of target detection. The target detection effect and speed are greatly improved through the cooperation of multiple aspects.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a road traffic target detection method that integrates CNN and deep learning deformers. Background Technology

[0002] Thanks to the development of big data and artificial intelligence, computer vision technology has seen unprecedented improvement and advancement. Object detection is widely used in security, transportation, surveillance, and medical fields. Deep convolutional neural networks (CNNs) are applied to object detection tasks due to their powerful ability to extract local feature information, but they lack the ability to acquire global information such as color, texture, and shape. Deep learning deformers, due to their outstanding ability to integrate long-range and short-range information, are widely used in natural language processing. Similarly, global feature information of images can also be extracted using deep learning deformers.

[0003] Sun Jian et al., in their 2016 paper *Computer Vision and Pattern Recognition*, proposed the Region Proposal Network (RPN) for Superfast Region Proposal Convolutional Neural Networks (Faster-RCNN). This network generates candidate regions by further convolving the feature map formed by the overall convolution of the input image. Furthermore, the RPN shares convolutional layers with the detection network, significantly improving the detection speed of Faster R-CNN. While the detection performance is significantly enhanced, relying solely on convolutional neural networks fails to extract global information, resulting in a relatively limited ability to do so.

[0004] Alexey et al.'s 2020 paper in *Computer Vision & Pattern Recognition* describes a deep learning deformer (ViT) for image recognition. Since Transformers can only accept one-dimensional information, it struggles with two-dimensional information like images. ViT utilizes the concept of patches to divide the image into small blocks, then transposes these blocks into one-dimensional information, which is then input into the Transformer network to obtain the image's distance information. While ViT obtains this distance information and thus captures the global picture, it fails to fully capture the image's features due to the lack of local feature extraction. Furthermore, ViT has a large number of parameters, making the model complex and resulting in long training and inference times.

[0005] Therefore, it is difficult for a standalone CNN or deep learning transformer network to extract all the features of an image. Thus, using a joint CNN and transformer network can effectively obtain global and local information of an image, and largely integrate all the feature information of the image, thereby providing useful information support for computer vision tasks.

[0006] One drawback of deep networks is their large number of parameters, which increases training and forward inference time. Combining CNNs with deep learning deformers further increases the number of parameters, hindering hardware implementation and practical deployment. Therefore, network compression is a necessary step. Different convolutional layers have parameter dependencies; leveraging these dependencies can reduce parameter redundancy and thus achieve network compression.

[0007] Label distribution in object detection significantly impacts detection accuracy. Optimal transport theory, a global optimization method, effectively addresses the relationship between suppliers and demanders. This problem can be solved using Sinkhorn-Knopp iteration. Summary of the Invention

[0008] The purpose of this invention is to propose an efficient and fast method for road traffic target detection that combines CNN and deep learning deformers.

[0009] The proposed road traffic target detection method combining CNN and deep learning deformers comprises four parts: CNN and deep learning deformer combined model construction, combined model parameter compression, label assignment, and loss function design; specifically:

[0010] (1) Construction of a combined model using CNN and deep learning deformers. For example... Figure 1 The combined model mainly consists of three parts: CNN branches, deep learning deformer branches, and feature bridges.

[0011] a. CNN Branch: The CNN branch mainly consists of 6 convolutional layers: 1×1, 3×3, 1×1, 1×1, 3×3, and 1×1. Each of these six convolutional layers is followed by a batch normalization layer. A residual network (ResNet) structure is used for feature extraction to prevent gradient vanishing or exploding.

[0012] b. Deep learning deformer branch: The deep learning deformer branch consists of four parts: batch normalization, multi-head self-attention mechanism, layer normalization and multilayer perceptron. Similarly, like the CNN branch, it also uses a residual network structure for feature extraction.

[0013] c. Feature bridge: In the information flow from the CNN branch to the deep learning deformer branch, it includes 1×1 convolution, downsampling, and layer normalization operations; in the information flow from the deep learning deformer to the CNN branch, it includes 1×1 convolution and batch normalization operations.

[0014] (2) Parameter compression of the combined model. This involves setting shared sublayers to achieve parameter sharing across different convolutional layers. Traditional convolutional neural networks use convolutional kernels for convolution operations. This invention refers to the traditional convolutional kernel as the original convolution. To reduce the number of convolutional kernels involved in the convolution operation, the convolutional kernel is decomposed into convolutional atoms and shared sublayers. The original convolution is obtained by matrix multiplication of the convolutional atoms and the shared sublayers. That is, by using shared sublayers, the number of parameters actually involved in the convolution operation is reduced. Let there be k convolutional kernels, C in For the number of input channels, C out Here, w and h represent the output channel number, w and h represent the width and height of the convolution kernel, and the original convolution kernel size is k×C. in ×C out ×w×h. If a shared parameter N is designed, then using the matrix multiplication principle, the existing network can be reduced by k×N×w×h, where the larger the shared parameter, the more parameters are reduced. For example... Figure 2 As shown, S is a shared sub-layer across different convolutional layers, A is a convolutional atom, and C is the original convolution. Now, S is the layer undergoing convolution. First, S performs the convolution operation, and then the convolution is multiplied by A to obtain the convolutional feature information. Since the number of parameters in S is less than the number of parameters in C, the number of parameters involved in the convolution can be significantly reduced.

[0015] By constructing the combined CNN and deep learning deformer model in step (1) and compressing the parameters of the combined model in step (2), the overall structure and parameters of the model can be obtained. The input image can be processed by the model to obtain feature information. Then, the classification and location regression of the target to be detected are achieved through the following step (3) label assignment.

[0016] (3). Label assignment. Each truth value (gt) is treated as a provider S that can provide a certain number of labels containing information to be detected. i Each assigned anchor can be viewed as a provider S that requires a unique label for the information to be detected. i If an assignment anchor receives enough measurement information labels from the ground truth (gt), then this assignment anchor is a positive sample of that gt. (Design provider S) i To demander D i Transmission scheme:

[0017] P={p(i,j)|i=1,2,…,m; j=1,2,…,n}, (1)

[0018] The optimization objective is:

[0019]

[0020] T ij ≥0, i=1,2,…,m,j=1,2,…,n;

[0021] Where m and n are the number of samples in the assigned anchor and the number of true value samples, respectively. C ij， T ij Let T be the information content of the allocator and the demander samples in each allocation process. Formula (1) refers to the overall allocation scheme from i to j. Formula (2) represents minimizing the cost function, that is, minimizing the total information content of the samples from each allocator to the demander. Formula (3) refers to T. ij The significance of this is that the Sinkhorn-Knopp iteration can be used to solve for and optimize the results. From this, the category and regression location of the target to be detected can be obtained during the target detection process.

[0022] (4). Loss function design;

[0023] The cost function in label assignment consists of two parts: positive labels and negative labels. For positive labels, the cost is derived from the truth value GT. i To the detection anchor a j cost It is the sum of classification loss and regression loss:

[0024]

[0025] Among them, L cls For classification loss, L reg To regress the loss, For the classification information of the allocation scheme, For the truth value of classification information; For the regression information in the allocation scheme, This represents the true regression information, where α is the weight parameter.

[0026] The specific process for determining the loss function scaling factor α in Formula 4 is as follows: First, as... Figure 3 We set seven different α values, taking values ​​of 0.3, 0.5, 1, 2, and 3 respectively. We trained on the MS-COCO dataset for 100 epochs and obtained the group with the smallest loss function value at convergence. We used 0.5 as the final ratio α.

[0027] The total loss function L consists of four parts, including the classification loss C of the CNN. cls The regression loss C of CNN reg The classification loss T of the transformer cls The regression loss T of the transformer reg :

[0028]

[0029] Where β is the ratio of the deep learning deformer to the CNN loss function. Step (4) introduces the composition of the loss function required during model training, which is only used during training and is not involved after the model parameters are determined.

[0030] The specific process for determining the loss function scaling factor β in formula (5) is as follows: First, as... Figure 4 Seven different β values ​​were set, namely 0.25, 0.33, 0.5, 1, 2, 3, and 4. The MS-COCO dataset was used for training for 12 epochs, and the group with the smallest loss function value at convergence was obtained. 3 was used as the final ratio β.

[0031] This invention combines CNNs and deep learning deformers, utilizing CNNs for local feature enhancement and deep learning deformers to build a global context module to enhance the cascaded network. Local feature enhancement expands the scope of feature extraction, while global context modeling captures global information from the feature map. Feature bridging facilitates the flow of feature information between the CNN and deep learning deformers. However, the fusion of CNNs and deep learning deformers increases the number of model parameters, raising model complexity and hindering training and normal prediction inference. To reduce model complexity, shared sublayers are designed to share weight parameters between adjacent or cross-convolutional layers, reducing the number of convolutional weight parameters and thus improving the inference and prediction speed of object detection. However, model compression affects the model's expressive power and object detection accuracy. To effectively improve the detection performance of the neural network without increasing network parameters, an optimal transfer allocation method is proposed to solve the label allocation problem. By optimizing the label distribution process, the detection accuracy of object detection is improved. Through these synergistic effects, the effectiveness and speed of object detection are significantly improved.

[0032] This invention designs a road image feature extraction model that combines CNN and deep learning deformers. By integrating both overall and local feature information, the model achieves better performance. Furthermore, the shared parameter design reduces parameter redundancy caused by the combination of CNN and deep learning deformers, thus compressing the model and accelerating training and prediction. After determining the feature extraction model, to further improve object detection accuracy, an optimal transmission scheme is used to optimize the label distribution during object detection, improving the accuracy of object detection classification and location regression. Testing on the publicly available MS-COCO dataset yields an average accuracy of 76.23%. Attached Figure Description

[0033] Figure 1 The proposed network structure is shown below.

[0034] Figure 2 Design the structure for the shared layer.

[0035] Figure 3 The curve is determined by the loss function's scaling factor α.

[0036] Figure 4 The curve is determined by the loss function scaling factor β.

[0037] Figure 5 The result is the target detection result. Detailed Implementation

[0038] The present invention will now be further described with reference to the accompanying drawings.

[0039] This invention proposes an efficient and fast algorithm for object detection, combining CNN and deep learning deformers to extract global and local feature information. The shared layer design reduces the number of network parameters, improving the speed of object detection. Optimal transport theory enhances the performance of label distribution, thereby improving object detection performance.

[0040] (1)Use Figure 1 The network structure uses a CNN and a deep learning deformer, with a feature bridge acting as a link between the two branches. The input to the CNN and the deep learning deformer is the image to be detected. Feature information flows between the two branches.

[0041] (2) Design a shared layer for the CNN network in step (1), setting the sharing coefficient to 128, and build a structure similar to... Figure 2 By using a shared layer network, the number of network parameters is reduced, thus obtaining the network structure for feature extraction.

[0042] (3) Input traffic images captured by road cameras, perform operations such as convolution, pooling, activation functions, and self-attention mechanisms on the pre-set network parameters, and complete the forward pass of the network in two branches: CNN and deep learning deformer, so as to obtain the feature information of the input traffic images.

[0043] (4) Using formulas (1), (2), and (3), construct the objective function for label distribution optimization, optimize the label distribution, obtain the optimal matching result, and obtain the target detection result, including the detection type and location.

[0044] The proposed model was used for testing, and partial test results for the target object were obtained as follows: Figure 4 Using the COCO dataset for testing, an average detection precision (AP) of 67.43% can be achieved.

Claims

1. A road traffic target detection method combining CNN and deep learning deformers, characterized in that, It includes four parts: building a combined CNN and deep learning deformer model, compressing the parameters of the combined model, label assignment, and designing the loss function; specifically: (1) Construction of a combined CNN and deep learning deformer model: The combined model consists of three parts: a CNN branch, a deep learning deformer branch, and a feature bridge. a. The CNN branch contains 6 convolutional layers, specifically 1×1 convolution, 3×3 convolution, 1×1 convolution, 1×1 convolution, 3×3 convolution, and 1×1 convolution. Each of these six convolutional layers is followed by a batch normalization layer. The residual network structure is used for feature extraction to prevent gradient vanishing or gradient exploding. b. The deep learning deformer branch consists of four parts: batch normalization, multi-head self-attention mechanism, layer normalization and multilayer perceptron. Similarly, like the CNN branch, it also uses a residual network structure for feature extraction. c. Feature bridge: In the information flow from the CNN branch to the deep learning deformer branch, it includes 1×1 convolution, downsampling, and layer normalization operations; in the information flow from the deep learning deformer to the CNN branch, it includes 1×1 convolution and batch normalization operations. (2). Parameter compression of combined models: By setting up shared sublayers, parameter sharing between different convolutional layers can be achieved, specifically as follows: The traditional convolution kernel is called the original convolution. In order to reduce the number of convolution kernels involved in the convolution operation, the convolution kernel is decomposed into convolution atoms and shared sub-layers. The original convolution is obtained by performing matrix multiplication on the convolution atoms and shared sub-layers. Let S be a shared sub-layer of different convolutional layers, A be a convolution atom, and C be the original convolution; then S is the one performing convolution now, that is, S is convolved first, and then the convolution is multiplied by A to obtain the convolved feature information; (3). Tag assignment: Each true value gt is treated as a provider S that provides a certain number of labels for the information to be detected. i Each allocation anchor is viewed as a provider S that requires a unique label of the information to be detected. i If a certain assignment anchor obtains enough test information labels from the ground truth (gt), then this assignment anchor is a positive sample of this gt; Design Provider S i To demander D i Transmission scheme: P={p(i,j)|i=1,2,…,m; j=1,2,…,n}, (1) The optimization objective is: T ij ≥0,i=1,2,…,m,j=1,2,…,n; Where m and n are the number of samples in the assigned anchor and the number of true value samples, respectively; C ij T ij These represent the information content of the allocator and the demander samples in each allocation process; Formula (1) refers to the overall allocation scheme from i to j; Formula (2) represents minimizing the cost function, that is, minimizing the total information content of the allocator and the demander samples; Formula (3) refers to T. ij The meaning it represents; the optimization results obtained by solving using Sinkhorn-Knopp iteration; (4) Loss function design: The cost function in label assignment consists of two parts: positive labels and negative labels. For positive labels, the cost is derived from the truth value gt. i To the detection anchor a j cost It is the sum of classification loss and regression loss: Among them, L cls For classification loss, L reg To regress the loss, For the classification information of the allocation scheme, For the truth value of classification information; For the regression information in the allocation scheme, For true regression information, α is the weight parameter; The total loss function L consists of four parts, including the classification loss C of the CNN. cls The regression loss C of CNN reg The classification loss T of the transformer cls The regression loss T of the transformer reg : Where β is the ratio of the deep learning deformer to the CNN loss function.

Images