End-to-end real-time complex road target detection method

Abstract

The invention discloses an end-to-end real-time complex road target detection method, and relates to the technical field of target detection, and the method comprises the following steps: S1, inputting a video frame into a backbone network, and obtaining three input image feature layers with different sizes; s2, inputting the image feature layer into a multi-domain feature redundancy elimination module to obtain a final redundancy elimination feature; s3, inputting the final redundancy-removed features into a hybrid encoder to perform feature extraction and integration, and then entering a hybrid position embedding module to perform fusion encoding on position information; and S4, inputting the output features into a decoder module for updating and learning, then generating learned target information through feedforward network processing, and finally generating coordinate information, category and confidence information containing a target in the image through a head network for identifying and positioning a vehicle in the image. According to the method, under the condition that the parameter quantity is kept unchanged, the performance of complex road target detection is remarkably improved, and the target can be rapidly and accurately detected.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to an end-to-end real-time complex road target detection method based on multi-domain feature decomposition and redundancy removal. Background Art

[0002] With the rapid development of autonomous driving technology and the increasing advancement of smart city construction, the demand for real-time, accurate perception and understanding of complex road environments has become more urgent. Autonomous driving systems and traffic flow analysis systems must be able to quickly and accurately detect and identify various types of targets on the road, such as vehicles, pedestrians, and traffic signs, under changing environmental conditions, including different weather, lighting, and various congestion conditions. However, the current road information perception technology still has problems of insufficient efficiency and accuracy when facing multiple challenges such as large changes in target size, occlusion between targets, and a sharp increase in the number of targets, especially in complex road scenes. Although deep learning-based methods, especially those using convolutional neural networks (CNNs) and Transformer technology, have made great achievements in image recognition and target detection, and have achieved excellent performance on multiple standard datasets, they still face challenges in dealing with actual complex road scenes. For example, due to mutual occlusion between targets and the high similarity of features between different categories of targets (such as taxis and ordinary cars), traditional methods often find it difficult to adapt to these complex scenes, resulting in poor detection results.

[0003] Road information perception is a core component in autonomous driving systems. It involves technologies for extracting, identifying, and understanding the surrounding environment from sensor data. Early research focused on using traditional image processing methods, such as edge detection and region segmentation, to identify objects such as roads and vehicles. With the development of machine learning technology, especially the rise of deep learning, methods based on deep convolutional neural networks (CNNs) have begun to dominate. These methods can automatically learn complex features, greatly improving the accuracy and robustness of perception. The current mainstream detection algorithms are mainly divided into single-stage and two-stage. The two-stage object detection algorithm first generates a series of candidate object regions (called region proposals), and then classifies and regresses these regions to determine the final object location and category. R-CNN is the pioneering work of the two-stage object detection algorithm. Its subsequent work, such as Fast R-CNN introduced the ROIPooling layer, Faster R-CNN introduced the region proposal network (RPN), and Mask R-CNN added the object segmentation branch, which further improved the detection accuracy and training speed. In the field of autonomous driving, many works have achieved good results by using this type of two-stage detector for road object detection.

[0004] However, the perception tasks in autonomous driving scenarios have extremely high requirements for real-time performance, and the speed of the two-stage detector is difficult to meet the needs of actual applications. In contrast, the one-stage detection algorithm can directly predict the category and location of the target from the input image without the need for a region extraction step, and the detection speed is usually better than the two-stage method. Representative one-stage detectors include SSD and the YOLO series. SSD uses feature maps of different scales to detect targets of different sizes; while the YOLO series converts the detection problem into a regression problem and directly predicts on the entire image. Thanks to its excellent design, the YOLO series has greatly improved the detection speed while maintaining high accuracy. Some studies, such as using one-stage detectors for road scenes, have demonstrated their potential in real-time perception.

[0005] In recent years, after the great success of Transformer in the field of NLP, it has been introduced into visual tasks. DETR was the first to propose an end-to-end object detection method based on Transformer. Unlike traditional methods, DETR abandons post-processing steps such as anchor boxes and NMS, and adopts a bipartite graph matching loss function for end-to-end training, which greatly simplifies the detection process. However, DETR has problems such as slow training and insufficient accuracy in small object detection. Subsequent work such as Deformable DETR and DN-DETR proposed improvement solutions to these shortcomings. RT-DETR is the first real-time DETR variant, which introduces a hybrid encoder to efficiently process multi-scale features and further improves the inference speed. Some studies have also performed traffic target perception tasks based on the DETR structure. The research progress of these Transformer detectors has injected new impetus into the development of road information perception systems. Although existing methods have made significant progress on standard datasets, they still face many challenges in complex road environments, such as large-scale changes, dense occlusion, and day-night illumination changes. At the same time, the features extracted by existing models are often redundant, especially when dealing with multi-target and multi-category detection tasks. Redundant features will waste computing resources and cover up effective information. Summary of the invention

[0006] The purpose of the present invention is to provide an end-to-end real-time complex road target detection method, which optimizes feature expression by decoupling and removing redundancy of multi-domain features, and adopts hybrid position coding to enhance the modeling of spatial information. It can achieve excellent performance in accuracy, real-time, complex background, target overlap detection, etc.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] An end-to-end real-time complex road target detection method comprises the following steps:

[0009] S1, input the video frame into the backbone network to obtain three layers of input image feature layers of different sizes;

[0010] S2: Input the image feature layer into the multi-domain feature de-redundancy module MDFD to obtain the final de-redundant feature X sfc ;

[0011] S3, the final de-redundant feature X sfc The input is a hybrid encoder for feature extraction and integration, and then enters a hybrid position embedding module for fusion encoding of position information;

[0012] S4: Input the output features in S3 into the decoder module and update the learning through the multi-scale deformable attention layer, then process it through the feedforward network to generate the learned target information, and finally generate the coordinate information, category, and confidence information of the target in the image through the head network, which is used to identify and locate the vehicle in the image.

[0013] Preferably, S2 comprises the following steps:

[0014] S21, input the image feature layer into the spatial feature de-redundancy module SDM to obtain the de-redundancy processed feature X s ;

[0015] S22, input the de-redundancy processed feature Xs into the frequency domain feature de-redundancy module FDM, and obtain the feature Xs after the frequency domain de-redundancy processing. sf ;

[0016] S23, input the feature Xsf after frequency domain de-redundancy processing into the channel de-redundancy module CDM to obtain the final de-redundancy feature X sfc .

[0017] Preferably, S21 comprises the following steps:

[0018] S211, the input image feature layer is subjected to group normalization processing, the feature channels are grouped, and a normalization operation is performed within each group;

[0019] S212, calculate the modulation factor S i , used to evaluate the importance of spatial information of each channel; for the modulation factor S i Perform normalization processing to generate spatial feature selection weights;

[0020] S213, using the generated selection weight W λ The normalized features of the group are weighted, and gated and nonlinearly transformed through Gate function and sigmoid to obtain the de-redundant feature X s .

[0021] Preferably, in S211, the calculation expression of the normalization operation is:

[0022]

[0023] Where μ and σ are the mean and variance of the features within the group, respectively; γ and β are trainable scaling and translation parameters; X redundant is the original frequency domain feature without redundancy removal processing; ε is an infinitely small constant;

[0024] In S212, the modulation factor S is calculated. i The expression is:

[0025] S i =|1-γ|+β (2);

[0026] The expression for generating spatial feature selection weights is:

[0027]

[0028] In S213, the expressions for gating operation and nonlinear transformation using Gate function and sigmoid are:

[0029]

[0030] Where C is the total number of feature channels; is element-wise multiplication.

[0031] Preferably, S22 comprises the following steps:

[0032] S221, feature X after redundancy removal s The frequency domain feature X is obtained by converting it to the frequency domain through fast Fourier transform FFT. frequency ;

[0033] S222, frequency domain characteristics X frequency Through the global feature filter w f Perform frequency domain filtering to obtain filtered frequency domain features;

[0034] S223, the filtered frequency domain features are converted back to the spatial domain through the inverse fast Fourier transform IFFT to obtain the feature X after frequency domain redundancy removal processing sf .

[0035] Preferably, the calculation expression of the Fast Fourier Transform FFT conversion is:

[0036]

[0037] The calculation expression of the frequency domain filtering is:

[0038]

[0039] The calculation expression of the inverse fast Fourier transform IFFT is:

[0040]

[0041] Where u, v are frequency domain coordinates; x, y are spatial coordinates; M x , M y are the lengths of the input in the x and y directions respectively; M u , M v are the lengths of the input in the u and v directions respectively; is element-wise multiplication; is the characteristic dimension; e is the natural logarithm; j is the imaginary unit.

[0042] Preferably, S23 comprises the following steps:

[0043] S231, output feature X from frequency domain feature de-redundancy module FDM sf To perform feature segmentation, the channel redundancy removal module CDM uses an adjustable factor α to convert the input feature X sf Divided into two groups along the channel dimension;

[0044] S232, the two sets of features are processed by point convolution and group convolution respectively;

[0045] S233, the processed features are fused through element-wise summation and element-wise multiplication operations;

[0046] S234: Concatenate the features processed by S233 along the channel dimension to obtain the final de-redundant feature X sfc .

[0047] Preferably, the input feature X is calculated by the following expression sf Divided into two groups along the channel dimension;

[0048] X sf_up =αC;

[0049] X sf_down =(1-α)C;

[0050] Point convolution and group convolution are processed as follows:

[0051] X sf_up After point convolution, we get X 1_pw ;

[0052] X sf_up After group convolution, we get X 1_gw ;

[0053] X sf_down After group convolution, we get X 2_gw ;

[0054] X sf_down After point convolution, we get X 2_pw ;

[0055] The calculation expression of element-wise summation and element-wise multiplication is:

[0056] Y1=PWC(X sf_up )⊕GWC(X sf_down ) (9);

[0057]

[0058] The calculation expression of the processed features concatenated along the channel dimension is:

[0059]

[0060] In the formula, Concat is concatenation along the channel dimension; PWC is point convolution; GWC is group convolution; ⊕ is element-level summation; is element-wise multiplication; C is the total number of feature channels.

[0061] Preferably, the hybrid position embedding module performs fusion encoding on the position information, including the following steps:

[0062] The input features enter the hybrid position embedding module and first undergo sine position encoding. The sine-cosine position encoding SPE uses sine and cosine functions of different frequencies to encode the position information:

[0063]

[0064] At the same time, the input features are also fused and encoded through a learnable position encoding module LPE:

[0065] query_pos=LPE(query)⊕SPE(query) (14);

[0066] Where pos is the position index; i is the dimension index; is the model dimension; SPE (pos,i) Encodes the position pos in dimension i; ⊕ is the element-wise sum.

[0067] Preferably, the calculation expression for updating the learning of the multi-scale deformable attention layer is:

[0068]

[0069] In the formula, is the input feature; q is the query element, which has content feature z p and the two-dimensional reference point p q ; m is the attention head; k is the key of the corresponding sample; p mqk and A mqk are the sampling offset and attention weight of the kth sampling point in the mth attention head respectively; W is the weight.

[0070] An end-to-end real-time complex road target detection system, the system is used to implement an end-to-end real-time complex road target detection method, comprising:

[0071] A memory storing executable program code;

[0072] a processor coupled to the memory;

[0073] The processor calls the executable program code stored in the memory to execute the steps of the end-to-end real-time complex road target detection method.

[0074] The present invention also provides an electronic device, comprising a memory and a processor, wherein the processor is used to implement the steps of an end-to-end real-time complex road target detection method when executing a computer management program stored in the memory.

[0075] The present invention also provides a computer-readable storage medium on which a computer management program is stored. When the computer management program is executed by a processor, the steps of the end-to-end real-time complex road target detection method are implemented.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] 1. Aiming at the feature extraction process of the existing detection model in complex road scenes, the present invention designs a new feature redundancy removal module MDFD, which can effectively perform multi-domain decomposition and redundancy removal on features in the space, channel and frequency domains, and optimize the processing of input features.

[0078] 2. The hybrid position encoder proposed in the present invention combines the high efficiency and generalization ability of fixed coding with the advantages of learnable coding to adaptively optimize the expression of position information, enhances the model's position perception ability, and allows the model to more accurately establish and utilize the position information between features, thereby improving the ability to process data with complex spatial relationships.

[0079] 3. While keeping the parameters unchanged, the present invention significantly improves the performance of complex road target detection and surpasses the current advanced real-time road information perception method, and can detect targets quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0081] Figure 2 It is a schematic diagram of the SDM structure of the present invention;

[0082] Figure 3 This is a schematic diagram of the structure of the frequency domain feature redundancy removal module FDFM of the present invention;

[0083] Figure 4 It is a schematic diagram of the structure of the hybrid position encoder MPE of the present invention;

[0084] Figure 5 It is a schematic diagram of the structure of the decoder of the present invention;

[0085] Figure 6 Schematic diagram comparing the detection effects of the present invention and other target detection models; wherein (a) is the original image; (b) is the detection result of the YOLOv5x model; (c) is the detection result of the YOLOv7x model; (d) is the detection result of the YOLOv8m model; and (e) is the detection result of the present invention. DETAILED DESCRIPTION

[0086] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0087] Embodiment 1:

[0088] See also Figure 1-Figure 5 , an end-to-end real-time complex road target detection method, comprising the following steps:

[0089] S1. Input the video frame into the backbone network Backbone. The backbone network Backbone can be ResNet series, Vision Transformer series, Swin-Transformer series, pResNet series, etc., supporting videos of different sizes such as 1080P, 2K, 4K as input. Before being sent to the backbone network, the input video frame will be processed into different sizes (416*416, 640*640, 960*960, 1280*1280), and normalization and other operations will be performed to obtain three layers of input image feature layers of different sizes.

[0090] S2, input the image feature layer (the last layer is input, and the other two layers enter the hybrid encoder for fusion) into the multi-domain feature de-redundancy module MDFD to obtain the final de-redundant feature X sfc ; Specifically include the following steps:

[0091] The image feature layer is input into the spatial feature de-redundancy module SDM. First, Group Normalization (GN) is used to group the input feature X into channels and normalize it within each group to balance the problems caused by Layer Norm (LN) and Batch Norm (BN) when normalizing the input features. The calculation expression is:

[0092]

[0093] Where μ and σ are the mean and variance of the features within the group, respectively; γ and β are trainable scaling and translation parameters; X redundant is the original frequency domain feature without redundancy removal processing; ε is an infinitely small constant; In the GN module, during the network learning process, it can decide whether and how to scale and offset each channel group to better adapt to different data distributions.

[0094] Next, using γ∈R C and β∈R C Constructing the modulation factor S i To measure the spatial information of each batch of input features. The larger the value of the modulation factor, the greater the change in spatial pixels, which indicates that the spatial information of the current channel is richer. The calculation formula is:

[0095] S i =|1-γ|+β (2);

[0096] Then the modulation factor S i After normalization, the spatial feature screening weights are obtained to screen effective spatial features.

[0097]

[0098] When the input features are filtered by weights W λ , the input features will be assigned weights of different importance. In order to avoid gradient disappearance during training, the input feature X and the new feature map are connected using the residual idea, and the obtained feature map is mapped to the range (0, 1) to ensure more stability during training. Finally, the feature map is gated using a preset threshold (through gate operation and nonlinear transformation using Gate function and sigmoid), and the effective information mask map of the input feature is obtained to remove redundant information of the input feature at the spatial level. (In this embodiment, the Gate threshold is set to 0.5).

[0099]

[0100] Where C is the total number of feature channels; is element-wise multiplication.

[0101] Finally, after the spatial feature redundancy removal module SDM, the de-redundant feature X is obtained. s These features have higher spatial information density, and redundant information is effectively reduced.

[0102] S22, remove the redundant feature X s Input it into the frequency domain feature de-redundancy module FDM to obtain the feature X after frequency domain de-redundancy processing sf ;

[0103] The frequency domain feature de-redundancy module FDM consists of a Fourier module and a learnable global feature filter. The features after the frequency domain feature de-redundancy module FDM will first undergo a fast Fourier transform to obtain the input frequency domain feature tensor.

[0104] The calculation expression of the fast Fourier transform FFT conversion is:

[0105]

[0106] At the same time, a set of learnable frequency domain feature global filters is initialized The frequency domain feature tensor is filtered. Since the global filter is learnable, the global filter will continuously update the frequency domain features during the network iteration, and then filter the input features in the frequency domain to achieve feature X. s The redundant feature removal work in the frequency domain.

[0107] The calculation expression of the frequency domain filtering is:

[0108]

[0109] Finally, the inverse Fourier transform IFFT is used to convert the filtered frequency domain features back to the spatial domain. The calculation expression is as follows:

[0110]

[0111] Where u, v are frequency domain coordinates; x, y are spatial coordinates; M x , M y are the lengths of the input in the x and y directions respectively; M u , M v are the lengths of the input in the u and v directions respectively; is element-wise multiplication; is the characteristic dimension; e is the natural logarithm; j is the imaginary unit.

[0112] After completing the frequency domain feature screening, in order to further remove feature redundancy, the feature Xsf after frequency domain redundancy removal processing is input into the channel redundancy removal module CDM to obtain the final redundancy removal feature X sfc .

[0113] like Figure 3 The left part shows the channel feature de-redundancy module FDM. In deep learning, operations related to "groups" are usually used to reduce the number of model parameters, improve computational efficiency, enhance the model's learning ability for different channels and features, and retain certain local feature relationships. The present invention divides the input features into two groups by channel through an adjustable factor α (0-1). In a specific embodiment, convolution is directly used for division. In order to extract more useful feature information at the channel layer, group convolution Group Wise (GWC) and point convolution Point Wise (PWC) are used. The calculation expressions of the group convolution and point convolution are:

[0114]

[0115] In the formula, The Cth out feature maps of output channels; The Cth in Feature maps of input channels; For group g from C in Input channel to C out The convolution kernel of the output channel; G is the total number of groups; For in Input channel to C out 1×1 convolution kernel with 1 output channels; For C out feature maps of output channels; For C in The feature map of the output channels.

[0116] After the adjustable factor α is divided, two sets of features are obtained. and X sf_up , X sf_down Send them to point convolution and group convolution respectively to extract features, and get

[0117] Four feature layers. After the adjustable factor α is divided, the two groups lose feature interaction, so X sf_upThe features obtained by the point convolution in the group and X sf_down The features obtained by group convolution in the group are subjected to EWS operation and the information extracted from them is added to obtain It helps to transfer more information in the network and alleviate the gradient disappearance problem. sf_up The features obtained by group convolution in the group and X sf_down The features obtained by the midpoint convolution of the group are subjected to the EWM operation to multiply the corresponding elements of the two tensors element by element to obtain Y2, and different weights are given to the features at different channel positions, which can strengthen or weaken the features at specific positions in the channel. The above calculation process is as follows:

[0118] Y1=PWC(X sf_up )⊕GWC(X sf_down ) (9);

[0119]

[0120] In order to make the model more likely to learn features with better generalization ability and capture the global information of the entire feature map, Y1 and Y2 are processed by global average pooling (GAP). The obtained features are then subjected to Softmax operation to obtain and Then, Y1 and c1, Y2 and c2 are respectively subjected to EWM and spliced ​​together according to the channel dimension to obtain Realize the redundancy of input features in frequency domain and channel features.

[0121] The description formula of the above process is as follows:

[0122]

[0123] In the formula, Concat is concatenation along the channel dimension; PWC is point convolution; GWC is group convolution; ⊕ is element-level summation; is element-wise multiplication; C is the total number of feature channels.

[0124] S3, the final de-redundant feature X sfc The input is a hybrid encoder (classical method RT-DETR) for feature extraction and integration, and the hybrid encoder combines different types of neural network layers, such as convolutional layers, attention layers, etc., for further extraction and integration (existing technology).

[0125] The features then enter the hybrid position embedding module to fuse the position information.

[0126] First, the input features are encoded by sine and cosine positions. This position encoding method does not increase the number of model parameters.

[0127] SPE uses sine and cosine functions of different frequencies to encode position information:

[0128]

[0129] Then, MLP is used to build a learnable position encoding module LPE, which can better handle inputs of various shapes and sizes, has high flexibility and generalization ability, and can effectively solve the problem of poor generalization of fixed position encoding SPE. Finally, the two obtained position encodings are fused to obtain a hybrid position encoding:

[0130]

[0131] query_pos=LPE(query)⊕SPE(query) (14);

[0132] Complex traffic scenes usually involve multiple types of targets and dynamically changing backgrounds. Learnable position encoding can be adaptively adjusted during the training process, allowing the model to better capture the spatial layout and relative position relationship of targets in specific scenarios. In complex traffic scenes, the relative position relationship between target objects (such as vehicles and pedestrians) is crucial for correct target detection. Learnable position encoding can help the model respond to these subtle position changes more sensitively, thereby improving the accuracy of detection. Combining the two position encoding methods can enrich the model's ability to express spatial information. Learnable position encoding provides position information optimized for specific tasks, while SPE position encoding introduces global position information that is mathematically continuous and periodic to the model. This combination enables the model to more comprehensively understand and express spatial relationships in complex traffic scenes.

[0133] S4. Input the output features in S3 into the decoder module and update the learning through the multi-scale deformable attention layer. First, the output features of the encoder enter the decoder. The input features pass through the IOU-aware query-selection module, which selects relevant queries and can be filtered based on the predicted intersection-over-union (IOU). The selected query passes through the denoising module to reduce noise and improve feature quality. The denoised query is combined with the traffic object position information (x, y, w, h) and processed by the hybrid position encoder (HPE). This step adds a clear physical meaning to the query, which helps the model better understand spatial information.

[0134] Next, the features are continuously updated and learned based on the query through the MSDeformAtten (multi-scale deformable attention) layer.

[0135]

[0136] In the formula, is the input feature; q is the query element, which has content feature z p and the two-dimensional reference point p q ; m is the attention head; k is the key of the corresponding sample; p mqk and A mqk are the sampling offset and attention weight of the kth sampling point in the mth attention head respectively; W is the weight.

[0137] The output of the deformable attention is processed through a series of feedforward network (FFN) layers and Add&Norm layers (residual connections and layer normalization). Finally, the decoder generates the learned target information through two parallel FFNs. The output of the decoder module passes through a head network. This network contains fully connected layers, convolutional layers, etc., which are used to map features to the final output space, including the coordinate information, category, and confidence information of the target in the image.

[0138] Finally, the output of the head network is processed into final predictions for identifying and localizing vehicles in the image, which are subsequently visualized on the original image to show the detected vehicles.

[0139] In a specific embodiment, the present invention collects and constructs a dataset focusing on road information perception in Chinese cities and highway environments. It contains a total of 100,000 images with a resolution of 1920x1200 pixels, covering labels such as "car", "pedestrian", "truck", "bus" and various traffic light states. The establishment of this dataset is intended to promote research on autonomous driving technology and road monitoring systems in complex urban and highway traffic scenarios.

[0140] The hardware environment used is: CPU model is Intel(R) Xeon(R) Gold 5218R@2.10GHz, memory size is 256.00GB, GPU model is NVIDIA GeForce RTX 4090(24G). The software environment is: operating system Ubuntu20.04.1LTS 64-bit, configuration environment is Pytorch1.9.0, python3.6, OpenCV3.4.0, CUDA11.1. The training parameters are batch size (Batch size) of 16, learning rate (Learning rate) of 0.001, training times (Epoch) of 100 times, using NVIDIA's parallel computing platform CUDA and programming model to increase the computing speed. All models use warm-up to accelerate the initial training.

[0141] like Figure 6 As shown, the detection results (FDR-DETR) of the present invention are compared with the current leading target detection models, including YOLOV5, YOLOV7 and YOLOV8. These visualizations will focus on showing the differences in the ability of each model to deal with target overlap and misidentification problems. Through the comparison of these actual application scenarios, better complex road detection effects can be achieved with fewer parameters and GFLOPs, with better FPS performance, and the superiority of the present invention in solving complex road perception tasks can be intuitively felt. The present invention not only opens up a new path for the development of complex road information perception technology, but also provides strong support for autonomous driving systems in dealing with target detection and recognition in complex environments.

[0142] Embodiment 2

[0143] An end-to-end real-time complex road target detection system, the system is used to implement an end-to-end real-time complex road target detection method, comprising:

[0144] A memory storing executable program code;

[0145] a processor coupled to the memory;

[0146] The processor calls the executable program code stored in the memory to execute the steps of the end-to-end real-time complex road target detection method.

[0147] Embodiment 3

[0148] The present invention also provides an electronic device, comprising a memory and a processor, wherein the processor is used to implement the steps of an end-to-end real-time complex road target detection method when executing a computer management program stored in the memory.

[0149] Embodiment 4

[0150] The present invention also provides a computer-readable storage medium on which a computer management program is stored. When the computer management program is executed by a processor, the steps of the end-to-end real-time complex road target detection method are implemented.

Claims

1. An end-to-end real-time complex road target detection method, characterized in that: The following steps are involved: S1, input the video frame into the backbone network to obtain three layers of input image feature layers of different sizes; S2: Input the image feature layer into the multi-domain feature de-redundancy module MDFD to obtain the final de-redundant feature X sfc ; S3, the final de-redundant feature X sfc The input is a hybrid encoder for feature extraction and integration, and then enters a hybrid position embedding module for fusion encoding of position information; S4: Input the output features in S3 into the decoder module and update the learning through the multi-scale deformable attention layer, then process it through the feedforward network to generate the learned target information, and finally pass it through the head network to generate the coordinate information, category, and confidence information of the target in the image, which is used to identify and locate the vehicle in the image.

2. The end-to-end real-time complex road target detection method according to claim 1, characterized in that: S2 includes the following steps: S21, input the image feature layer into the spatial feature de-redundancy module SDM, and obtain the de-redundancy processed feature X s ; S22, input the de-redundancy processed feature Xs into the frequency domain feature de-redundancy module FDM, and obtain the feature Xs after the frequency domain de-redundancy processing. sf ; S23, the feature X after frequency domain redundancy removal processing sf Input channel redundancy removal module CDM to obtain the final redundancy removal feature X sfc .

3. The end-to-end real-time complex road target detection method according to claim 2, characterized in that: S21 includes the following steps: S211, the input image feature layer is subjected to group normalization processing, the feature channels are grouped, and a normalization operation is performed within each group; S212, calculate the modulation factor S i , used to evaluate the importance of spatial information of each channel; for the modulation factor S i Perform normalization processing to generate spatial feature selection weights; S213, using the generated selection weight W λ The normalized features of the group are weighted, and gated and nonlinearly transformed through Gate function and sigmoid to obtain the de-redundant feature X s .

4. The end-to-end real-time complex road target detection method according to claim 3, characterized in that: In S211, the calculation expression of the normalization operation is: Where μ and σ are the mean and variance of the features within the group, respectively; γ and β are trainable scaling and translation parameters; X redundant It is the original frequency domain feature without redundancy removal processing; ε is an infinitely small constant; In S212, the modulation factor S is calculated. i The expression is: S i =|1-γ|+β (2); The expression for generating spatial feature selection weights is: In S213, the expressions for gating operation and nonlinear transformation using Gate function and sigmoid are: Where C is the total number of feature channels; is element-wise multiplication.

5. The end-to-end real-time complex road target detection method according to claim 3, characterized in that: S22 includes the following steps: S221, feature X after redundancy removal s The frequency domain feature X is obtained by converting it to the frequency domain through fast Fourier transform FFT. frequency ; S222, frequency domain characteristics X frequency Through the global feature filter w f Perform frequency domain filtering to obtain filtered frequency domain features; S223, the filtered frequency domain features are converted back to the spatial domain through the inverse fast Fourier transform IFFT to obtain the feature X after frequency domain redundancy removal processing sf .

6. The end-to-end real-time complex road target detection method according to claim 5, characterized in that: The calculation expression of the fast Fourier transform FFT conversion is: The calculation expression of the frequency domain filtering is: The calculation expression of the inverse fast Fourier transform IFFT is: Where u, v are frequency domain coordinates; x, y are spatial coordinates; M x , M y are the lengths of the input in the x and y directions respectively; M u , M v are the lengths of the input in the u and v directions respectively; is element-wise multiplication; is the feature dimension; e is the natural logarithm; j is the imaginary unit.

7. The end-to-end real-time complex road target detection method according to claim 2, characterized in that: S23 includes the following steps: S231, output feature X from frequency domain feature de-redundancy module FDM sf To perform feature segmentation, the channel redundancy removal module CDM uses an adjustable factor α to convert the input feature X sf Divided into two groups along the channel dimension; S232, the two sets of features are processed by point convolution and group convolution respectively; S233, the processed features are fused through element-level summation and element-level multiplication operations; S234: Concatenate the features processed by S233 along the channel dimension to obtain the final de-redundant feature X sfc .

8. The end-to-end real-time complex road target detection method according to claim 7, characterized in that: The input feature X is calculated by the following expression sf Divided into two groups along the channel dimension; X sf_up =αC; X sf_down =(1-α)C; Point convolution and group convolution are processed as follows: X sf_up After point convolution, we get X 1_pw ; X sf_up After group convolution, we get X 1_gw ; X sf_down After group convolution, we get X 2_gw ; X sf_down After point convolution, we get X 2_pw ; The calculation expression of element-wise summation and element-wise multiplication fusion is: Y1=PWC(X sf_up )⊕GWC(X sf_down ) (9); The calculation expression of the processed features concatenated along the channel dimension is: In the formula, Concat is concatenation along the channel dimension; PWC is point convolution; GWC is group convolution; ⊕ is element-level summation; is element-wise multiplication; C is the total number of feature channels.

9. The end-to-end real-time complex road target detection method according to claim 1, characterized in that: The hybrid position embedding module integrates the position information into the encoding The following steps are involved: The input features enter the hybrid position embedding module and first undergo sine position encoding. The sine-cosine position encoding SPE uses sine and cosine functions of different frequencies to encode the position information: At the same time, the input features are also fused and encoded through a learnable position encoding module LPE in two encoding methods: query_pos=LPE(query)⊕SPE(query) (14); In the formula, pox is the position index; i is the dimension index; is the model dimension; SPE (pos,i) Encodes the position pos in dimension i; ⊕ is the element-wise sum.

10. The end-to-end real-time complex road target detection method according to claim 1, characterized in that: The calculation expression for updating the learning of the multi-scale deformable attention layer is: In the formula, is the input feature; q is the query element, which has content feature z p and the two-dimensional reference point p q ; m is the attention head; k is the key of the corresponding sample; p mqk and A mqk are the sampling offset and attention weight of the kth sampling point in the mth attention head respectively; W is the weight.

Images