Real-time highway spilled object detection method, system and equipment based on improved RT-DETR and medium
Through the improved RT-DETR model, combined with Repelan and multi-scale hybrid encoders, the real-time detection of highway throwing objects was achieved, solving the problem of high artificial detection costs and lack of real-time models, and significantly improved the identification of multi-scale throwing objects. ability.
Patent Information
- Application Number
- CN202411275501.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2025-01-28
AI Technical Summary
In the prior art, manual detection of sprinklers is costly and inefficient, and sprinkler detection models lack real-time performance and difficulty in detecting multi-scale sprinklers.
The improved RT-DETR model is called the Le-Detr model, combined with the Repelan feature extraction of the main network, multi-scale mixed encoder, and Transformer decoder to achieve real-time detection of highway throwing objects.
While maintaining real -time, it has significantly improved the ability to identify the multi -scale throwing objects. The design model can quickly and efficiently perform feature extraction and detection, reducing the detection cost and improving efficiency.
Smart Images
Figure CN119380289A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and intelligent transportation technology, and in particular to a real-time highway spilled object detection method, system, equipment and medium based on improved RT-DETR. Background Art
[0002] Spilled objects on highways seriously endanger driving safety and are very likely to cause traffic accidents. Therefore, it is very important to detect and clean up the spilled objects in a timely manner. At present, there are two main methods for detecting spilled objects: one is through manual detection, which relies on the staff of the traffic control center to conduct real-time monitoring through surveillance videos or road inspections through manual patrols. However, with the rapid expansion of highways and the increase in the number of monitoring equipment, this method has an increasing demand for human resources, resulting in a significant increase in management costs. At the same time, during the manual detection process, the spilled objects may not be discovered in time due to negligence, thereby reducing the detection efficiency.
[0003] Another approach is to use deep learning models and image processing technology for spill detection. This method has gradually become an effective means of spill detection and a future development trend. However, spill detection models still face the following challenges: First, although existing spill detection models have achieved certain results in accuracy, they are often not lightweight enough, lack real-time performance, and are difficult to be widely deployed in practical applications. Second, the inconsistent distance between the spill and the monitoring equipment, as well as the inherent size differences between different categories of spills, cause significant scale changes, which together exacerbate the recognition difficulty of the model. Summary of the invention
[0004] The present invention aims at the deficiencies in the prior art and provides a real-time highway spill detection method, system, device and medium based on improved RT-DETR, so as to solve the problems in the prior art of high cost and low efficiency of manual detection, lack of real-time performance of the spill detection model and difficulty in detecting multi-scale spills.
[0005] To achieve the above object, the present invention adopts the following technical solutions:
[0006] A real-time highway spilled object detection method based on improved RT-DETR, wherein the deep learning model used is improved on the basis of the RT-DETR model, and is called the LE-DETR model. The method comprises the following steps:
[0007] Collect surveillance videos of different sections and at different times on the highway, and obtain highway spillage image data by taking frames of the surveillance videos and removing duplicate images;
[0008] The highway spilled object image data is annotated using a rectangular annotation tool to obtain a highway spilled object dataset, which is then divided into a training set, a validation set, and a test set.
[0009] The LE-DETR model is trained with the training set, the parameters of each layer of LE-DETR are trained, and the LE-DETR model is tuned with the validation set to obtain the LE-DETR model to be tested; the LE-DETR model to be tested is tested with the test set, and if the accuracy rate meets the standard, the LE-DETR model that passes the test is obtained; otherwise, the LE-DETR model is retrained with the training set;
[0010] The tested LE-DETR model is used to detect spilled objects on highways and identify potential safety risks.
[0011] To optimize the above technical solutions, the specific measures taken also include:
[0012] Furthermore, the LE-DETR model includes: a feature extraction backbone network RLNet based on RepELAN, an efficient multi-scale hybrid encoder, an intersection-over-parallelization-aware selector, and a Transformer decoder equipped with an auxiliary prediction head;
[0013] The image of spilled objects on highways is input into the feature extraction backbone network, which extracts features to obtain feature maps. The feature maps are input into the efficient multi-scale hybrid encoder, which extracts multi-scale feature information of spilled objects from different receptive fields and performs feature fusion to obtain an image feature sequence. The intersection-over-union-ratio perceptual selector selects a fixed number of features from the image feature sequence as the initial object query for the decoder. The Transformer decoder equipped with an auxiliary prediction head iteratively optimizes the initial object query to generate the bounding box and confidence score of the detected spilled objects in the image.
[0014] Furthermore, the RepELAN-based feature extraction backbone network RLNet includes: two convolutional layers connected in series, a RepELAN, and three combination modules of Adown and RepELAN;
[0015] The input highway spilled object image first passes through two convolutional layers to output a 160×160×128 P2 layer feature map. Then, after passing through the first RepELAN, it passes through three Adown and RepELAN combination modules in succession to extract the feature information of the highway spilled object image. The first Adown and RepELAN combination module outputs a P3 layer feature map, the second Adown and RepELAN combination module outputs a P4 layer feature map, and the third Adown and RepELAN combination module outputs a P5 layer feature map.
[0016] Furthermore, the efficient multi-scale hybrid encoder includes: a CMI module, an AIFI module, two TFusion modules, two Fusion modules, two 1×1 Conv layers, and two 3×3 Conv layers with a step size of 2;
[0017] The CMI module fuses the P3 layer feature map, P4 layer feature map, and P5 layer feature map in the feature extraction backbone network RLNet to generate a 20×20×384 feature map. This feature map is sent to the AIFI module for intra-feature interaction. The result of the intra-feature interaction is adjusted through a 1×1 convolutional layer to generate a feature map T1. Feature map T1 is cross-layer fused with the P3 layer feature map and the P4 layer feature map in the first TFusion module. Subsequently, the output of the first TFusion module is again adjusted through a 1×1 convolutional layer to generate a feature map T2. Feature map T2 is fused with the P2 layer feature map. The feature map and the P3 layer feature map are fused in the second TFusion module. The output result of the second TFusion module is downsampled using a 3×3Conv layer with a step size of 2, and the result is fused with the feature map T2 in the first Fusion module. The fusion result of the first Fusion module is downsampled using a 3×3Conv layer with a step size of 2, and the downsampled result is fused with the feature map T1 in the second Fusion module. Finally, the output of the second TFusion module is concatenated with the results of the two Fusion modules in the channel dimension.
[0018] Further, the RepELAN includes the following components: a 1×1 convolutional layer for converting the input channel number c1 into an output channel number 2c; a Split function for splitting a feature map; a RepConv layer for converting the hidden channel number c into the intermediate channel number mid; a module list containing n-1 3×3 convolutional layers for convolution processing the feature map with the intermediate channel number mid; a 1×1 convolutional layer for converting the number of channels after splicing multiple feature maps into the output channel c2; a 1×1 convolutional layer for adding redundant feature maps; c1 represents the number of channels of the input feature map; c2 represents the number of channels of the output feature map; c represents the product of c2 and the hidden channel number scaling factor e, and the intermediate channel number mid is the product of the scaling factor and c; n-1 represents the number of convolutional layers in the module list;
[0019] The feature map is adjusted to a channel dimension through a 1×1 convolution layer. Then, a Split operation is performed to divide the feature map into a first part and a second part along the channel dimension. The second part passes through a RepConv layer, n-1 3×3 convolution layers, and a 1×1 convolution layer in sequence. After this process, the output results of each convolution layer passed by the first and second parts of the feature map are fused through a splicing operation. Finally, a 1×1 convolution layer is used to adjust the output channel dimension.
[0020] Furthermore, the structure of the CMI module is specifically as follows: the CMI module receives a P3 layer feature map, a P4 layer feature map, and a P5 layer feature map, the P3 layer feature map is downsampled by the Adown module, the P4 layer feature map is processed by a 1×1 convolution layer, and the P5 layer feature map is first upsampled by the nn.ConvTranspose2d() function, and then processed by a 1×1 convolution layer; the processed P3 layer feature map and the P5 layer feature map are fused with the P4 layer feature map along the channel dimension to form a fused feature map F, and then the fused feature map F is processed in parallel by DW convolutions of sizes 5×5, 7×7, 9×9, and 11×11, respectively, to extract multi-scale spatial features of different receptive fields;
[0021] Afterwards, the DW convolution results of each size are added element-by-element to the identity map of the fused feature map F, and the multi-scale features of different receptive fields are integrated through point-by-point convolution. The results after point-by-point convolution are again added element-by-element to the identity map of the fused feature map F. Finally, a 3×3 convolution layer is used to reduce the spatial dimension to generate a 20×20×384 feature map.
[0022] Furthermore, the TFusion module fuses the large, medium and small feature maps from the feature extraction backbone network RLNet; for the large-size feature map, the channel dimension of the feature map is first adjusted through a 1×1 convolutional layer, and then down-sampled using a parallel structure of maximum pooling and average pooling to reduce its spatial dimension, and the two pooling results are added point by point; for the small-size feature map, up-sampled using the nearest neighbor interpolation method to increase its spatial dimension; then, the feature maps of the large, medium and small branches are spliced in the channel dimension to fuse detail information and semantic information.
[0023] The present invention also proposes a real-time highway spilled object detection system based on improved RT-DETR, wherein the deep learning model used is improved on the basis of the RT-DETR model, and is called the LE-DETR model. The system comprises:
[0024] The data acquisition module is used to collect surveillance videos of different sections and at different times on the highway, and obtain the image data of spilled objects on the highway by taking frames of the surveillance videos and removing duplicate images;
[0025] A data partitioning module is used to annotate the highway spilled object image data using a rectangular annotation tool to obtain a highway spilled object data set, and to divide the highway spilled object data set into a training set, a validation set, and a test set;
[0026] The training module is used to train the LE-DETR model through the training set, train the parameters of each layer of LE-DETR, use the validation set to tune the LE-DETR model, and obtain the LE-DETR model to be tested; use the test set to test the LE-DETR model to be tested, if the accuracy rate meets the standard, the LE-DETR model that passes the test is obtained; otherwise, the LE-DETR model is retrained with the training set;
[0027] The detection module detects spilled objects on the highway based on the tested LE-DETR model and identifies potential safety risks.
[0028] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the real-time highway spilled object detection method based on the improved RT-DETR as described above is implemented.
[0029] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the real-time highway spilled object detection method based on the improved RT-DETR as described above.
[0030] The beneficial effects of the present invention are:
[0031] The present invention significantly improves the recognition capability of multi-scale scattered objects while maintaining real-time performance.
[0032] The RepELAN-based feature extraction backbone network RLNet designed in the present invention can perform fast and efficient feature extraction.
[0033] In the training phase, RepConv enhances the training performance of the model by executing 3×3Conv layers, 1×1Conv layers, and identity mapping in parallel. In the inference phase, RepConv is equivalently converted into a single-path 3×3Conv layer, making the inference faster. A scaling factor is also added to RepConv, which allows the scale and complexity of the model to be controlled by adjusting the factor, achieving flexible adjustment to adapt to different application scenarios.
[0034] The TFusion module is an improvement on the original Fusion module by connecting shallow low-level features to the neck part for feature enhancement, so as to better retain and utilize detailed feature information. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 is a flow chart of the method of the present invention;
[0036] Figure 2 This is the original RT-DETR network structure diagram;
[0037] Figure 3 It is the overall structure diagram of the LE-DETR network in the present invention;
[0038] Figure 4 It is a structural diagram of the RepELAN module in the present invention;
[0039] Figure 5 It is the structural diagram of TFusion module in the present invention;
[0040] Figure 6 It is a structural diagram of the CMI module in the present invention. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0042] Embodiment 1
[0043] The present invention proposes a real-time highway spilled object detection method based on improved RT-DETR. The deep learning model used is improved on the basis of the RT-DETR model, called the LE-DETR model. The overall process of the detection method is as follows: Figure 1 As shown, the following steps are included:
[0044] We collected surveillance videos from different sections and at different times on the highway, processed the surveillance videos, and removed duplicate images to obtain image data of spilled objects on the highway; a total of 4,878 images;
[0045] The highway spilled object image data is annotated using the rectangle annotation tool (Labelimg). Each image corresponds to a yolo format label file with the same name. Each line in the label file represents an object to be detected, as well as the information of each object detection rectangle. The highway spilled object dataset is obtained and divided into a training set, a validation set, and a test set according to the ratio of 8:1:1.
[0046] The LE-DETR model is trained with the training set, the parameters of each layer of LE-DETR are trained, and the LE-DETR model is tuned with the validation set to obtain the LE-DETR model to be tested; the LE-DETR model to be tested is tested with the test set, and if the accuracy rate meets the standard, the LE-DETR model that passes the test is obtained; otherwise, the LE-DETR model is retrained with the training set;
[0047] The tested LE-DETR model is used to detect spilled objects on highways and identify potential safety risks.
[0048] The structure of the original RT-DETR model is as follows Figure 2 As shown in the figure, it consists of four parts: a feature extraction backbone network, an efficient hybrid encoder, an IoU-aware Query Selection mechanism, and a Transformer decoder equipped with an auxiliary prediction head.
[0049] The structure of the LE-DETR model is as follows: Figure 3 As shown, it includes: a feature extraction backbone network RLNet based on RepELAN, an efficient multi-scale hybrid encoder, an IoU-aware Query Selection, and a Transformer decoder equipped with an auxiliary prediction head;
[0050] The image of spilled objects on highways is input into the feature extraction backbone network, which extracts features to obtain feature maps. The feature maps are input into the efficient multi-scale hybrid encoder, which extracts multi-scale feature information of spilled objects from different receptive fields and performs feature fusion to obtain an image feature sequence. The intersection-over-union-ratio perceptual selector selects a fixed number of features from the image feature sequence as the initial object query for the decoder. The Transformer decoder equipped with an auxiliary prediction head iteratively optimizes the initial object query to generate the bounding box and confidence score of the detected spilled objects in the image.
[0051] The RepELAN-based feature extraction backbone network RLNet includes: two convolutional layers connected in series, a RepELAN, and three combination modules of Adown and RepELAN;
[0052] The input highway spilled object image first passes through two convolutional layers to output a 160×160×128 P2 layer feature map. Then, after passing through the first RepELAN, it passes through three Adown and RepELAN combination modules in succession to extract the feature information of the highway spilled object image. The first Adown and RepELAN combination module outputs a P3 layer feature map, the second Adown and RepELAN combination module outputs a P4 layer feature map, and the third Adown and RepELAN combination module outputs a P5 layer feature map.
[0053] The structure of RepELAN is as follows Figure 4 As shown, it includes the following components: a 1×1 convolution (Conv) layer for converting the input channel number c1 into the output channel number 2c; a Split function for splitting the feature map; a RepConv layer for converting the hidden channel number c into the intermediate channel number mid; a module list containing n-1 3×3 convolution (Conv) layers for convolution processing the feature map with the intermediate channel number mid; a 1×1 convolution layer for converting the number of channels after splicing multiple feature maps into the output channel c2; a 1×1 convolution (Conv) layer for adding redundant feature maps; c1 represents the number of channels of the input feature map; c2 represents the number of channels of the output feature map; c represents the product of c2 and the hidden channel number scaling factor e, and the intermediate channel number mid is the product of the scaling factor and c; n-1 represents the number of convolution layers in the module list;
[0054] The feature map is adjusted to a channel dimension through a 1×1 convolution layer. Then, a Split operation is performed to divide the feature map into a first part and a second part along the channel dimension. The second part passes through a RepConv layer, n-1 3×3 convolution layers, and a 1×1 convolution layer in sequence. After this process, the output results of each convolution layer passed by the first and second parts of the feature map are fused through a splicing operation. Finally, a 1×1 convolution layer is used to adjust the output channel dimension.
[0055] In RepELAN, in order to speed up information flow and make the model lighter, the residual structure of the bottleneck layer in the traditional C2f module is removed, and a simpler n-1 3×3Conv layer is used to replace the n bottleneck layers in the C2f module. Although the residual structure helps improve model performance, it will significantly increase memory usage and access costs in actual operation, thereby reducing the detection speed. In addition, NVIDIA's computing library has been specially optimized for the 3×3Conv layer to ensure the efficiency of the operation. In order to compensate for the performance loss that may be caused by removing the residual connection during the training phase, RepConv, which decouples training and inference, is introduced and placed before n-1 3×3 convolutional layers. During the training phase, RepConv enhances the training performance of the model by executing 3×3Conv layers, 1×1Conv layers, and identity mapping in parallel. During the inference phase, RepConv is equivalently converted into a single-path 3×3Conv layer, making the inference speed faster.
[0056] After RepConv and n-1 3×3Conv layers, a 1×1Conv layer is added to generate redundant feature maps in a low-cost way, which is beneficial to improving network performance. Finally, the key part of RepELAN at inference time consists of n 3×3Conv layers and one 1×1Conv layer, which greatly improves the speed of information circulation and makes the model lighter while ensuring the improvement of detection accuracy. Subsequently, the results of each Conv layer are spliced and fused along the channel dimension. In addition, a scaling factor is added to this module, which allows the scale and complexity of the model to be controlled by adjusting this factor, so as to achieve flexible adjustment to adapt to different application scenarios.
[0057] The efficient multi-scale hybrid encoder includes: CMI module, AIFI module, two TFusion modules, two Fusion modules, two 1×1 Conv layers and two 3×3 Conv layers with a stride of 2;
[0058] The CMI module fuses the P3 layer feature map, P4 layer feature map, and P5 layer feature map in the feature extraction backbone network RLNet to generate a 20×20×384 feature map. This feature map is sent to the AIFI module for intra-feature interaction. The result of the intra-feature interaction is adjusted through a 1×1 convolutional layer to generate a feature map T1. Feature map T1 is cross-layer fused with the P3 layer feature map and the P4 layer feature map in the first TFusion module. Subsequently, the output of the first TFusion module is again adjusted through a 1×1 convolutional layer to generate a feature map T2. Feature map T2 is fused with the P2 layer feature map. The feature map and the P3 layer feature map are fused in the second TFusion module. The output of the second TFusion module is downsampled using a 3×3Conv layer with a step size of 2, and the result is fused with the feature map T2 in the first Fusion module. The fusion result of the first Fusion module is downsampled using a 3×3Conv layer with a step size of 2, and the downsampled result is fused with the feature map T1 in the second Fusion module. Finally, the output of the second TFusion module is concatenated with the results of the two Fusion modules in the channel dimension. This provides richer and more enhanced feature input for the subsequent IoU-aware Query Selection mechanism.
[0059] The structure of the CMI module is as follows Figure 6 As shown in the figure, specifically: the CMI module receives the P3 layer feature map, the P4 layer feature map and the P5 layer feature map, the P3 layer feature map is downsampled through the Adown module, the P4 layer feature map is processed by a 1×1 convolution layer, and the P5 layer feature map is first upsampled by the nn.ConvTranspose2d() function, and then processed by a 1×1 convolution layer; the processed P3 layer feature map and the P5 layer feature map are fused with the P4 layer feature map along the channel dimension to form a fused feature map F, and then the fused feature map F is processed in parallel by DW convolutions of sizes 5×5, 7×7, 9×9 and 11×11 respectively to extract multi-scale spatial features of different receptive fields; and to avoid producing overly sparse feature representations.
[0060] After that, the DW convolution results of each size are added element by element to the identity map of the fused feature map F, and the multi-scale features of different receptive fields are integrated through point-by-point convolution. The results after point-by-point convolution are again added element by element to the identity map of the fused feature map F. Finally, a 3×3 convolution layer is used to reduce the spatial dimension to generate a 20×20×384 feature map. This feature map contains rich multi-scale information. This feature map is then sent to the AIFI module for intra-feature interaction to generate a feature map with rich global context information, and then sent to the subsequent TFusion module for feature fusion to optimize detection performance.
[0061] In order to further enhance the performance of spilled object detection, the TFusion module is improved on the basis of the original Fusion module. It enhances the features by connecting the shallow low-level features to the neck part, so as to better retain and utilize the detailed feature information. Figure 5 As shown in the figure, the TFusion module fuses the large, medium and small size feature maps from the backbone. For the large size feature map, it is first processed by a 1×1 Conv layer, and then downsampled using a parallel structure of maximum pooling and average pooling to reduce the spatial dimension of the feature and achieve translation invariance, which helps to enhance the network's robustness to spatial changes and translations of the input image. For the small size feature map, it is upsampled by the nearest neighbor interpolation method to retain local features and prevent the loss of feature information of small objects. This method can effectively fill the feature map with neighboring pixels and take into account sub-pixel neighborhoods. Finally, the feature maps of these three sizes will be spliced in the channel dimension for further fusion processing.
[0062] The first TFusion module corresponds to Figure 3 The TFusion module below receives the feature maps P3 and P4 output by the last two RepELAN modules of RLNet and the output of AIFI through the 1×1 Conv layer, corresponding to large, medium, and small sizes respectively. After feature fusion, it outputs a feature map of size 40×40×256. The second TFusion module corresponds to Figure 3 The TFusion module above receives the feature maps of the P2 and P3 layers and the output of the first TFusion module through a 1×1 Conv layer, corresponding to large, medium, and small sizes respectively. After feature fusion, it outputs a feature map of size 80×80×256.
[0063] The LE-DETR model is trained using the training set to train the parameters of each layer of the network, and the model is tuned using the validation set to obtain the trained model; the trained model is tested using the test set to obtain the output; the model that passes the test is used for spilled objects detection on highways to identify potential safety risks.
[0064] The training set is input into the above LE-DETR model. After setting the hyperparameters, 150 rounds of training are started. The trained network model is verified with the validation set. Finally, the weight file with the best detection effect is selected as the weight file of the LE-DETR model, and the detection weight file is saved.
[0065] During the training process, the training set is used as the image for each round of training, mainly to determine the parameter variables obtained by the model through each round of learning, such as weight w and bias b. The validation set is used to determine the network structure and adjust the parameters of the network model in the later stage of the training process.
[0066] In this embodiment, the hyperparameters are set as follows: the image size / pixel is 640×640, the batch processing number is 4, the number of iterations is 150 rounds, the optimizer is AdamW, the initial learning rate is 0.0001, and the weight decay is 0.0001. Data enhancement includes random color distortion, expansion, cropping, flipping, and resizing operations.
[0067] The hardware system used in this experiment is Ubuntu, the GPU is NVIDIA TESLA P100 (16G), and the environment uses Python == 3.8, torch == 1.12.1, torchvision == 0.13.1, and CUDA == 11.3.
[0068] The test set is input into the above-trained LE-DETR model to obtain the prediction results after highway spilled objects detection.
[0069] Embodiment 2
[0070] The present invention proposes a real-time highway spilled object detection system based on improved RT-DETR corresponding to the method of embodiment 1. The deep learning model used is improved on the basis of the RT-DETR model, which is called LE-DETR model. The system includes:
[0071] The data acquisition module is used to collect surveillance videos of different sections and at different times on the highway, and obtain the image data of spilled objects on the highway by taking frames of the surveillance videos and removing duplicate images;
[0072] A data partitioning module is used to annotate the highway spilled object image data using a rectangular annotation tool to obtain a highway spilled object data set, and to divide the highway spilled object data set into a training set, a validation set, and a test set;
[0073] The training module is used to train the LE-DETR model through the training set, train the parameters of each layer of LE-DETR, use the validation set to tune the LE-DETR model, and obtain the LE-DETR model to be tested; use the test set to test the LE-DETR model to be tested, if the accuracy rate meets the standard, the LE-DETR model that passes the test is obtained; otherwise, the LE-DETR model is retrained with the training set;
[0074] The detection module detects spilled objects on the highway based on the tested LE-DETR model and identifies potential safety risks.
[0075] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.
[0076] Embodiment 3
[0077] The present invention proposes an electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the real-time highway spilled object detection method based on improved RT-DETR as described in Example 1 are implemented.
[0078] Embodiment 4
[0079] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the steps of the real-time highway spilled object detection method based on improved RT-DETR as described in Example 1.
[0080] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.
[0081] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0082] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.
Claims
1. A real-time highway spilled object detection method based on improved RT-DETR, characterized in that: The deep learning model used is improved on the basis of the RT-DETR model, called the LE-DETR model. The method comprises the following steps: Collect surveillance videos of different sections and at different times on the highway, and obtain highway spillage image data by taking frames of the surveillance videos and removing duplicate images; The highway spilled object image data is annotated using a rectangular annotation tool to obtain a highway spilled object dataset, which is then divided into a training set, a validation set, and a test set. The LE-DETR model is trained with the training set, the parameters of each layer of LE-DETR are trained, and the LE-DETR model is tuned with the validation set to obtain the LE-DETR model to be tested; the LE-DETR model to be tested is tested with the test set, and if the accuracy rate meets the standard, the LE-DETR model that passes the test is obtained; otherwise, the LE-DETR model is retrained with the training set; The tested LE-DETR model is used to detect spilled objects on highways and identify potential safety risks.
2. The real-time highway spilled object detection method based on improved RT-DETR as claimed in claim 1, characterized in that: The LE-DETR model includes: a feature extraction backbone network RLNet based on RepELAN, an efficient multi-scale hybrid encoder, an intersection-over-union-aware selector, and a Transformer decoder equipped with an auxiliary prediction head; The image of spilled objects on highways is input into the feature extraction backbone network, which extracts features to obtain feature maps. The feature maps are input into the efficient multi-scale hybrid encoder, which extracts multi-scale feature information of spilled objects from different receptive fields and performs feature fusion to obtain an image feature sequence. The intersection-over-union-ratio perceptual selector selects a fixed number of features from the image feature sequence as the initial object query for the decoder. The Transformer decoder equipped with an auxiliary prediction head iteratively optimizes the initial object query to generate the bounding box and confidence score of the detected spilled objects in the image.
3. The real-time highway spilled object detection method based on improved RT-DETR as claimed in claim 2, characterized in that: The RepELAN-based feature extraction backbone network RLNet includes: two convolutional layers connected in series, a RepELAN, and three combination modules of Adown and RepELAN; The input highway spilled object image first passes through two convolutional layers to output a 160×160×128 P2 layer feature map. Then, after passing through the first RepELAN, it passes through three Adown and RepELAN combination modules in succession to extract the feature information of the highway spilled object image. The first Adown and RepELAN combination module outputs a P3 layer feature map, the second Adown and RepELAN combination module outputs a P4 layer feature map, and the third Adown and RepELAN combination module outputs a P5 layer feature map.
4. The real-time highway spilled object detection method based on improved RT-DETR as claimed in claim 3, characterized in that: The efficient multi-scale hybrid encoder includes: a CMI module, an AIFI module, two TFusion modules, two Fusion modules, two 1×1 Conv layers and two 3×3 Conv layers with a step size of 2; The CMI module fuses the P3 layer feature map, P4 layer feature map and P5 layer feature map in the feature extraction backbone network RLNet to generate a 20×20×384 feature map, which is sent to the AIFI module for intra-feature interaction. The result of the intra-feature interaction is adjusted through a 1×1 convolution layer to generate a feature map T1. The feature map T1 is cross-layer fused with the P3 layer feature map and the P4 layer feature map in the first TFusion module. Subsequently, the output of the first TFusion module is again adjusted through a 1×1 convolution layer to generate a feature map T2. The feature map T2 is fused with the P2 layer feature map and the P3 layer feature map in the second TFusion module. The output result of the second TFusion module is downsampled using a 3×3Conv layer with a step size of 2, and the result is fused with the feature map T2 in the first Fusion module. After the Conv layer downsamples the fusion result of the first Fusion module, the downsampled result is fused with the feature map T1 in the second Fusion module; finally, the output of the second TFusion module is concatenated with the results of the two Fusion modules in the channel dimension.
5. The real-time highway spilled object detection method based on improved RT-DETR as claimed in claim 3, characterized in that: The RepELAN includes the following components: a 1×1 convolutional layer for converting the input channel number c1 into the output channel number 2c; a Split function for splitting the feature map; a RepConv layer for converting the hidden channel number c into the intermediate channel number mid; a module list containing n-1 3×3 convolutional layers for convolution processing the feature map with the intermediate channel number mid; a 1×1 convolutional layer for converting the channel number after splicing multiple feature maps into the output channel c2; A 1×1 convolutional layer for adding redundant feature maps; c1 represents the number of channels of the input feature map; c2 represents the number of channels of the output feature map; c represents the product of c2 and the hidden channel number scaling factor e, and the number of intermediate channels mid is the product of the scaling factor and c; n-1 represents the number of convolutional layers in the module list; The feature map is adjusted to a channel dimension through a 1×1 convolution layer. Then, a Split operation is performed to divide the feature map into a first part and a second part along the channel dimension. The second part passes through a RepConv layer, n-1 3×3 convolution layers, and a 1×1 convolution layer in sequence. After this process, the output results of each convolution layer passed by the first and second parts of the feature map are fused through a splicing operation. Finally, a 1×1 convolution layer is used to adjust the output channel dimension.
6. The real-time highway spilled object detection method based on improved RT-DETR as claimed in claim 4, characterized in that: The structure of the CMI module is as follows: the CMI module receives the P3 layer feature map, the P4 layer feature map and the P5 layer feature map, the P3 layer feature map is downsampled by the Adown module, the P4 layer feature map is processed by a 1×1 convolution layer, and the P5 layer feature map is first upsampled by the nn.ConvTranspose2d() function, and then processed by a 1×1 convolution layer; the processed P3 layer feature map and the P5 layer feature map are fused with the P4 layer feature map along the channel dimension to form a fused feature map F, and then the fused feature map F is processed in parallel by DW convolutions of sizes 5×5, 7×7, 9×9 and 11×11, respectively, to extract multi-scale spatial features of different receptive fields; Afterwards, the DW convolution results of each size are added element-by-element to the identity map of the fused feature map F, and the multi-scale features of different receptive fields are integrated through point-by-point convolution. The results after point-by-point convolution are again added element-by-element to the identity map of the fused feature map F. Finally, a 3×3 convolution layer is used to reduce the spatial dimension to generate a 20×20×384 feature map.
7. The real-time highway spilled object detection method based on improved RT-DETR as claimed in claim 4, characterized in that: The TFusion module fuses the large, medium and small feature maps from the feature extraction backbone network RLNet. For the large-size feature map, the channel dimension of the feature map is first adjusted by a 1×1 convolution layer, and then down-sampled by a parallel structure of maximum pooling and average pooling to reduce its spatial dimension, and the two pooling results are added point by point. For the small-size feature map, the nearest neighbor interpolation method is used to up-sample and increase its spatial dimension. Subsequently, the feature maps of the large, medium and small branches are concatenated in the channel dimension to fuse the detail information with the semantic information.
8. A real-time highway spilled object detection system based on improved RT-DETR, characterized in that: The deep learning model used is improved on the basis of the RT-DETR model, called the LE-DETR model. The system includes: The data acquisition module is used to collect surveillance videos of different sections and at different times on the highway, and obtain the image data of spilled objects on the highway by taking frames of the surveillance videos and removing duplicate images; A data partitioning module is used to annotate the highway spilled object image data using a rectangular annotation tool to obtain a highway spilled object data set, and to divide the highway spilled object data set into a training set, a validation set, and a test set; The training module is used to train the LE-DETR model through the training set, train the parameters of each layer of LE-DETR, use the validation set to tune the LE-DETR model, and obtain the LE-DETR model to be tested; use the test set to test the LE-DETR model to be tested, if the accuracy rate meets the standard, the LE-DETR model that passes the test is obtained; otherwise, the LE-DETR model is retrained with the training set; The detection module detects spilled objects on the highway based on the tested LE-DETR model and identifies potential safety risks.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the real-time highway spilled object detection method based on the improved RT-DETR as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables the computer to execute the real-time highway spilled object detection method based on improved RT-DETR as described in any one of claims 1-7.