Road thrown object classification detection method integrated with environmental information perception

Abstract

The invention discloses a road thrown object classification detection method integrated with environmental information perception. The method comprises the following steps: acquiring road thrown object image data; the method comprises the following steps: marking pixel point information belonging to different types by using a semantic segmentation model according to various real object information, such as roads, vehicles, thrown objects and the like, in an environment in front of a vehicle in running, and extracting refined information of each pixel point in thrown object image data; and using the trained target detection model to perform feature extraction on the refined information of the road spilled objects obtained by semantic segmentation, and outputting the detected road spilled objects and categories thereof. The method can detect the thrown objects which are static in the area in front of the running of the vehicle, is suitable for various complex scenes in the running process of the vehicle, is excellent in detection performance of various thrown objects, and has high practical value.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision technology and intelligent transportation, and in particular to a road spilled object classification detection method that incorporates environmental information perception. Background Art

[0002] Road spill detection is an important application in the Intelligent Transportation System (ITS), which is of great significance for maintaining road safety and reducing traffic accidents. With the continuous increase in traffic volume on the road, the number of accidents caused by spilled objects is also increasing. Most of these spilled objects are goods that have fallen from trucks that are not tied tightly or debris dropped by car failures, which poses a great threat to the safety of drivers and passengers.

[0003] Traditional methods of detecting road spills rely on manual inspections, which is not only inefficient but also costly. In addition, due to the complexity of the road driving environment, such as lighting changes and weather conditions, detection accuracy and timeliness become challenges.

[0004] To solve these problems, researchers began to explore automated detection methods based on video surveillance, and the application of computer vision technology has become an effective solution. By installing cameras on vehicles or mobile devices, combined with advanced image processing and deep learning algorithms, automatic detection and identification of spilled objects on the road can be achieved. This method not only saves labor costs, but also improves the accuracy and efficiency of detection. Summary of the invention

[0005] The present invention proposes a method based on image processing, semantic segmentation technology and deep learning, aiming to solve the problems faced by existing road spilled object classification detection, such as low detection robustness, poor timeliness and high labor costs caused by the various shapes and sizes of spilled objects and the complex and changeable environment.

[0006] In order to overcome the above-mentioned problems, the present invention provides a road spillage classification detection method integrating environmental information perception, and the specific steps are as follows:

[0007] S1: Use the vehicle-mounted high-definition IPC camera to capture clear video streams; use the NVIDIA GPU's high-performance video codec to reduce the video stream frame rate, and extract frames from the reduced-frame video to reduce the amount of data;

[0008] S2: Improved U-Net algorithm; in the standard U-Net structure, the network is improved by adding a convolutional block attention model (CBAM) in the skip connection, so that when the road spilled object image data is input into the U-Net for segmentation, the irrelevant areas in the input image are suppressed, and the significant features that are useful for specific tasks can be highlighted;

[0009] S3: Deploy RT-DETR object detector; and introduce convolutional conditional random field (ConvCRF) into the network to optimize the segmentation results;

[0010] S4: Modify the neck network of the RT-DETR object detector; replace the neck network with BiFPN (Bidirectional Feature Pyramid Network - Cross-Scale Connection and Weighted Feature Fusion), and use a multi-scale feature fusion network so that the model can consider the importance of different input features and improve model efficiency;

[0011] S5: U-Net environment perception network and RT-DETR target detector are efficiently connected; based on the improved U-Net algorithm, the environmental information of the area in front of the vehicle is quickly perceived, and the semantic information of potential spilled objects on the road is extracted; and based on the improved RT-DETR target detector, the spilled objects on the road are quickly and accurately identified;

[0012] Furthermore, the step S1 includes:

[0013] S101. Use the vehicle-mounted high-definition IPC camera to capture a high-definition video stream of the area in front of the vehicle.

[0014] S102. To reduce the amount of data and improve processing efficiency, the high-performance video codec of NVIDIA GPU is used to reduce the frame rate of the video stream. First, ensure that the NVIDIA graphics card and corresponding driver that supports CUDA are installed in the system; then obtain the FFmpeg source code and install the corresponding compilation environment, and configure the corresponding parameters to enable CUDA support; after successfully compiling FFmpeg, you can use NVIDIA's hardware codec for video processing.

[0015] Furthermore, the step S2 includes:

[0016] S201. The improved U-Net mainly adds a convolutional attention block (CBAM) to the original network. The attention block enhances the network's attention to important features by learning the weights of the feature map. This method can improve the network's ability to capture details, so that when the road spilled object image data is input into the U-Net for segmentation, it can adapt to different target features and suppress irrelevant areas in the input image, while highlighting significant features that are useful for specific tasks.

[0017] S202, CBAM module first applies channel attention to the input feature map. The purpose of the channel attention module is to determine which channels are important. It compresses the spatial dimension through global average pooling (AvgPool) and global maximum pooling (MaxPool), retains channel information, and then uses a shared multi-layer perceptron (MLP) to extract features. The output of channel attention can be calculated by the following formula:

[0018] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0019] Among them, F is the input feature map, AvgPool and MaxPool represent global average pooling and maximum pooling operations respectively, MLP is a multi-layer perceptron, and σ represents the Sigmoid activation function.

[0020] S203, the output of the channel attention is passed through the spatial attention module, and finally a feature map processed by both the channel and spatial attention mechanisms is obtained. The purpose of the spatial attention module is to determine which spatial positions in the feature map are important. It compresses the channel dimension to 1 by performing maximum pooling and average pooling on the channel dimension, retains the spatial information, and then extracts the spatial features through a convolutional layer. The output of the spatial attention can be calculated by the following formula:

[0021] M S (F) = σ(f 7×7 ([AvgPool(F); MaxPool(F)]))

[0022] Among them, f 7×7 represents the convolution operation, [AvgPool(F); MaxPool(F)] means connecting the feature maps of average pooling and maximum pooling, and σ represents the Sigmoid activation function.

[0023] Furthermore, the step S3 includes:

[0024] S301. Deploy RT-DETR as a target detector for road spills. This model is an end-to-end target detection model. It removes the post-processing strategy of non-maximum suppression (NMS), which requires less training time, and RT-DETR has less reliance on data enhancement. Under the same test conditions, it performs better and has better performance and balance. In addition, RT-DETR uses an efficient hybrid encoder and IoU perception, which effectively reduces computational costs and improves detection accuracy. It exceeds YOLOv8 in both speed and accuracy.

[0025] S302. A convolutional conditional random field (ConvCRF) is introduced after the RT-DETR decoder to optimize the detection results in the post-processing stage of the model, especially in terms of boundary refinement.

[0026] S303, introduce the convolutional conditional random field target detector, in the post-processing process, the input feature vector f1...f d, define a Gaussian kernel g, whose kernel matrix K has the shape of [bs,h,w], where [bs,c,h,w] represent the batch size, number of categories, input height and width respectively. The formula of Gaussian kernel g is:

[0027]

[0028] Among them, k g represents the Gaussian kernel function, b represents the batch index, dx, dy represents the offset of the convolution kernel in the x and y directions, x, y represents the pixel position currently processed, d represents the number of feature dimensions, and f i represents the function of the i-th feature dimension, θ i are learnable parameters.

[0029] S304. The information transmission result Q of all s cores is defined as:

[0030]

[0031] Where Q represents the message passing output of category c at (x, y), c represents the category index, k represents the filter size, K represents the fused kernel, and P represents the input segmentation probability map.

[0032] S305, ConvCRF is applied to the output feature map of RT-DETR, which can refine and improve the segmentation results of the U-Net network and improve the boundary accuracy of the scattered objects on the road.

[0033] Furthermore, the step S4 includes:

[0034] S401. In the RT-DETR target detector, BiFPN (as an improvement strategy, it is mainly used in the neck part of the network to enhance the ability of feature fusion. BiFPN achieves efficient aggregation of multi-scale features through bidirectional cross-scale connections and weighted feature fusion.

[0035] S402, cross-scale connection: BiFPN first fuses features of different scales. Given a multi-scale feature list in Represents the i-th input feature, BiFPN uses a learnable weight w i Perform weighted fusion on these features:

[0036] O=∑ i w i I i

[0037] Among them, O is the fused feature, w i are learnable weights that can be scalars (per feature), vectors (per channel), or multi-dimensional tensors (per pixel).

[0038] S403, weighted feature fusion; BiFPN uses fast normalized fusion to normalize weights so that all weights are normalized to probabilities between 0 and 1, indicating the importance of each input. Fast normalized fusion is calculated as:

[0039]

[0040] Where ∈ is a small value to avoid numerical instability, usually 0.0001. The value of each normalized weight of this method also falls between 0 and 1, and there is no Softmax operation, so it is more efficient.

[0041] Beneficial benefits of the present invention:

[0042] 1. Use NVIDIA GPU high-performance video codec to process the collected high-definition video stream, which can improve the speed and efficiency of video processing.

[0043] 2. Introducing the CBAM convolutional attention module containing channel attention blocks and spatial attention blocks into the U-Net semantic segmentation model can enable the model to fully capture the environmental information in the scattered object image data, thereby improving the model performance.

[0044] 3. Using the end-to-end target detector RT-DETR based on the Transformer architecture as the target detection model for road spilled objects can effectively overcome the speed delay and optimization difficulties brought by the non-maximum suppression (NMS) used in traditional target detectors such as the YOLO series, and its end-to-end detection process can greatly enhance the timeliness and accuracy of road spilled object detection.

[0045] 4. Introducing the convolutional conditional random field (ConvCRF) in the RT-DETR target detector for post-processing optimization can refine and improve the segmentation results of the U-Net network and improve the feature expression effect of scattered objects on the road.

[0046] 5. Modify the neck network of the RT-DETR target detector to a BiFPN network with optimized cross-scale connections and weighted feature fusion, which can enable the target detector to change the better features of capturing scattered objects, improve the feature expression ability, and use fewer parameters and calculations to achieve the same detection effect, thereby making the model lighter. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 is a flow chart of a road spillage classification detection method incorporating environmental information perception in the present invention;

[0049] Figure 2 It is a network architecture diagram of the improved U-Net model proposed in the present invention;

[0050] Figure 3 This is the architecture diagram of the CBAM convolutional attention module added to the U-Net network.

[0051] Figure 4 This is a comparison of the neck network before and after the improvement of the RT-DETR target detector.

[0052] Figure 5 It is the overall framework diagram of the neck network of the improved RT-DETR target detector proposed in the present invention. DETAILED DESCRIPTION

[0053] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0054] like Figure 1 As shown: The present invention proposes a method for classifying and detecting road spilled objects by integrating environmental information perception, and the specific implementation method is as follows:

[0055] Step S1: Obtaining image data of road spills

[0056] (1) In the process of acquiring image data, the vehicle-mounted high-definition IPC camera is first used to capture a high-resolution video stream to ensure that the details in the image are clear enough for subsequent processing and analysis. The camera can capture a video with a resolution of 1080P, that is, a high-definition video of 1920×1080 pixels. This resolution can provide a clear picture, so that details such as vehicles, pedestrians, and spilled objects on the road can be accurately captured.

[0057] (2) Use the high-performance video codec of NVIDIA GPU to perform frame reduction processing on the video stream. The original video stream is recorded at 30 frames per second (FPS). For real-time processing, this frame rate may be too high, resulting in too much data. Through frame reduction processing, the frame rate of the video stream can be reduced to a reasonable level, such as 15 frames per second, which can maintain the smoothness of the video and reduce the amount of data processed.

[0058] (3) Next, frame extraction is performed to form image data. It is set to extract an image frame every 5 frames, which means that in a video stream of 15 frames per second, 3 frames of images will be extracted for analysis. Such a frame extraction strategy can further reduce the amount of data while still being able to obtain enough information for effective image analysis.

[0059] (4) Summary of road spilled material image data. The present invention uses a total of 30,000 data images, of which 20,000 were collected by my team and the other 10,000 were from open source datasets on the Internet. About 10,000 images were used for subsequent model training and validation, which were divided into training set, validation set and test set according to the ratio of 6:2:2.

[0060] Step S2: Perception of environmental information of road spillage.

[0061] The present invention adopts an improved U-Net network with a convolutional attention block (CBAM) to perceive environmental information of road spills. The CBAM module added to the U-Net jump connection contains channel attention blocks and spatial attention blocks, which will perform feature map weighting at the end to highlight significant features that are useful for specific tasks.

[0062] The improved U-Net model uses RGB images with a resolution of 640×480 as input; ResNet34 pre-trained on ImageNet is used as the encoder to capture deeper feature representations; and the SGD (stochastic gradient descent) optimizer is used with a batch size of 16, a momentum of 0.95, an initial learning rate of 0.01, a weight decay of 0.0005, and training for 30 epochs; the learning rate decays by a factor of 0.96 after each epoch. Finally, the U-Net network converts the feature map into the final segmentation map and outputs the segmentation result with the same resolution as the input image. The segmentation result is the perceived environmental information of road spills. These semantically segmented image data will be used as the input of target detection, and finally identify potential spills on the road.

[0063] Step S2: Object Detection

[0064] The present invention uses the RT-DETR target detector with the neck network modified to BiFPN to detect and classify road spills, and the input is the semantic segmentation result of the U-Net network. The target detector will detect the pixel points belonging to the spilled objects in the U-Net segmentation result, and filter out the pixel points that do not belong to the spilled objects, and finally complete the detection of the spilled objects on the road; for the classification results of the spilled objects, the classification of the road spilled objects will be reflected from the semantic labels of the detected spilled objects, and anchor frames of different colors will be used to reflect different categories of spilled objects. The detection accuracy of each spilled object will be presented in a specific numerical value through the statistical mAP50-90 at the end of the detection.

[0065] In the model parameter setting, firstly, the number of BiFPN repetitions is adjusted continuously, and finally set to 5 to achieve the best model detection effect. BiFPN (Bidirectional Feature Pyramid Network) is a structure of feature fusion layer, which improves accuracy through multi-level feature pyramid and bidirectional information transmission. It realizes the effective aggregation of features of different resolutions through cross-scale connection and weighted feature fusion; secondly, the epsilon value is set to 0.0001 to avoid numerical instability; finally, the parameters use_swish and conv_bn_act_pattern are set to 1. These two parameters will make the target detector use the Swish activation function and directly connect Batch Normalization after convolution. This mode will provide better target detection performance.

Claims

1. A road spillage classification detection method incorporating environmental information perception, characterized in that: The following steps are involved: S1: Use the vehicle-mounted high-definition IPC camera to capture clear video streams; use the NVIDIA GPU's high-performance video codec to reduce the video stream frame rate, and extract frames from the reduced-frame video to reduce the amount of data; S2: Improved U-Net algorithm; in the standard U-Net structure, the network is improved by adding a convolutional block attention model (CBAM) in the skip connection, so that when the road spilled object image data is input into the U-Net for segmentation, the irrelevant areas in the input image are suppressed, and the significant features that are useful for specific tasks can be highlighted; S3: Deploy RT-DETR object detector; and introduce convolutional conditional random field (ConvCRF) into the network to optimize the segmentation results; S4: Modify the neck network of RT-DETR object detector; Replace the neck network with BiFPN (Bidirectional Feature Pyramid Network - Cross-Scale Connection and Weighted Feature Fusion), and use a multi-scale feature fusion network so that the model can consider the importance of different input features and improve model efficiency; S5: U-Net environment perception network and RT-DETR target detector are efficiently connected; based on the improved U-Net algorithm, the environmental information of the area in front of the vehicle is quickly perceived and the semantic information of potential spilled objects on the road is extracted; Based on the improved RT-DETR target detector, the spilled objects on the road can be quickly and accurately identified.

2. The method for detecting road spillage classification by integrating environmental information perception according to claim 1 is characterized in that: In the step S1: using the high-performance video codec of the NVIDIA GPU to perform frame reduction and frame extraction processing on the video stream can reduce the overall data volume and improve the video processing speed and efficiency.

3. The method for detecting road spillage classification by integrating environmental information perception according to claim 1, characterized in that: In the step S2: a plug-and-play CBAM attention module is introduced, which combines a channel attention block for calculating channel weights by adaptive average pooling and a spatial attention block for calculating the weight of each spatial position by convolution operation. This dual attention mechanism enables the model to fully capture the key information in the features and improves the performance of the model.

4. The method for detecting road spillage classification by integrating environmental information perception according to claim 1, characterized in that: In the step S3: RT-DETR is used as a road spilled object target detector. It is an end-to-end target detector based on the Transformer architecture, which can effectively overcome the speed delay and optimization difficulties brought by the non-maximum suppression (NMS) used in traditional target detectors such as the YOLO series, and can greatly enhance the timeliness and accuracy of spilled object detection by taking advantage of its end-to-end detection process.

5. The method for detecting road spillage classification by integrating environmental information perception according to claim 1, characterized in that: In the step S3: introducing a convolutional conditional random field (ConvCRF) into the RT-DETR target detector for post-processing optimization can refine and improve the segmentation result of the U-Net network and improve the boundary accuracy of the scattered objects on the road.

6. The method for detecting road spillage classification by integrating environmental information perception according to claim 1, characterized in that: In step S4: the neck network of RT-DETR is modified to BiFPN, and the target detector is enabled to better fuse multi-scale features by optimizing cross-scale connections and weighted feature fusion. Improve the expressiveness of features, use fewer parameters and computation while achieving similar accuracy, and improve model efficiency.

Images