A method and system for vehicle detection on enhanced thermal infrared images based on improved SSD

Abstract

This invention discloses a method and system for vehicle detection on enhanced thermal infrared images based on an improved SSD (Simulated Radial Sampling Deposition). It utilizes an infrared thermal imaging sensor camera to acquire and create a raw thermal infrared image dataset; performs image enhancement processing on the raw dataset to obtain an enhanced thermal infrared image dataset; constructs an enhanced vehicle detection network ERSSD based on SSD, replacing the Stem block downsampling module in the feature extraction network to reduce information loss from the input thermal infrared image; replaces the residual module in the feature extraction network with the Stage module to improve the network's feature extraction and representation capabilities; trains the ERSSD network using the enhanced thermal infrared dataset to generate a thermal infrared vehicle detection model; and performs real-time vehicle detection using the thermal infrared vehicle detection model. This invention offers high detection accuracy and fast detection speed, and the image enhancement effectively improves the quality of thermal infrared images, making it applicable to driver assistance systems and intelligent transportation systems.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning target detection, and relates to a vehicle detection method and system, in particular to an enhanced thermal infrared image vehicle detection method and system based on an improved SSD deep learning target detection network, which can be used in an auxiliary driving system and an intelligent traffic system. BACKGROUND

[0002] Deep learning target detection is widely used in intelligent traffic systems, vehicle auxiliary driving systems and other computer vision applications. With the continuous expansion of city size, the number of vehicles has increased significantly, and the population density is also increasing. Deep learning vehicle detection is becoming more and more important. At present, visible light vehicle detection performs well, but it is sensitive to light, so it is difficult to extract clear features of vehicles in night, backlight and weak light scenes. Unlike visible light, thermal infrared imaging is based on the temperature difference between objects and their environment. Due to the difference in heat distribution between vehicle types and various objects, thermal infrared imaging can be unaffected by light, has strong anti-interference ability, and is not easily affected by harsh environments. Compared with vehicle detection using visible light, thermal infrared vehicle detection is more important and has great significance for in-depth research.

[0003] Target detection technology has been widely developed in recent years. Due to the powerful automatic learning function of deep convolutional neural network (CNN) on large-scale training data sets, CNN has made remarkable achievements in target detection. For vehicle detection research methods on thermal infrared images, when vehicles drive on the road, the engine, tires, exhaust pipe and headlights will generate a lot of heat. Compared with the traffic environment, these areas of the infrared image have obvious gray scale features and texture features and other appearance features, which can be used for vehicle detection based on image processing. At present, the detection methods based on deep learning mainly have two-stage and one-stage detection methods. The two-stage detection network represented by Fast R-CNN, Faster R-CNN and Mask R-CNN generally has high detection accuracy. However, the region proposal-based algorithm is often complex and time-consuming. The one-stage detection network is represented by YOLO series and SSD. The region proposal generation process is abandoned, which significantly shortens the algorithm inference time, and the final detection result can be obtained in an end-to-end manner, so that the detection speed is significantly improved, and the detection accuracy is relatively low, but the practicability is stronger.

[0004] Deep learning autonomously learns vehicle features and trains classifiers using data-driven methods, making vehicle detection more efficient. In existing technologies, these algorithms have also been applied to vehicle detection in thermal infrared images, such as Faster R-CNN, SSD, and YOLO. Jing Gong et al. proposed a vehicle detection method based on YOLOv3-Tiny thermal infrared images, which combats darkness or strong light to better extract vehicle features from thermal images, thereby improving vehicle detection accuracy. Chin-WeiChang et al. proposed an improved convolutional network model; the proposed method can generate sophisticated thermal images, which facilitates more effective learning of the network model. Unlike the high resolution of visible light images, due to the limitations of infrared sensors and unfavorable environmental conditions, raw thermal infrared images typically have low resolution, high noise, low contrast, and lack of structure (such as shape and texture information), thus reducing the quality of infrared images and posing significant challenges to subsequent vehicle detection and backbone network feature extraction. Due to the complex image quality of thermal infrared images, simply improving the algorithm is insufficient; existing technologies are limited by low-quality thermal infrared images, thus affecting target detection accuracy. Summary of the Invention

[0005] To address the aforementioned problems in target detection, this invention provides a vehicle detection method and system based on an improved SSD deep learning target detection network for enhanced thermal infrared images. The aim is to reduce the impact of low-quality thermal infrared images on subsequent detection by combining thermal infrared image enhancement with a deep learning framework. This method uses target detection as the primary function and the enhanced thermal image as input for vehicle detection on thermal infrared images.

[0006] The method of this invention is a vehicle detection method based on improved SSD enhanced thermal infrared images, comprising the following steps:

[0007] Step 1: Construct the vehicle detection network ERSSD on thermal infrared images, which is responsible for extracting vehicle contour features, including vehicle body and tire features;

[0008] The vehicle detection network ERSSD on thermal infrared images is an improved version of the deep learning object detection network SSD (feature extraction network is Resnet50).

[0009] Specifically, the Stem block downsampling module in the feature extraction network is replaced; the Stem block downsampling module consists of a 7×7 kernel convolution with stride = 2 and a 3×3 max pooling with stride = 2; the replacement is a downsampling module consisting of a 3×3 kernel convolution with stride = 2, two 3×3 kernel convolutions with stride = 1, and a 3×3 max pooling with stride = 1.

[0010] The residual module in the feature extraction network is replaced with a Stage module. The Stage module propagates and concatenates feature information in different network gradients and consists of backbone gradients and branch gradients. The backbone gradient is composed of stacked original residual modules. The branch gradient consists of 3×3 max pooling with stride = 2 and 1×1 kernel convolution with stride = 1. The feature dimensions output by the backbone gradient and branch gradient are consistent and are concatenated using concat.

[0011] Step 2: Train the thermal infrared vehicle detection network ERSSD obtained in Step 1 and configure the thermal infrared vehicle detection model weight file;

[0012] Step 3: The real-time input thermal infrared video image is first processed by image enhancement, and then input into the thermal infrared vehicle detection network ERSSD. The thermal infrared vehicle detection model weight file in Step 2 is used as the weight of the vehicle detection network to perform real-time vehicle detection on the thermal infrared video data.

[0013] The technical solution adopted by the system of the present invention is: a vehicle detection system based on improved SSD enhanced thermal infrared images, comprising the following modules:

[0014] Module 1 is used to construct the vehicle detection network ERSSD on thermal infrared images, which is responsible for extracting vehicle contour features, including vehicle body and tire features;

[0015] The vehicle detection network ERSSD on thermal infrared images is an improved version of the deep learning object detection network SSD (feature extraction network is Resnet50).

[0016] Specifically, the Stem block downsampling module in the feature extraction network is replaced; the Stem block downsampling module consists of a 7×7 kernel convolution with stride = 2 and a 3×3 max pooling with stride = 2; the replacement is a downsampling module consisting of a 3×3 kernel convolution with stride = 2, two 3×3 kernel convolutions with stride = 1, and a 3×3 max pooling with stride = 1.

[0017] The residual module in the feature extraction network is replaced with a Stage module. The Stage module propagates and concatenates feature information in different network gradients and consists of backbone gradients and branch gradients. The backbone gradient is composed of stacked original residual modules. The branch gradient consists of 3×3 max pooling with stride = 2 and 1×1 kernel convolution with stride = 1. The feature dimensions output by the backbone gradient and branch gradient are consistent and are concatenated using concat.

[0018] Module 2 is used to train the thermal infrared vehicle detection network ERSSD obtained from Module 1 and configure the thermal infrared vehicle detection model weight file.

[0019] Module 3 is used to first perform image enhancement processing on the real-time input thermal infrared video image, and then input it into the thermal infrared vehicle detection network ERSSD. The thermal infrared vehicle detection model weight file described in Module 2 is used as the weight of the vehicle detection network to perform real-time vehicle detection on the thermal infrared video data.

[0020] The beneficial effects of this invention are as follows:

[0021] (1) This invention combines thermal infrared image enhancement function with deep learning framework and applies it to vehicle detection in scenarios where visible light cannot cope with backlight, no light at night, complex background and harsh environment and climate. It can achieve real-time, accurate and efficient vehicle detection in scenarios such as driver assistance system and intelligent transportation system.

[0022] (2) The present invention constructs a guided filtering image enhancement based on grayscale histogram. The thermal infrared image is used as the guided image, and the thermal infrared image after adaptive segmented stretching transformation is used as the input image. The output image of the guided filter is the enhanced thermal infrared image. The image enhancement adaptively enhances the contrast between the foreground and background of the thermal image, enhances the detail texture information of non-self-heating objects and vehicles in the thermal infrared image, so as to make up for their weakness in thermal image details, and effectively suppresses noise.

[0023] (3) The present invention is adapted to target detection of low-resolution thermal infrared images. In the feature extraction network, the original Stem block is replaced to reduce the loss of information in the input infrared image. The Stage module is used to replace the residual module in the feature extraction network, so that the feature information is propagated and spliced ​​in different network gradients, which enriches the feature map information and improves the network feature extraction and feature expression capabilities. Attached Figure Description

[0024] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the improved feature extraction network structure in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the ERSSD network structure for vehicle detection on thermal infrared images in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the image enhancement process based on grayscale histogram in an embodiment of the present invention;

[0028] Figure 5 This is a comparison image of the thermal infrared image before and after enhancement in an embodiment of the present invention;

[0029] Figure 6 This is a PR curve diagram of the detection accuracy test results in an embodiment of the present invention;

[0030] Figure 7 This is a diagram showing the vehicle detection effect on a thermal infrared image in an embodiment of the present invention. Detailed Implementation

[0031] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0032] Please see Figure 1 The present invention provides a vehicle detection method based on improved SSD in enhanced thermal infrared images, comprising the following steps:

[0033] Step 1: Construct the vehicle detection network ERSSD on thermal infrared images;

[0034] This embodiment is based on the SSD network model framework in deep learning. ResNet50 is used as the feature extraction network to construct the thermal infrared vehicle detection network ERSSD. The vehicle feature extraction part in the feature extraction network is improved to obtain an improved deep learning vehicle detection model (thermal infrared vehicle detection network ERSSD). The improved thermal infrared vehicle detection network ERSSD feature extraction algorithm is responsible for extracting the vehicle contour features, including vehicle body and tire features.

[0035] Please see Figure 2 The replaced Stem block downsampling module replaces the Stem block downsampling module in the feature extraction network (ResNet50). The Stem block downsampling module consists of a 7×7 kernel convolution with stride = 2 and a 3×3 max pooling with stride = 2. The replacement module consists of a downsampling module consisting of a 3×3 kernel convolution with stride = 2, two 3×3 kernel convolutions with stride = 1, and a 3×3 max pooling with stride = 1.

[0036] Please see Figure 2 The residual module in the feature extraction network is replaced with a Stage module. The Stage module allows feature information to propagate and be concatenated in different network gradients, and consists of backbone gradients and branch gradients. The backbone gradient is composed of stacked original residual modules. The branch gradient consists of 3×3 max pooling with stride = 2 and 1×1 kernel convolution with stride = 1. The feature dimensions output by the backbone gradient and the branch gradient are consistent and are concatenated by concat.

[0037] Please see Figure 3This is a schematic diagram of the Thermal Infrared Vehicle Detection Network (ERSSD) structure in this embodiment, consisting of a feature extraction network and additional feature layers. The improved deep learning vehicle detection model selects feature maps of different resolutions for multi-scale detection, consisting of multi-scale feature maps of Stage2_4, Stage3_6, Conv4_2, Conv5_2, Conv6_2, and Conv7_2. The feature maps of the shallower network, with larger sizes, contain more details, so they are responsible for detecting small targets. The feature maps extracted by the deep network are mainly used to detect large targets, as they contain less information due to their smaller size. The sizes of the six feature maps are 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1, respectively. After inputting a single image, the improved deep learning vehicle detection model distributes default boxes of different scales and proportions on the multi-scale feature maps. Each default box has positional offset and target class probability information. Then, non-maximum suppression (NMS) is used to remove redundant bounding boxes and generate prediction results.

[0038] Step 2: Train the thermal infrared vehicle detection network ERSSD obtained in Step 1 and configure the thermal infrared vehicle detection model weight file;

[0039] Specifically, it includes the following sub-steps:

[0040] Step 2.1: In this embodiment, a FLIR DUO Pro R infrared thermal imaging and visible light dual-sensor camera is used to acquire thermal infrared video data under different viewpoints and scenes. The thermal infrared video data is decoded into thermal infrared images, and a raw thermal infrared image dataset is created. Then, LabelImg software is used to annotate the vehicle target location information, vehicle size information, and vehicle category information in the thermal infrared image data and store them in an XML tag file. The vehicle category information is based on the vehicle outline information and size information. After the annotation is completed, the tag dataset corresponding to the raw thermal infrared image dataset is obtained.

[0041] This invention utilizes a FLIR DUO Pro R dual-sensor camera with infrared thermal imaging and visible light capabilities. Its basic parameters are: uncooled vanadium oxide (VOx) focal plane array detector, resolution 640×512, pixel pitch 17μm, wavelength range 7.5~13.5μm, and output MOV format video. Infrared light can capture the heat emitted by an object itself and can be used for vehicle recognition in various scenarios.

[0042] Step 2.2: Perform image enhancement processing on the original thermal infrared image dataset to obtain an enhanced thermal infrared image dataset; the enhanced thermal infrared image dataset is generated from the thermal infrared image dataset through image enhancement, and the two datasets maintain the same label dataset;

[0043] In this embodiment, an image enhancement method is used to construct an adaptive piecewise stretching guided filtering process. The guided filtering process image enhancement includes a guided image, an input image, and an output image. The thermal infrared image is used as the guided image, and the image of the thermal infrared image after adaptive piecewise stretching transformation is used as the input image. The output image of the guided filtering is the enhanced thermal infrared image.

[0044] In this embodiment, the adaptive piecewise stretching transformation transforms each gray value of the thermal infrared image data according to the function shown in Equation (1);

[0045] (1)

[0046] in, F i This represents the grayscale value of the input image pixels. F r I represents the grayscale value of the output image pixels after stretching. min It is the minimum value among the minimum grayscale values ​​with a proportion of n, I max It is the minimum value among the largest grayscale values ​​with a proportion of n, where n is the compression factor;

[0047] In this embodiment, the guided filter performs filtering on the input image using the guided image. The filtered output at pixel i is represented as a weighted average, as shown in formula (2); the guided filter kernel W ij (G), as shown in formula (3);

[0048] (2)

[0049] Where i and j represent the pixel indices of the image, G is the guide image, p is the input image, q is the output image, and w ij (G) is the guided filter kernel, which is a function that guides the image I and is independent of p.

[0050] (3)

[0051] Where the parameter ω k Let r represent the k-th kernel function window, with radius r, |ω| being the number of pixels in the window, and μ being the number of pixels in the window. k and σ k 2 ε represents the mean and variance of the guided plot within the window, and ε is the smoothing factor.

[0052] In this embodiment, the guiding filter parameter ε = 0.1 is selected. 2 r=2.

[0053] Please see Figure 4As shown in the diagram, this embodiment illustrates the image enhancement process: (a) thermal infrared image; (b) input image I; (c) the image after adaptive segmented stretching is the input image p; (d) guiding image G; (e) the filtered output image is the enhanced image q. First, the original thermal infrared image is used as both the input and guiding images and input to the guiding filter. Second, in this embodiment, the input image I is stretched in grayscale values, resulting in the stretched input image as shown... Figure 4 Image p is shown in (c). The purpose of the grayscale stretching process of the input image I is to enhance the contrast between the target and the background, stretch the grayscale with a high pixel ratio, compress the grayscale with a low pixel ratio, and transform the image stretching segment by segment. The pixels of the input image I are mapped to the interval [0, 255]. In this embodiment, the adaptive segmented stretching formula is used as shown in (1), and the grayscale compression factor is set to n=5%. The image p after adaptive segmented stretching is used as input, and the guiding image G is a thermal infrared image. The detailed structural information of the thermal infrared image will be merged into the output image. The purpose of the guided filtering is to retain and enhance the detailed information while eliminating noise and clutter.

[0054] Step 2.3: Allocate the enhanced thermal infrared image dataset into training and testing datasets according to a predetermined ratio; input the training dataset into the thermal infrared vehicle detection network ERSSD for training; during training, generate and save a target detection weight file for each iteration until the network reaches convergence and training stops; use the thermal infrared vehicle detection weight file generated during the training of the thermal infrared vehicle detection network ERSSD to detect vehicles in all enhanced thermal infrared images in the testing dataset, detect and save all vehicle information in the testing dataset, compare the detection results with the real vehicle information in the corresponding label dataset to calculate the detection accuracy, and select the weight file with the highest test accuracy as the weight file of the thermal infrared vehicle detection model.

[0055] Step 3: The real-time input thermal infrared video image is first image-enhanced, and then input into the thermal infrared vehicle detection network ERSSD. The thermal infrared vehicle detection model weight file is used as the weight of the vehicle detection network to perform real-time vehicle detection on the thermal infrared video data.

[0056] In this embodiment, the detection accuracy is calculated by comparing the detection results with the real vehicle information in the corresponding tag dataset, including precision P, recall R, mean average accuracy mAP and detection frames per second FPS.

[0057] (4)

[0058] (5)

[0059] (6)

[0060] Where TP represents the number of vehicles correctly detected on all thermal infrared images in the test dataset, FP represents the number of vehicles falsely detected on all thermal infrared images in the test dataset, and FN represents the number of vehicles not detected on all thermal infrared images in the test dataset; in formula (6), P represents precision and R represents recall.

[0061] In this embodiment, the Thermal Infrared Vehicle Detection Network (ERSSD) uses a target detection weight file as the weight for vehicle detection. It classifies and regresses vehicles in thermal infrared video data, and marks the classified and regressed vehicle information in the video data as bounding boxes. The final vehicle detection result is obtained by outputting the detected video data.

[0062] The effects of this embodiment will be further illustrated below through specific implementation.

[0063] The software environment used in this embodiment is: Buntu 18.04 operating system, GPU acceleration software CUDA 11.1 and CUDNN 8.2, PyTorch deep learning framework, and Python programming language; the hardware environment is: Intel Core i9-10900K processor and NVIDIA GeForce RTX 3070 graphics card.

[0064] This embodiment uses image entropy (H) e The enhanced thermal infrared image was quantitatively evaluated using the structural similarity index (SSIM) and formulas (7 and (8), respectively.

[0065] (7)

[0066] (8)

[0067] in L It is the total number of gray levels in the image. p i Indicates the first i The frequency of occurrence of each gray level; SSIM X,F Represents the source image X Compared with the enhanced image F Structural similarity between them x and f These represent the source image and the enhanced image patch in the sliding window, respectively. This represents the covariance between the original image and the enhanced image. Let these represent the standard deviations of the original image and the enhanced image, respectively; and C1, C2, and C3 represent the average values ​​of the original image and the enhanced image, respectively, and are constants.

[0068] See Figure 5 As shown in Table 1, the image enhancement before and after enhancement is compared, where (a) is the original thermal infrared image and (b) is the enhanced thermal infrared image. The processing performance of this image enhancement was analyzed experimentally and quantitatively evaluated. First, this embodiment uses one-dimensional image entropy to compare the details of the original thermal infrared image (a) and the enhanced thermal infrared image (b). The structural similarity index (SSIM), a full-reference image quality evaluation index, is used to measure image similarity, combining brightness, contrast, and structure. The structural similarity before and after data enhancement is compared. The one-dimensional entropy and structural similarity of the original thermal image dataset and the enhanced thermal image dataset are calculated. As shown in Table 1, the enhanced thermal infrared image produces a higher entropy value and more image information, and the structural similarity approaches 1. Even after noise removal through guided filtering, the structure remains highly consistent. The constructed image enhancement algorithm effectively improves the quality of thermal infrared images.

[0069] Table 1 Enhancement Evaluation

[0070]

[0071] In this embodiment, a dataset was constructed before and after thermal image enhancement for training and testing. Two datasets were used for comparative experiments on the Thermal Infrared Vehicle Detection Network (ERSSD): the original thermal infrared dataset and the enhanced thermal infrared dataset. Table 2 shows a comparison of vehicle target detection performance parameters before and after thermal image enhancement. The results indicate that the ERSSD network more easily detects the enhanced thermal infrared images, improving the evaluation parameters by 2.59% compared to the original thermal infrared dataset. These results demonstrate that the thermally enhanced images constructed in this invention are more easily recognized by the Deep Thermal Infrared Vehicle Detection Network (ERSSD), providing reliable information for vehicle detection on thermal infrared images.

[0072] Table 2 shows a comparison of vehicle target detection performance parameters before and after thermal imaging enhancement.

[0073]

[0074] This embodiment utilizes an image enhancement method based on an adaptive piecewise stretching guided filtering process. The thermal infrared image is used as the guiding image, and the image of the thermal infrared image after adaptive piecewise stretching transformation is used as the input image. The output image of the guided filtering is the enhanced thermal infrared image. The contrast between the foreground and background of the thermal image is adaptively enhanced, which enhances the detail and texture information of non-self-heating objects and vehicles in the thermal infrared image, thereby compensating for their weakness in thermal image details and effectively suppressing noise.

[0075] In this embodiment, the enhanced thermal infrared dataset created in step 2.2 is divided into a training dataset and a test dataset in an 8:2 ratio. Based on the improved deep learning object detection network SSD, an improved thermal infrared vehicle detection network ERSSD is obtained. The improved thermal infrared vehicle detection network ERSSD is trained using the training dataset. During training, the initial learning rate is set to 0.001, the momentum is set to 0.9, and the image input pixel size is 300×300. Mini-batch training is performed with 8 images as a batch until all images in the training dataset have been trained once to complete one round of iteration. A total of 300 rounds of iteration are performed on the thermal infrared vehicle detection network ERSSD. After each round of iteration, an object detection weight file is generated and saved. The object detection weight file is used for testing the improved deep learning vehicle detection model.

[0076] This embodiment aims to verify the reliability and robustness of the ERSSD thermal infrared vehicle detection network in detecting vehicles in thermal infrared images. It compares the network with SSD and Faster R-CNN through quantitative and qualitative analysis. Using the same experimental parameters and an enhanced thermal infrared image dataset, the ERSSD network was compared with Faster R-CNN and SSD models using three metrics: accuracy, recall, and mean accuracy. The quantitative analysis results are shown in Table 3. Faster R-CNN, being a two-stage detection network, achieves a high detection accuracy of 91.49%, but its detection speed is relatively slow, with only 17 FPS, resulting in poor real-time performance. The SSD network achieves a detection accuracy of 83.62% and a faster detection speed than Faster R-CNN. The ERSSD network built in this embodiment achieves an accuracy of 91.71% and a detection speed of 46 FPS, maintaining accuracy while significantly improving speed compared to the original SSD and Faster R-CNN.

[0077] Table 3 Comparison of detection parameters for different methods

[0078]

[0079] See Figure 6Precision-recall (PR) curves for ERSSD, SSD, and Faster R-CNN models were plotted to qualitatively analyze the three algorithms. The PR curves, using recall and precision as the x and y axes respectively, are commonly used curves to measure the performance of detection algorithms. High precision corresponds to low recall. High precision indicates a low probability of false positives. High recall indicates a low probability of false negatives. The improved PR curve and the area enclosed by the two axes are larger than the other two models, reflecting the significant advantage of the ERSSD thermal infrared vehicle detection network in average detection accuracy. The constructed ERSSD thermal infrared vehicle detection network replaces Stemblock and builds a Stage module, allowing gradient information to propagate and fuse in different network paths, improving the network's feature extraction and representation capabilities. In terms of overall detection speed, ERSSD is faster than SSD.

[0080] See Figure 7 The improved thermal infrared vehicle detection model is shown in the image. The detection results show that the detection accuracy is high and there are no false detections or missed detections.

[0081] This embodiment embeds an improved deep learning vehicle detection model (Thermal Infrared Vehicle Detection Network ERSSD) into an embedded development system such as NVIDIA or Raspberry Pi. In the embedded development system, the Thermal Infrared Vehicle Detection Model ERSSD uses a model weight file to perform real-time vehicle detection on thermal infrared video data acquired by the device. During the detection process, the real-time acquired thermal infrared video is decoded into video frame images, and then image enhancement based on adaptive segmented stretching guided filtering is performed before being input into the thermal infrared vehicle detection model. The detection model uses the thermal infrared vehicle weight file from step 2.3 as the weights for vehicle detection, classifies and regresses vehicles in the thermal infrared video data, and annotates the classified and regressed vehicle information in the video data as bounding boxes. The final vehicle detection result is obtained by outputting the detected video data.

[0082] This invention offers high detection accuracy and fast detection speed, and its image enhancement method effectively improves the quality of thermal infrared images.

[0083] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0084] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for vehicle detection on enhanced thermal infrared images based on improved SSD, characterized in that, Comprising the following steps: Step 1: build a vehicle detection network ERSSD on a thermal infrared image, responsible for extracting vehicle contour features including vehicle body and tire features; The vehicle detection network ERSSD on the thermal infrared image is an improved version of the deep learning target detection network SSD, which is composed of a feature extraction network and an additional feature layer; Wherein, the Stem block downsampling module in the feature extraction network part is replaced; the Stem block downsampling module is composed of a 7x7 kernel convolution, stride = 2 and a 3x3 max pooling, stride = 2; the replacement is replaced by a 3x3 kernel convolution, stride = 2, two 3x3 kernel convolutions, stride = 1 and a 3x3 max pooling, stride = 1; The Stage module is used to replace the residual module in the feature extraction network part; the Stage makes the feature information propagate in different network gradients and is spliced, which is composed of a main gradient and a branch gradient; the main gradient is stacked by the original residual module; the branch gradient is composed of 3x3 max pooling, stride = 2 and 1x1 kernel convolution, stride = 1; the feature dimensions of the main gradient and the branch gradient output are consistent in length, width and height, which are spliced by concat; Step 2: training the thermal infrared vehicle detection network ERSSD obtained in step 1, and configuring the thermal infrared vehicle detection model weight file; Step 3: inputting the real-time input thermal infrared video image to the thermal infrared vehicle detection network ERSSD after image enhancement processing, and using the thermal infrared vehicle detection model weight file in step 2 as the vehicle detection network weight to perform real-time vehicle detection on the thermal infrared video data.

2. The improved SSD based enhanced vehicular detection on thermal infrared images method according to claim 1, wherein, The specific implementation of step 2 includes the following substeps: Step 2.1: using an infrared thermal imaging and visible light dual sensor camera to collect thermal infrared video data at different angles and scenes, decoding the thermal infrared video data into thermal infrared images, and making original thermal infrared image data sets, and storing the position information, vehicle size information and vehicle category information of the vehicle target in the thermal infrared image data to a label file, wherein the vehicle category information is classified according to the vehicle contour information and size information; after labeling, the original thermal infrared image data set corresponding to the label data set is obtained; Step 2.2: image enhancement processing is performed on the original thermal infrared image data set to obtain an enhanced thermal infrared image data set; the enhanced thermal infrared image data set is generated by image enhancement of the thermal infrared image data set, and the two data sets maintain the same label data set; Step 2.3: the enhanced thermal infrared image data set is allocated into a training data set and a test data set according to a predetermined proportion; The training data set is input into the thermal infrared vehicle detection network ERSSD for ERSSD model training; During the training process, a target detection weight file is generated and saved every time a round of iteration is performed until the network reaches a state of convergence and stops training; the weight file of the thermal infrared vehicle detection model is configured.

3. The improved SSD based enhanced vehicular detection on thermal infrared images method according to claim 1, characterized in that: The image enhancement processing in step 3 is a guided filtering process based on adaptive segmented stretch transformation, and the guided filtering process image enhancement includes a guide image, an input image and an output image, wherein the thermal infrared image is taken as the guide image, and the image after the adaptive segmented stretch transformation of the thermal infrared image is taken as the input image; The output image of the guided filtering is an enhanced thermal infrared image.

4. The improved SSD based enhanced vehicle detection on thermal infrared images method according to claim 3, characterized in that: The adaptive segmented stretch transformation is a transformation of each gray value of the thermal infrared image data according to the function shown in formula (1); (1) wherein, F i represents the input image pixel gray value, F r represents the output image pixel gray value after stretching, I min is the minimum value in the minimum gray value of the proportion n, I max is the minimum value in the maximum gray value of the proportion n, n is the compression factor; The guided filtering filters the input image through a guided image, and a filtering output at a pixel i is represented as a weighted average value, as shown in equation (2). (2) wherein, i and j respectively denote indices of image pixels, G is a guide image, p is an input image, q is an output image, w ij (G) is a guide filter kernel, is a function of the guide image I, and is independent of p ​ Guiding filter kernel W ij (G), as shown in equation (3). (3) where the parameter ω k represents the k th kernel window, the radius of the window is r , |ω| is the number of pixels in the window, μ k and σ k 2 are the mean and variance of the guidance map in the window, and ε is a smoothing factor.

5. The improved SSD based enhanced vehicular detection on thermal infrared images method according to claim 3, wherein: In step 2.3, the thermal infrared vehicle detection weight is obtained by using the vehicle detection weight file generated during the training process of the thermal infrared vehicle detection network ERSSD to perform vehicle detection on all enhanced thermal infrared images in the enhanced thermal infrared image test data set, detecting all vehicle information in the test data set and saving them, comparing the detection results with the real vehicle information in the corresponding label data set to calculate the detection accuracy, and selecting the vehicle detection weight file with the highest test accuracy as the weight file of the thermal infrared vehicle detection network ERSSD.

6. The improved SSD based enhanced vehicle detection on thermal infrared images method according to claim 3, characterized in that: The detection accuracy includes precision P, recall R, accuracy AP and frames per second FPS, which are calculated according to the detection results and the real vehicle information in the corresponding label data set. (4) (5) (6) In formula (6), P represents the precision, and R represents the recall.

7. The improved SSD based enhanced vehicle detection on thermal infrared images method according to any one of claims 1-6, characterized in that: In step 3, the thermal infrared vehicle detection network ERSSD uses the target detection weight file as the weight for vehicle detection, classifies and regresses the vehicles in the thermal infrared video data, and labels the classified and regressed vehicle information in the form of a label frame in the video data, and outputs the detected video data to obtain the final vehicle detection result.

8. A system for vehicle detection on enhanced thermal infrared images based on improved SSD, characterized in that, The system includes the following modules: Module 1 is used to build a thermal infrared image vehicle detection network ERSSD, which is responsible for extracting vehicle contour features including vehicle body and tire features. The thermal infrared image vehicle detection network ERSSD is an improved version of the deep learning target detection network SSD, which is composed of a feature extraction network and an additional feature layer. The Stem block downsampling module in the feature extraction network part is replaced; the Stem block downsampling module is composed of a 7x7 kernel convolution, stride = 2 and a 3x3 maximum pooling, stride = 2; the replacement is replaced by a 3x3 kernel convolution, stride = 2, two 3x3 kernel convolutions, stride = 1 and a 3x3 maximum pooling, stride = 1. The residual module in the feature extraction network part is replaced by a Stage module; the Stage makes the feature information propagate and splice in different network gradients and is composed of a main gradient and a branch gradient; the main gradient is stacked by original residual modules; the branch gradient is composed of 3x3 max pooling, stride = 2 and 1x1 kernel convolution, stride = 1; the feature dimensions output by the main gradient and the branch gradient are consistent in length, width and height, and are spliced by concat; Module 2 is configured to train the thermal infrared vehicle detection network ERSSD obtained by module 1 and configure a thermal infrared vehicle detection model weight file; Module 3 is configured to input the real-time input thermal infrared video image to the thermal infrared vehicle detection network ERSSD after image enhancement processing, and use the thermal infrared vehicle detection model weight file in module 2 as the vehicle detection network weight to perform real-time vehicle detection on the thermal infrared video data.

Images