All-weather road obstacle detection and warning method based on visual fusion perception

Abstract

The application discloses a visual fusion perception-based all-weather road obstacle detection and early warning method, belongs to the technical field of automobile safe driving, and is characterized in that the method is based on a visual perception method, utilizes image fusion technology of a vehicle-mounted camera and an infrared camera, cooperates with a target detection algorithm based on deep learning, and further accurately identifies obstacles through calibration technology of the obstacles. The visual fusion technology of infrared images and visible light images and the target identification and detection technology are adopted, the limitation and uncertainty of single sensor identification are overcome, different characteristics of various sensors are fused, the detection picture quality is improved to reduce detection error, the advantages of infrared images in extracting edge contour information according to object radiation difference and visible light images in clearly highlighting texture information are comprehensively utilized, and road obstacles are highlighted, so that the detection precision is improved.

Description

Technical Field

[0001] This invention belongs to the field of automotive safe driving technology, and in particular relates to a method for detecting and warning of obstacles in front of a vehicle by integrating infrared imaging technology and traditional visual imaging technology. Background Technology

[0002] In recent years, with the continuous development of deep learning and visual perception technologies, many research projects on vision-based automotive safety driving systems have emerged in the field of safe driving. Automotive safety is a perpetual topic in the automotive industry; without safety, various aspects of driving comfort, power, and intelligent extensions such as entertainment and communication are meaningless. Ensuring a safe driving environment is one aspect of driving safety, including ensuring the vehicle can identify and avoid obstacles ahead. Obstacles ahead include potholes and roadblocks. For various reasons, drivers may fail to recognize them in time, leading to accidents. Potholes are particularly problematic at night, as the color difference between potholes and the road surface is smaller, and lighting is poor. Drivers' vision is 50% lower than before sunset, and they may mistakenly believe the pothole is small and poses no threat, failing to take appropriate detour measures and causing accidents. According to traffic department statistics, although highway traffic volume at night is only one-third of that during the day, the number of accidents accounts for 60% of the total. The main cause of nighttime traffic accidents is the inability to clearly see obstacles and make timely emergency reactions such as braking or swerving, sometimes even resulting in serious mistakes like pressing the accelerator instead of the brake. Furthermore, potholes and standing water in rainy weather can mislead drivers, creating various safety hazards. Therefore, research into how vehicles can identify and detect obstacles ahead and promptly warn drivers is crucial for ensuring driving safety.

[0003] Currently, there is research on vision-based real-time detection and warning of road obstacles in the field of advanced driver assistance systems. However, advanced driver assistance systems are expensive and are generally only available in mid- to high-end vehicles. In order to make this technology more accessible, it is urgent to explore a new implementation method in the existing technology, taking into account factors such as cost, real-time performance, and recognition accuracy, in order to achieve all-weather detection of road obstacles. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide an all-weather road obstacle detection and early warning method based on visual fusion perception. By using visual perception, image fusion technology of vehicle-mounted camera and infrared camera, combined with target detection algorithm based on deep learning, and obstacle calibration technology, obstacles can be identified more accurately, thereby ensuring driving safety to a greater extent.

[0005] The all-weather road obstacle detection and early warning method based on visual fusion perception employs an all-weather road obstacle detection and early warning system based on visual fusion perception, including an infrared visual sensor, a CMOS visual sensor, an obstacle detection module, independent buttons, a power module, a main control chip, a GPS module, an LED warning light display module, an OLED display module, and a buzzer. The method includes the following steps, performed sequentially.

[0006] Step 1: Install an onboard camera and an infrared camera on the vehicle, and input the road image information captured by the two cameras into the microprocessor for image preprocessing;

[0007] Step 2: Perform a fusion process on the images obtained in Step 1.

[0008] ①Image calibration

[0009] Visible light image IV n×n and infrared images IR n×n Perform corresponding point pairing; identical pixels in two images satisfy IV ij =ir i′j′ Furthermore, the pixels within a certain neighborhood of the corresponding point have similarity. Based on the visible light image, the infrared image is transformed for calibration.

[0010] ②Image fusion

[0011] Image fusion is performed using the unsupervised framework VIF-NET, which combines infrared and visible light. The feature map extracted from the visible light image and the contour image extracted from the infrared image are fed into generator 1 to generate the fused image. The discriminator results are used to construct a loss function and theoretically determined obstacle position images. Generator 2 generates a reference image for training generator 1. The loss function is used to constrain generator 1 so that the model reaches the Nash equilibrium point. The connections and biases between neurons in the hidden layer reach their optimal values, i.e., the weights w and biases b: the parameters that measure the connection between two neurons, reach their optimal values. Training ends, and image fusion is completed.

[0012] Step 3: Input the fused image obtained in Step 2 into the YOLOv3SPP object detection network for detection. Use a standardized training set to combine global and local features to obtain the output feature map.

[0013] Step 4: Based on the output feature map from Step 3, the location information of road obstacles is obtained in real time through a target detection algorithm, the distance to the obstacles is calculated, and a warning is issued based on the set obstacle safety distance threshold.

[0014] The discriminator loss function construction in step two employs an improved SSIM algorithm, which, combined with the road obstacle detection principle of not recognizing obstacles but only detecting the size of the obstacle's outer contour and its distance from the vehicle, yields the SSIM result.

[0015]

[0016] In the formula, μ x and μ y It is the average value of the pixels, σ x and σ y It is the standard deviation of pixels, σ xy C1 and C2 are the covariance of the pixels, and C1 and C2 are the structure constants, taken as 9 × 10⁻⁶. -4 .

[0017] The image fusion processing in step two requires consideration of fusion evaluation metrics including mutual information (MI), phase congruency (PC), and nonlinear correlation information entropy (Q). NCIE And the universal image quality index (UIQI).

[0018] The third step, the YOLOv3SPP object detection network, includes three convolutional layers and one SPP module. An input image undergoes a four-dimensional upsampling process to obtain four output feature maps. The four images are then randomly cropped and stitched together onto a single image for training and fusion.

[0019] The obstacle safety distance threshold in step four is determined by the vehicle braking distance.

[0020]

[0021] In the formula, s1, s2, s3, and s4 represent the driver's reaction time travel distance, brake application distance, continuous braking distance, and brake release distance, respectively; τ1, τ2, τ3, and τ4 represent the duration (s) of the four stages; u a0 Let a be the speed before braking. bmax α is the maximum braking deceleration, β is the reaction time correction factor, γ is the deceleration correction factor, and γ is the initial velocity correction factor.

[0022] Through the above design scheme, the present invention can bring the following beneficial effects: the all-weather road obstacle detection and early warning method based on visual fusion perception adopts the visual fusion technology of infrared image and visible light image and target recognition and detection technology, which overcomes the limitations and uncertainties of single sensor recognition, integrates the different characteristics of various sensors, highlights the area of ​​interest, and thus forms a more comprehensive perception of the environment, improving the perception ability of the entire system to the outside world.

[0023] Furthermore, this invention employs the YOLOv3 SPP architecture to address image processing issues in the Batch Normalization (BN) layer of the network. It randomly crops and stitches four images onto a single image for training, which improves detection speed to some extent, while also enriching the image background and increasing the batch size. In calculating the loss function, DIOU replaces the traditional IOU, significantly improving training convergence speed and detection accuracy. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0025] Figure 1 This is a flowchart of the all-weather road obstacle detection and early warning method based on visual fusion perception of the present invention.

[0026] Figure 2 This is a framework diagram of the VIF-NET framework for the all-weather road obstacle detection and early warning method based on visual fusion perception of the present invention.

[0027] Figure 3 This is a block diagram of visible light feature extraction for the all-weather road obstacle detection and early warning method based on visual fusion perception of the present invention.

[0028] Figure 4 The diagram shows the YOLOv3SPP target detection network framework for the all-weather road obstacle detection and early warning method based on visual fusion perception, as presented in this invention.

[0029] Figure 5 This is a flowchart illustrating the specific implementation process of the all-weather road obstacle detection and early warning method based on visual fusion perception of the present invention.

[0030] Figure 6 This is a schematic diagram of the dataset directory structure for a specific implementation of the all-weather road obstacle detection and early warning method based on visual fusion perception of the present invention.

[0031] Figure 7 This is a structural block diagram of the all-weather road obstacle detection and early warning system based on visual fusion perception of the present invention. Detailed Implementation

[0032] All-weather road obstacle detection and early warning method based on visual fusion perception, such as Figure 1 As shown, it includes the following steps:

[0033] 1. Use the vehicle-mounted camera and an infrared camera installed on the vehicle to collect image information of the road ahead. The images are then fed into a microprocessor for preliminary processing.

[0034] 2. The input visible light and infrared images are fused together. The specific processing measures are as follows:

[0035] ①Image calibration

[0036] For the input visible light image IV n×n and infrared images IR n×n Perform corresponding point pairing: search for identical pixels in two images, i.e., satisfy the following conditions:

[0037] iv ij =ir i′j′ Furthermore, the corresponding pixels within a certain neighborhood have a certain degree of similarity. Based on the visible light image, the infrared image is transformed accordingly to achieve calibration.

[0038] ②Image fusion

[0039] This invention employs the unsupervised framework VIF-NET, which fuses infrared and visible light, for image fusion. The specific fusion concept is as follows: Figure 2 As shown.

[0040] The input infrared and visible light images are used to extract features, the extracted features are fused, and finally the features are reconstructed to obtain the fused image.

[0041] This invention makes corresponding improvements to VIF-NET to better suit the application scenarios of this invention. The specific improvements are as follows:

[0042] 1) Improved SSIM Algorithm: SSIM is an algorithm for measuring image structural similarity, combining brightness, contrast, and structure to measure image quality. The original SSIM formula was:

[0043] SSIM(x, y) = [l(x, y)] α ×[c(x, y)] β ×[s(x, y)] γ

[0044] In the formula, l(x, y) is the luminance matrix, c(x, y) is the contrast matrix, and s(x, y) is the structure matrix. The specific calculation formula is shown below:

[0045]

[0046]

[0047]

[0048] Where μ x and μ y It is the average value of the pixels, σ x and σ y It is the standard deviation of pixels, σ xy For pixel covariance, the VIF-NET method considers luminance unimportant and discards it. This is because, when there are potholes with water inside, the reflection from the water at night can obscure the presence of the pothole, creating a hazard. Therefore, the luminance component is not discarded. Secondly, since this invention assumes that for obstacle detection, we don't need to know what the obstacle is, only its size and distance, the structure of the obstacle is less important. Therefore, we discard the structural component, generally α = β = γ = 1. The resulting SSIM is as follows:

[0049]

[0050] 2) Selection of Fusion Evaluation Metrics: This invention selects appropriate fusion evaluation metrics for VIF-NET based on the application scenario, including mutual information (MI), phase congruency (PC), and nonlinear correlation information entropy (Q). NCIE And the Universal Image Quality Index (UIQI). This excludes the edge-preservation metric.

[0051] The specific indicators are calculated according to the following formula:

[0052]

[0053] Where p(i A i F ): i A and i F The joint probability distribution, P(i) A ): i A The marginal probability distribution.

[0054] PC = (P p ) α (P M ) β (P m ) γ

[0055] Where p, M|, m: phase, maximum torque, and minimum torque, and α = β = γ = 1

[0056]

[0057] λ i eigenvalues ​​of the nonlinear correlation matrix

[0058]

[0059] Where μ and σ are the mean and standard deviation, respectively. For I A I P The interrelationship between them.

[0060] 3. Input the fused image into the YOLOv3SPP object detection network for detection.

[0061] Object detection algorithms are classified into two categories based on their processing methods for the input image: one-stage and two-stage. The network architecture of YOLOv3 is roughly as follows: Figure 3 As shown.

[0062] The principle is as follows: Given an input image X (taking a size of 416×416 as an example), it is input into the Darknet-53 network. This network consists of convolution, Batch Normalization (BN), and Leaky ReLU activation functions. It obtains three feature maps of different sizes through a residual network. These three feature maps are one-eighth, one-sixteenth, and one-thirty-second of the source image, respectively, thus extracting features of targets of different sizes from the source image. This process is the feature extraction process. To synthesize the characteristics of the three feature layers and comprehensively reflect the target information of the original image, thereby obtaining stronger feature representation capabilities, the three images are upsampled and downsampled to make their sizes consistent. Then, the images are stacked, fused, and convolved to obtain three final feature layers: 13×13×255 (Y1), 26×26×255 (Y2), and 52×52×255 (Y3). After classification and regression prediction, the target bounding box and target category and location information can be output.

[0063] This invention adopts the YOLOv3 SPP architecture based on YOLOv3. Its network is basically the same as YOLOv3, but it has three significant advantages.

[0064] ①Mosaic image enhancement:

[0065] To address the issues related to image processing in Batch Normalization (BN) layers in networks, four images are randomly cropped and stitched together onto a single image for training. This approach can improve detection speed to some extent, while also enriching the image background and increasing the batch size.

[0066] ②DIOU

[0067] Replacing the traditional IOU with DIOU in the calculation of the loss function has the following two advantages: First, it significantly improves the training convergence speed; second, it improves the detection accuracy to a certain extent. DIOU is calculated using the following formula:

[0068]

[0069] In the formula, ρ 2 (b,b gt ): b, b gt Euclidean distance, c:b and b gt The longest diagonal of the union.

[0070] b represents the predicted candidate box, b gt For target candidate boxes

[0071] Its range is between -1 and 1, which overcomes the unpredictability caused by the minimum IoU being zero, and has stronger application value in the application scenarios of this invention.

[0072] ③SPP module

[0073] Compared to the YOLOv3 series, the network architecture adds an SPP module, specifically as follows: Figure 4 As shown.

[0074] The SPP module is placed below the three convolutional layers and utilizes the spatial pyramid concept to fuse global and local features. It performs upsampling on an input image across different dimensions, ultimately obtaining four output feature maps, which are then fused to combine global and local features.

[0075] This invention draws on the application ideas of YOLOv3SPP and targets its network. According to the invention's purpose, it aims to achieve real-time all-weather road obstacle detection. The standardized training set is obtained by training with target images of calibrated road obstacles. The training converges to obtain relevant training parameters, and the target images can then be detected.

[0076] 4. Obstacle Distance Calculation and Alarm System

[0077] By using target detection algorithms, the relevant location information of road obstacles is obtained in real time, which completes the obstacle detection step. Next, the distance to the obstacle needs to be calculated, and an alarm is issued based on whether the distance reaches the set safety threshold.

[0078] The obstacle distance is calculated using the vehicle's GPS system, which acquires a point cloud matrix of the surrounding environment. The GPS system provides 3D data, allowing for the calculation of distances to obstacles on the road ahead. Since the obstacle location has already been determined by the target detection module, GPS obstacle localization is more targeted. The distances obtained from GPS are integrated into the output map and marked on the corresponding obstacle bounding boxes. When the distance exceeds a safety threshold A, the judgment parameters are transmitted to the processor, which then sends a control signal to command the alarm system to issue an alert. The judgment parameters are determined by a specific table in the following implementation plan.

[0079] Specifically, the practical application design process of this invention is as follows: Figure 5 As shown:

[0080] I. Dataset Acquisition

[0081] The dataset of this invention was obtained through two methods: one is through the Coco dataset on the Internet, and the other is through on-site collection via camera.

[0082] II. Dataset Grouping and Preprocessing

[0083] The obtained dataset is divided into training, validation, and prediction sets. First, the training data is prepared and its directory structure is defined. When labeling the data, a YOLO format label file (.txt) is directly generated. Simultaneously, the image sizes in the dataset are scaled to match those in Cocoa, ensuring that the image size is a multiple of 32. The dataset is then arranged according to a specific directory structure, as detailed in [link to directory structure]. Figure 6 The generated directory structure is used to generate a series of training files.

[0084] III. YOLO Model Network Training

[0085] Before training, the basic network structure needs to be configured with relevant parameters according to the research objectives of this invention. The specific configuration is as follows:

[0086] ① In the convolution module, set:

[0087] batch_normaliation=1, filters=32, size=3, stride=1, pad=1, activation=leaky.

[0088] ②Shortcut connection layer module settings: from = -3, activation = linear.

[0089] ③ Maxpool module settings: stride=1, size=5.

[0090] ④route layer: The method depends on the specific situation. When layers take one value or multiple values, concatenate will concatenate them in the depth direction.

[0091] ⑤ Upsample layer: Set stride=2

[0092] ⑥ YOLO layer: Settings

[0093] mask = 0, 1, 2; anchors = 10, 13, 16, 30, 33, 23, 30, 61, 62, 45, 59, 119, 116, 90, 156, 198, 373, 326; classes = 80; uncheck the settings for num, litter, ignore_thresh, truth_thresh, and random.

[0094] ⑦ Setting up a YOLOv3-SPP network: This can be done using the files parsed above.

[0095] IV. Establishment of Alarm System

[0096] This invention utilizes an STM32 microcontroller as the microprocessor, and the alarm information is executed through a vibration motor and a buzzer. The specific system design is as follows: Figure 7 As shown:

[0097] ①STM32 microcontroller circuit design

[0098] The design of a microcontroller consists of the design of the main board circuit, the design of various module circuits, and system programming.

[0099] ② Determining the safety threshold

[0100] Determining the minimum distance at which an obstacle poses a safety hazard requires considering multiple factors. From the moment a driver detects an obstacle to the moment the vehicle comes to a complete stop, there is a certain braking distance. Only when the safe distance is greater than the braking distance can passenger safety be guaranteed to a basic level. However, considering only braking factors is insufficient. Braking distance is directly and indirectly related to various factors such as driver fatigue, road surface adhesion coefficient, and visibility in the driving environment. Based on the analysis of driving distances, braking distance can typically be calculated using the following formula:

[0101]

[0102] In the formula, s1, s2, s3, and s4 represent the driver's reaction time travel distance, brake application distance, continuous braking distance, and brake release distance, respectively; τ1, τ2, τ3, and τ4 represent the duration (s) of the four stages; ua0 Let a be the speed before braking. bmax This is the maximum braking deceleration.

[0103] To comprehensively consider factors such as driver fatigue and slippery road conditions, a reaction time correction coefficient α is added to τ1 in the above formula. bmax Add a deceleration correction factor β in front; consider that the car may not be traveling at a constant speed before braking, in u a0 Adding an initial velocity correction coefficient γ, the total braking distance is expressed by the following formula (Note: The braking distance here is different from the traditional braking distance in automotive theory. To ensure safety, multiple factors are considered, including driver reaction time and vehicle travel distance, and the calculation is not based on the minimum distance, i.e., the braking deceleration is not calculated based on the maximum braking deceleration.)

[0104]

[0105] The specific correction coefficients are determined by adjusting them according to the specific application scenarios of the present invention.

[0106] For the safety threshold A, let A = S.

[0107] ③ Determining the judgment coefficient

[0108] To better achieve the purpose of early warning, this invention proposes to set up alarm types with different levels of danger, from mild to severe. This invention selects four alarm types, the details of which are shown in the table below:

[0109]

[0110] V. Installation and debugging of the physical system

[0111] The physical system installation and debugging includes the installation and debugging of the infrared image camera and visible light camera of the obstacle detection module, the connection and debugging of the STM32 main circuit, and the installation and debugging of the alarm system.

[0112] This invention reduces detection errors by improving the quality of the detected images. Specifically, it utilizes the aforementioned method of fusing infrared and visible light images. This combines the advantages of infrared images (excellent at extracting edge contour information based on differences in object radiation) and visible light images (clearly highlighting texture and other information) to emphasize road obstacles and improve detection accuracy. Currently, most research focuses on fusing radar and visible light sensors. Infrared and visible light fusion techniques are primarily applied in fields such as medicine and detection. Target detection algorithms generally focus on improving accuracy through algorithm upgrades. This invention, however, offers a novel approach by improving the quality of the input image to enhance detection accuracy while maintaining high detection speed.

Claims

1. A method for all-weather road obstacle detection and early warning based on visual fusion perception, comprising an all-weather road obstacle detection and early warning system based on visual fusion perception, including an infrared visual sensor, a CMOS visual sensor, an obstacle detection module, independent buttons, a power module, a main control chip, a GPS module, an LED warning light display module, an OLED display module, and a buzzer, characterized by: The steps are as follows, and the steps are performed in sequence. Step 1: Install an onboard camera and an infrared camera on the vehicle, and input the road image information captured by the two cameras into the microprocessor for image preprocessing; Step 2: Perform a fusion process on the images obtained in Step 1. ①Image calibration Visible light image IV n×n and infrared images IR n×n Perform corresponding point pairing; identical pixels in two images satisfy IV ij =ir i′j′ Furthermore, the pixels within a certain neighborhood of the corresponding point have similarity. Based on the visible light image, the infrared image is transformed for calibration. ②Image fusion Image fusion is performed using the unsupervised framework VIF-NET, which combines infrared and visible light. The feature map extracted from the visible light image and the contour image extracted from the infrared image are fed into generator 1 to generate the fused image. The discriminator results are used to construct a loss function and theoretically determined obstacle position images. Generator 2 generates a reference image for training generator 1. The loss function is used to constrain generator 1 so that the model reaches the Nash equilibrium point. The connections and biases between neurons in the hidden layer reach their optimal values, i.e., the weights w and biases b: the parameters that measure the connection between two neurons, reach their optimal values. Training ends, and image fusion is completed. Step 3: Input the fused image obtained in Step 2 into the YOLOv3SPP object detection network for detection. Use a standardized training set to combine global and local features to obtain the output feature map. The third step, the YOLOv3SPP object detection network, includes three convolutional layers and one SPP module. An input image undergoes a four-dimensional upsampling process to obtain four output feature maps. The four images are then randomly cropped and spliced ​​onto one image for training and fusion operation. Step 4: Based on the output feature map of Step 3, obtain the location information of road obstacles in real time through the target detection algorithm, calculate the distance to the obstacles, and issue a warning based on the set obstacle safety distance threshold. The obstacle safety distance threshold in step four is determined by the vehicle braking distance. In the formula, s1, s2, s3, and s4 represent the driver's reaction time travel distance, brake application distance, continuous braking distance, and brake release distance, respectively; τ1, τ2, τ3, and τ4 represent the duration (s) of the four stages; u a0 Let a be the speed before braking. bmax α is the maximum braking deceleration, β is the reaction time correction factor, γ is the deceleration correction factor, and γ is the initial velocity correction factor.

2. The all-weather road obstacle detection and early warning method based on visual fusion perception according to claim 1, characterized in that: The discriminator loss function construction in step two employs an improved SSIM algorithm, which, combined with the road obstacle detection principle of not recognizing obstacles but only detecting the size of the obstacle's outer contour and its distance from the vehicle, yields the SSIM result. In the formula, μ x and μ y It is the average value of the pixels, σ x and σ y It is the standard deviation of pixels, σ xy C1 and C2 are the covariance of the pixels, and C1 and C2 are the structure constants, taken as 9 × 10⁻⁶. -4 .

3. The all-weather road obstacle detection and early warning method based on visual fusion perception according to claim 1, characterized in that: The image fusion processing in step two requires consideration of fusion evaluation metrics including mutual information (MI), phase congruency (PC), and nonlinear correlation information entropy (Q). NCIE And the Universal Image Quality Index (UIQI).

Images