Road obstacle detection system and method based on visual semantic segmentation

Abstract

The invention discloses a road obstacle detection system and method based on visual semantic segmentation, belongs to the technical field of computer vision, and aims to solve the problems that in a conventional method, the detection precision is greatly reduced when a detection scheme based on feature extraction and a classifier is combined, and the detection precision is greatly reduced when the conventional road obstacle detection scheme based on semantic segmentation is used for a complex scene. Missing detection or false detection is easy to occur; and a detection strategy cannot be dynamically optimized according to the complexity of a scene. The road obstacle detection system based on visual semantic segmentation comprises a multi-source image sensing module, an image preprocessing module, a self-adaptive image enhancement module, an improved semantic segmentation module, an obstacle recognition and positioning module, a dynamic optimization decision module and a data storage and interaction module. According to the method, multi-source image fusion and the improved segmentation network are combined, the number of times of missing detection and false detection can be effectively reduced, the method is particularly suitable for detection of small targets or shielding obstacles, and the detection precision is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a road obstacle detection system and method based on visual semantic segmentation. Background Technology

[0002] The road obstacle detection system is an intelligent traffic perception system based on sensor and artificial intelligence technologies. Its core function is to identify and locate static or dynamic obstacles on the road in real time and to issue timely warnings to vehicles or road management systems.

[0003] Currently, vision-based road obstacle detection methods mainly include traditional computer vision methods and deep learning methods. Traditional methods, such as those based on feature extraction combined with classifiers, rely on manually designed features. While effective in simple traffic scenarios, they suffer from insufficient feature extraction capabilities in complex lighting, inclement weather, and scenarios with diverse obstacle shapes, leading to a significant drop in detection accuracy and poor generalization ability. Existing semantic segmentation-based road obstacle detection schemes mostly use a single segmentation network to segment the road image and then perform simple post-processing for obstacle recognition. However, these schemes have several shortcomings, specifically as follows: 1. The image sensing stage often uses monocular cameras, which are limited by the field of view and struggle to acquire depth information of obstacles, resulting in insufficient positioning accuracy and an inability to accurately determine the distance between obstacles and vehicles. Secondly, segmentation networks perform poorly on low-resolution obstacles and those with severe occlusion, easily leading to missed or false detections. Thirdly, there is a lack of optimization processing for image quality in complex environments; when images contain noise, blur, or uneven illumination, segmentation accuracy and real-time detection performance significantly decrease. Fourthly, existing detection processes lack adaptive adjustment mechanisms, failing to dynamically optimize detection strategies based on scene complexity, making it difficult to balance detection accuracy and real-time performance. To address these issues, a road obstacle detection system and method based on visual semantic segmentation is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a road obstacle detection system and method based on visual semantic segmentation. By using this device, the invention solves the problems mentioned above, such as the significant decrease in detection accuracy of traditional methods based on feature extraction and classifiers when facing complex scenes, and the problems of existing road obstacle detection schemes based on semantic segmentation being prone to missed detections or false detections and unable to dynamically optimize the detection strategy according to the complexity of the scene.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a road obstacle detection method based on visual semantic segmentation, comprising the following steps: S1: Multi-source image acquisition and synchronization: Multi-view and multi-resolution image data of road scene are acquired synchronously through the left eye camera, right eye camera, forward-looking high-definition camera and infrared camera of the multi-source image sensing module, and the temporal consistency of multi-source images is ensured through the image synchronization control unit. S2: Image preprocessing: The original image acquired in S1 is subjected to image calibration, image alignment, noise removal, and image size standardization to eliminate image distortion and noise interference, thereby obtaining standardized image data; S3: Adaptive Image Enhancement: The preprocessed image is evaluated for quality, and an image enhancement strategy is adaptively selected based on the evaluation results to enhance the image in a targeted manner, thereby improving the image details and quality in obstacle areas; S4: Improved semantic segmentation: Employs a semantic segmentation network based on attention mechanism and multi-scale feature fusion to perform pixel-level semantic segmentation on the enhanced image and output segmentation results for different categories, including roads, obstacles and background; S5: Obstacle Recognition and Localization: Post-process the semantic segmentation results and extract the obstacle region. Then, construct the feature vector of the obstacle, identify the obstacle category through a classifier, and combine the depth information of the stereo image to realize the three-dimensional localization of the obstacle. S6: Dynamic Optimization and Result Output: Based on the complexity of the current scene and the image quality, dynamically adjust the detection parameters and process to optimize detection performance, and store and transmit the obstacle detection results to external devices.

[0006] Furthermore, in S1, the image synchronization control unit realizes the synchronous acquisition of the left eye camera, right eye camera, forward-looking high-definition camera and infrared camera through hardware triggering. Multi-source image acquisition also includes initial settings for camera parameters, including exposure time, gain, and white balance, which can be automatically adjusted according to the current lighting conditions.

[0007] Furthermore, in S2, image calibration includes intrinsic parameter calibration and extrinsic parameter calibration, as detailed below: The intrinsic parameter calibration adopts the Zhang Zhengyou calibration method, which calculates the camera's intrinsic parameter matrix and distortion coefficient by taking multiple chessboard images, in order to eliminate lens distortion; External parameter calibration obtains the relative positional relationships between the left eye camera, right eye camera, forward-looking high-definition camera, and infrared camera through stereo calibration, and the relative positional relationships include rotation matrix and translation vector; Image alignment is achieved based on extrinsic calibration results, that is, by using a homography transformation algorithm to align multiple source images to the same coordinate system.

[0008] Furthermore, in S2, noise removal is achieved using an adaptive median filtering algorithm, and the size of the filtering window can be dynamically adjusted according to the density of noise in the image; Image size normalization adjusts all preprocessed images to a uniform resolution, and bilinear interpolation is used for image scaling to ensure the integrity of image edge details.

[0009] Furthermore, in S3, image quality assessment is performed across four dimensions: brightness uniformity, contrast, sharpness, and signal-to-noise ratio, as detailed below: Brightness uniformity is evaluated by calculating the mean and variance of the image brightness; contrast is evaluated by calculating the dynamic range of the image grayscale values; sharpness is evaluated by calculating the edge strength of the image using the Laplacian operator; signal-to-noise ratio is evaluated by calculating the ratio of the image signal to noise; and then, based on the evaluation results of the four dimensions, a weighted summation method is used to obtain the image quality score.

[0010] Furthermore, in S3, the selection rules for the adaptive enhancement strategy are as follows: When the image quality score and brightness uniformity score are less than the corresponding thresholds, a multi-scale enhancement algorithm based on Retinex theory is used. When the image quality score and contrast score are less than the corresponding thresholds, an adaptive histogram equalization algorithm is used. When the image quality score and sharpness score are less than the corresponding thresholds, a deep learning-based image deblurring algorithm is used. When the image quality score and signal-to-noise ratio score are less than the corresponding thresholds, an algorithm combining wavelet thresholding denoising and contrast enhancement is used. If the image quality score is greater than or equal to the corresponding threshold, no additional image enhancement processing will be performed.

[0011] Furthermore, in S4, the improved semantic segmentation network includes an encoder, a decoder, and an attention fusion unit, as detailed below; The encoding end uses ResNet-50 as the backbone network, which extracts features through several convolutional blocks. Each convolutional block contains multiple convolutional layers, batch normalization layers, and ReLU activation functions. Finally, it downsamples through max pooling layers to output several feature maps of different scales. The decoding end adopts the symmetrical structure of U-Net, and performs several upsampling operations on the smallest scale feature map output by the encoding end. After each upsampling, it is concatenated with the feature map of the corresponding scale of the encoding end, and then the feature information is fused through convolutional layers to gradually improve the resolution of the feature map, and finally outputs a semantic segmentation result with the same size as the original image. The attention fusion unit is located at the feature concatenation point between the encoder and decoder. It includes a spatial attention module and a channel attention module. The spatial attention module focuses on the spatial region where the obstacle is located by calculating the spatial weight matrix of the feature map. The channel attention module can adaptively adjust the weights of different feature channels through the squeeze-excitation operation, thereby highlighting effective feature information.

[0012] Furthermore, in S5, obstacle region extraction is achieved by performing connected component analysis on the semantic segmentation results, as follows: In the semantic segmentation results, obstacle category pixels are labeled as foreground and other category pixels are labeled as background. An eight-neighbor connected component detection algorithm is used to extract foreground connected components, and the area of ​​each connected component is calculated. By removing connected components with an area smaller than a preset threshold, the final obstacle region can be obtained. Feature extraction uses the SIFT algorithm to extract key feature points in the obstacle region and calculate a 128-dimensional feature descriptor for each feature point. Then, combined with the category prior information obtained from semantic segmentation, the feature descriptors can be weighted and optimized to construct the feature vector of the obstacle. Category recognition uses a support vector machine classifier, which is trained on a pre-labeled obstacle sample dataset. The training samples include multiple obstacle types, and cross-validation is used to optimize the classifier parameters during training. 3D localization combines stereo images acquired by the left and right cameras, calculates the depth map of the obstacle area using the SGBM stereo matching algorithm, and then converts the pixel coordinates of the obstacle area into world coordinates based on the camera's intrinsic and extrinsic parameters and depth information to obtain the obstacle's 3D position information. At the same time, it calculates the bounding box of the obstacle area to obtain the obstacle's size information.

[0013] Furthermore, in S6, scene complexity assessment can be classified according to the number, density, occlusion degree of obstacles in the image and the type of road environment, and can be divided into simple scene, general scene and complex scene. The dynamically adjusted detection parameters include the kernel size, learning rate, and batch size of the improved semantic segmentation network, and the detection parameters can be adaptively adjusted according to simple, general, and complex scenarios. The process optimization is adjusted based on the results of image quality scores and scene complexity classifications. Specifically, when the image quality score is greater than or equal to the corresponding threshold and the scene is simple, the image enhancement process in S3 is skipped and semantic segmentation in S4 is performed directly. When the image quality score is less than the corresponding threshold and the scene is complex, multi-level image enhancement processing is performed in S3, followed by denoising, contrast enhancement and deblurring processing, and then semantic segmentation in S4.

[0014] The present invention also proposes another technical solution: a road obstacle detection system based on visual semantic segmentation, comprising a multi-source image sensing module, an image preprocessing module, an adaptive image enhancement module, an improved semantic segmentation module, an obstacle recognition and localization module, a dynamic optimization decision module, and a data storage and interaction module; Multi-source image sensing module: including left eye camera, right eye camera, forward-looking high-definition camera, infrared camera and image synchronization control unit, mainly used to acquire multi-view and multi-resolution image data of road scenes; Image preprocessing module: mainly used for calibration, alignment, noise removal, and size normalization of the original image; Adaptive Image Enhancement Module: Primarily used to assess the quality of preprocessed images and adaptively select enhancement strategies based on the assessment results; Improved semantic segmentation module: used for pixel-level semantic segmentation of the enhanced image; Obstacle recognition and localization module: used to extract obstacle regions, identify obstacle categories, and achieve three-dimensional localization; Dynamic optimization decision module: It can dynamically adjust detection parameters and processes based on scene complexity and image quality; Data storage and interaction module: Used to store various types of data during the detection process and to interact with external devices.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention combines multi-source image fusion with an improved segmentation network, which can effectively reduce the number of missed and false detections, and is especially suitable for the detection of small targets or occluded obstacles, thus significantly improving detection accuracy.

[0016] 2. The infrared camera in this invention can adapt to low light and severe weather, and adaptive enhancement solves problems such as uneven lighting and blurring, enabling the system to work stably in complex environments such as strong light, rain, and fog, thus overcoming the limitations of traditional methods in terms of weak environmental adaptability.

[0017] 3. The present invention adopts a dynamic optimization decision-making mechanism that can adjust the strategy according to the complexity of the scene, which has the effect of faster detection speed in simple scenes and better accuracy in complex scenes, and can meet the dual requirements of real-time performance and reliability.

[0018] 4. This invention uses stereo matching between a binocular camera and an SGBM to achieve three-dimensional positioning, which can accurately output the world coordinates and size information of obstacles, solving the problems of insufficient positioning accuracy and inability to determine the distance to obstacles in the prior art.

[0019] 5. This invention employs a combination of multi-source data acquisition and data augmentation training, enabling the model to adapt to various scenarios such as highways, urban roads, and rural roads, thus having a wide range of applications. Attached Figure Description

[0020] Figure 1 This is a system flowchart of the present invention; Figure 2 This is a diagram illustrating the method steps of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To address the shortcomings of traditional feature extraction-based classifier-based detection methods, such as insufficient feature extraction capabilities leading to significant drops in detection accuracy in complex scenes, and the limitations of existing semantic segmentation-based road obstacle detection schemes in acquiring obstacle depth information due to viewpoint constraints, susceptibility to false positives or false negatives, and inability to dynamically optimize detection strategies based on scene complexity, such as… Figure 1 As shown, the following preferred technical solutions are provided: A road obstacle detection system based on visual semantic segmentation includes a multi-source image sensing module, an image preprocessing module, an adaptive image enhancement module, an improved semantic segmentation module, an obstacle recognition and localization module, a dynamic optimization decision-making module, and a data storage and interaction module, as detailed below: Multi-source image sensing module: including left eye camera, right eye camera, forward-looking high-definition camera, infrared camera and image synchronization control unit, mainly used to acquire multi-view and multi-resolution image data of road scenes; Image preprocessing module: mainly used for calibration, alignment, noise removal, and size normalization of the original image; Adaptive Image Enhancement Module: Primarily used to assess the quality of preprocessed images and adaptively select enhancement strategies based on the assessment results; Improved semantic segmentation module: used for pixel-level semantic segmentation of the enhanced image; Obstacle recognition and localization module: used to extract obstacle regions, identify obstacle categories, and achieve three-dimensional localization; Dynamic optimization decision module: It can dynamically adjust detection parameters and processes based on scene complexity and image quality; Data storage and interaction module: Used to store various types of data during the detection process and to interact with external devices.

[0023] like Figure 2 As shown, the present invention also proposes another implementation method: a road obstacle detection method based on visual semantic segmentation, comprising the following steps: S1: Multi-source image acquisition and synchronization: Multi-view and multi-resolution image data of the road scene are simultaneously acquired through the left eye camera, right eye camera, forward-looking high-definition camera, and infrared camera of the multi-source image sensing module, and the temporal consistency of the multi-source images is ensured by the image synchronization control unit, as detailed below: The multi-source image sensing module adopts a multi-camera fusion scheme, including a left-eye camera, a right-eye camera, a forward-looking high-definition camera, an infrared camera, and an image synchronization control unit. The left-eye and right-eye cameras are used to acquire stereo images of the road scene, providing data support for subsequent depth calculations. The forward-looking high-definition camera is used to acquire high-resolution color images and capture detailed features of obstacles. The infrared camera can acquire clear scene images in harsh environments such as low light, rain, and fog, compensating for the shortcomings of the visible light camera.

[0024] The image synchronization control unit is used to synchronize the triggering of the left eye camera, right eye camera, forward-looking high-definition camera, and infrared camera. It controls the shutter opening time of these cameras by outputting synchronization pulse signals to ensure that the acquisition time difference of the multi-source images does not exceed 1ms, thus guaranteeing the temporal consistency of the multi-source images. Simultaneously, the image synchronization control unit also has camera parameter configuration functions, automatically adjusting the exposure time, gain, and white balance parameters of each camera according to the current lighting conditions to ensure stable image quality. In addition, the multi-source image sensing module includes an image acquisition card, which can transmit image data acquired by each camera to other modules for subsequent processing.

[0025] S2: Image Preprocessing: The raw images acquired in S1 undergo image calibration, image alignment, noise removal, and image size standardization to eliminate image distortion and noise interference, thereby obtaining standardized image data, as detailed below: The image preprocessing module is mainly used to perform functions such as image calibration, image alignment, noise removal, and image size standardization. It can eliminate distortion and noise interference generated during image acquisition and provide high-quality image data for subsequent processing steps.

[0026] Image calibration includes intrinsic parameter calibration and extrinsic parameter calibration, as detailed below: The intrinsic parameter calibration adopts the Zhang Zhengyou calibration method. The specific steps are as follows: prepare a checkerboard calibration board, place the calibration board at different positions and angles in the road scene, and then take 15-20 checkerboard images with each camera; use the calibrateCamera function in the OpenCV library to process the captured checkerboard images and calculate the intrinsic parameter matrix (including focal length and principal point coordinates) and distortion coefficients (radial distortion coefficient and tangential distortion coefficient) of each camera; then, based on the calculated distortion coefficients, use the undistort function to perform distortion correction on the original image to eliminate the influence of lens distortion.

[0027] Extrinsic parameter calibration is achieved through stereo calibration to obtain the relative positional relationships between the cameras. Taking the calibration of the left and right cameras as an example, the specific steps are as follows: fix the left and right cameras on the same bracket and keep their relative positions unchanged. Then, take 15-20 stereo image pairs containing the checkerboard calibration plate. Then, use the stereoCalibrate function in the OpenCV library to process the stereo image pairs and calculate the rotation matrix and translation vector between the two cameras. Then, use the same method to complete the extrinsic parameter calibration between the forward-looking HD camera and the infrared camera, and between the forward-looking HD camera and the left camera, and obtain the relative positional relationship parameters between the cameras.

[0028] Image alignment is achieved based on the rotation matrix and translation vector obtained from extrinsic parameter calibration. The homography transformation algorithm is used to align multi-source images to the same coordinate system, with the coordinate system of the foreground high-definition camera as the reference coordinate system. The specific steps are as follows: according to the extrinsic parameters, the homography matrix of each camera image to the reference coordinate system is calculated. Then, the warpPerspective function is used to perform homography transformation on the images acquired by each camera, thereby realizing the alignment of multi-source images and ensuring that the position of the same obstacle is consistent in different images.

[0029] Noise removal is achieved using an adaptive median filtering algorithm. This algorithm dynamically adjusts the size of the filtering window based on the density and intensity of noise in the image, effectively removing noise while preserving the edge details of obstacles. The specific steps are as follows: traverse each pixel in the image, set the initial filtering window size to 3×3 with that pixel as the center, and calculate the maximum, minimum, and median values ​​of the pixels within the window; if the current pixel value is greater than the maximum value or less than the minimum value of the window, it is determined to be a noise point. In this case, the size of the filtering window is increased and the median value within the window is recalculated, and then the current pixel value is replaced with the median value; if it is not a noise point, the current pixel value remains unchanged.

[0030] Image size standardization adjusts all preprocessed images to a uniform resolution, and then bilinear interpolation is used for image scaling. Bilinear interpolation calculates the weighted average of the four neighboring pixels of the current pixel as the value of the scaled pixel, which can effectively ensure the smoothness of the edges and the integrity of details after image scaling, and avoid the phenomenon of jagged distortion. In the specific implementation, the image size standardization process can be completed by using the resize function in the OpenCV library and setting the interpolation mode to INTER_LINEAR.

[0031] S3: Adaptive Image Enhancement: The preprocessed image undergoes quality assessment. Based on the assessment results, an adaptive image enhancement strategy is selected for targeted enhancement, improving image details and quality in obstacle areas. Specifically: The adaptive image enhancement module can adaptively select enhancement strategies based on the quality of the preprocessed image to specifically improve image quality, especially the detailed information of obstacle areas.

[0032] The adaptive image enhancement module includes an image quality assessment unit and an enhancement strategy selection unit, as detailed below: The image quality assessment unit quantitatively evaluates the preprocessed image from four dimensions: brightness uniformity, contrast, sharpness, and signal-to-noise ratio. The specific evaluation method is as follows: Brightness uniformity evaluation: First, convert the image to grayscale, then calculate the average brightness of the grayscale image. and variance Brightness uniformity score The rating ranges from 0 to 100 points, with a higher rating indicating more uniform brightness.

[0033] Contrast evaluation: Calculate the maximum grayscale value of a grayscale image. and minimum gray value Contrast is expressed as Contrast rating is expressed as The rating ranges from 0 to 100 points, with higher ratings indicating higher contrast.

[0034] Sharpness assessment: Edge detection is performed on the grayscale image using the Laplacian operator, and the mean of the edge detection results is calculated. Clarity rating: The rating ranges from 0 to 100 points, with higher ratings indicating clearer images.

[0035] Signal-to-noise ratio (SNR) evaluation: A wavelet transform-based method is used to decompose the grayscale image into low-frequency and high-frequency components. The low-frequency component represents the image signal, and the high-frequency component represents noise. The energy of the low-frequency component is then calculated. and the energy of high-frequency components Signal-to-noise ratio Signal-to-noise ratio score The scoring range is 0-100 points, with a higher score indicating a higher signal-to-noise ratio.

[0036] Based on the scores from the four dimensions mentioned above, a weighted summation method is used to calculate the total image quality score. The total score ranges from 0 to 100 points.

[0037] The enhancement strategy selection unit can select based on the total image quality score. Based on the scores across each dimension, the corresponding enhancement algorithm is adaptively selected, with the specific selection rules as follows: when and When an image exhibits uneven illumination, a multi-scale enhancement algorithm based on Retinex theory is employed. This algorithm decomposes the image into illuminance and reflectance components and enhances the reflectance component to achieve uniform illumination. The specific steps are as follows: The image is converted to the logarithmic domain and then decomposed into logarithmic illuminance and logarithmic reflectance components. Multi-scale Gaussian filtering is then used to smooth the logarithmic illuminance component. The enhanced logarithmic reflectance component can be obtained by calculating the logarithmic reflectance component = logarithmic image - smoothed logarithmic illuminance component. Finally, the logarithmic reflectance component is converted back to the spatial domain to obtain the image with uniform illumination.

[0038] when and When the image contrast is low, an adaptive histogram equalization algorithm (CLAHE) is used. CLAHE divides the image into multiple sub-blocks and then equalizes the histogram of each sub-block while limiting the contrast gain to avoid noise amplification. Specifically: For example, the image is divided into 8×8 sub-blocks, the histogram of each sub-block is calculated, and the cropping threshold of the histogram is determined, which is usually set to 0.02. Then, the cropped histogram is equalized to obtain the mapping function of each sub-block. Then, the mapping results of adjacent sub-blocks are smoothly transitioned by the bilinear interpolation algorithm, thus obtaining the enhanced image.

[0039] when and When an image is blurry, a deep learning-based image deblurring algorithm is used. This algorithm employs an improved U-Net network as the deblurring model, with the input being the blurry image and the output being the deblurred, clear image. The deblurring model is trained using a dataset containing 100,000 pairs of blurry-clear road images. The training loss function is the L1 loss function, the optimizer is the Adam optimizer, the learning rate is set to 0.001, and the number of training iterations is set to 50,000. The model trained according to the above method can quickly deblur images, thereby improving image clarity.

[0040] when and When the image signal-to-noise ratio is low, an algorithm combining wavelet thresholding denoising and contrast enhancement is used for processing, as follows: By performing a two-stage wavelet decomposition on the image, low-frequency and high-frequency components can be obtained. Then, a soft thresholding function is used to threshold the high-frequency components to remove noise. The processed high-frequency components and low-frequency components are then subjected to inverse wavelet transform to obtain a denoised image. Finally, an adaptive histogram equalization algorithm is used to enhance the contrast of the denoised image, thereby improving the image quality.

[0041] when At this time, the image quality is good, and no additional image enhancement processing is required. The preprocessed image can be directly transmitted to the subsequent modules.

[0042] S4: Improved Semantic Segmentation: Employs a semantic segmentation network based on attention mechanisms and multi-scale feature fusion to perform pixel-level semantic segmentation on the enhanced image and output segmentation results for different categories, including roads, obstacles, and background, as detailed below: The improved semantic segmentation module employs a semantic segmentation network based on attention mechanism and multi-scale feature fusion, which can achieve pixel-level semantic segmentation of the enhanced image and output segmentation results for seven different categories, including roads, pedestrians, motor vehicles, non-motor vehicles, scattered objects, construction facilities, and background.

[0043] The structure of the improved semantic segmentation network includes an encoder, a decoder, and an attention fusion unit, as detailed below: Encoding end: ResNet-50 is used as the backbone network to extract multi-scale features of the image. ResNet-50 includes 5 convolutional blocks (Conv1-Conv5), and each convolutional block consists of multiple convolutional layers, batch normalization layers (BN) and ReLU activation functions. Residual connections are introduced to avoid the gradient vanishing problem of deep networks. The Conv1 block contains a 7×7 convolutional layer (stride 2), a BN layer, and a ReLU activation function, with the output feature map scale being 1 / 2 of the input image. The Conv2 block contains three 3×3 convolutional layers, a BN layer, and a ReLU activation function, with the output feature map scale being 1 / 4 of the input image. The Conv3 block contains four 3×3 convolutional layers, a BN layer, and a ReLU activation function, with the output feature map scale being 1 / 8 of the input image. The Conv4 block contains six 3×3 convolutional layers, a BN layer, and a ReLU activation function, with the output feature map scale being 1 / 16 of the input image. The Conv5 block contains three 3×3 convolutional layers, a BN layer, and a ReLU activation function, with the output feature map scale being 1 / 32 of the input image. This allows the encoder to output five feature maps (C1-C5) of different scales, corresponding to the outputs of the five convolutional blocks.

[0044] Decoding end: The U-Net symmetric structure is adopted to restore the resolution of the feature map and fuse multi-scale features; the decoding end includes 5 upsampling blocks (Up1-Up5), and each upsampling block uses transposed convolution to achieve upsampling (stride of 2), which can double the resolution of the input feature map; The Up1 block upsamples the C5 feature map output from the encoder to obtain a feature map with the same scale as the C4 feature map. This feature map is then concatenated with the C4 feature map, and the feature information is fused using two 3×3 convolutional layers, a BN layer, and a ReLU activation function to output a fused feature map. The Up2 block upsamples the feature map output from the Up1 block and concatenates and fuses it with the C3 feature map to output a fused feature map. This process continues, with the Up3 block concatenating and fusing with the C2 feature map, the Up4 block concatenating and fusing with the C1 feature map, and the Up5 block upsampling the feature map output from the Up4 block to obtain a feature map with the same scale as the original input image. Finally, a 1×1 convolutional layer converts the number of channels in the feature map into the number of classes (7 classes), resulting in the semantic segmentation result.

[0045] Attention fusion unit: Located at the feature concatenation point between the encoder and decoder (i.e., the concatenation point between Up1-Up4 block and C4-C1 feature map), it is used to enhance the expression of effective feature information. The attention fusion unit includes a spatial attention module and a channel attention module, and adopts a parallel structure. The output feature maps of the spatial attention module and the channel attention module are added element by element to obtain the final attention fusion feature map.

[0046] The spatial attention module focuses on the spatial region where obstacles are located, enhancing the ability to express spatial features, as detailed below: Global average pooling and global max pooling are performed on the feature map output from the encoder to obtain two 1×1×C feature vectors, where C is the number of channels in the feature map. The two feature vectors are concatenated and passed through a 1×1 convolutional layer (reducing the dimension to C / 2 channels), a ReLU activation function, and a 1×1 convolutional layer (increasing the dimension to C channels) to obtain a spatial weight matrix. Then, the spatial weight matrix is ​​multiplied element-wise with the feature map output from the encoder to obtain the feature map with spatial attention enhancement.

[0047] The channel attention module adaptively adjusts the weights of different feature channels to highlight feature information that is effective for obstacle segmentation, as detailed below: Global average pooling is applied to the feature map output from the encoder to obtain a 1×1×C feature vector. Channel weight vectors are obtained by passing two fully connected layers (the first fully connected layer reduces the dimension to C / 16 channels, and the second fully connected layer increases the dimension to C channels), ReLU activation function, and Sigmoid activation function. Then, the channel weight vector is multiplied element-wise with the feature map output from the encoder to obtain the feature map after channel attention enhancement.

[0048] The training process of the improved semantic segmentation network is as follows: For example, a dataset containing 200,000 labeled road scene images is used as the training set, with the image resolution set to 1024×512 pixels. The labeled categories include seven types: roads, pedestrians, motor vehicles, non-motor vehicles, debris, construction facilities, and background. The cross-entropy loss function is used as the training loss function, and the Dice loss function is introduced to improve the segmentation accuracy of small target obstacles (such as debris). Then, the Adam optimizer is used, with the initial learning rate set to 0.0001 and a multinomial decay strategy used to adjust the learning rate. The training batch size is set to 16, and the number of training iterations is set to 80,000. During training, data augmentation techniques such as random flipping, random cropping, and random scaling are used to improve the model's generalization ability. After training, the model is tested on the test set and meets the requirements for accurate segmentation.

[0049] S5: Obstacle Recognition and Localization: Post-processing of semantic segmentation results and extraction of obstacle regions are performed. Then, feature vectors of the obstacles are constructed, and obstacle categories are identified through a classifier. Combined with depth information from stereo images, the obstacles are located in 3D. The details are as follows: Obstacle region extraction is achieved by performing connected component analysis on the semantic segmentation results. The specific steps are as follows: The semantic segmentation result is a single-channel image, where each pixel value represents the corresponding category label: 0-background, 1-road, 2-pedestrian, 3-motor vehicle, 4-non-motor vehicle, 5-scattered objects, 6-construction facilities. Pixels with category labels 2-6 are marked as foreground, i.e., obstacle areas; pixels with category labels 0 and 1 are marked as background.

[0050] An eight-neighbor connected component detection algorithm is used to extract foreground connected components. Specifically, each pixel in the foreground image is traversed. If the pixel is unmarked and is a foreground pixel, it is used as a seed point. Then, all foreground pixels in its eight-neighbor area are searched to form a connected component, and a unique identifier is assigned to the connected component.

[0051] Calculate the area (in pixels) of each connected component, set an area threshold of 50 pixels, and then remove connected components with an area less than 50 pixels (considered as noise or missegmented regions). This yields the final obstacle regions, with each obstacle region corresponding to a connected component, and the bounding box coordinates of each connected component can be output. ,in The coordinates of the top left corner of the bounding box. These are the coordinates of the bottom right corner of the bounding box.

[0052] Obstacle category recognition employs a scheme combining SIFT feature extraction and an SVM classifier, as detailed below: Feature extraction: For the bounding box of each obstacle region, the corresponding region is cropped from the enhanced color image and converted into a grayscale image; then, the SIFT algorithm is used to extract the key feature points of the grayscale image, specifically: constructing a Gaussian difference pyramid and detecting extreme points; among them, the extreme points are accurately located to remove unstable extreme points, which can determine the principal direction of the extreme points, thereby achieving rotation invariance of the feature points; then, a 128-dimensional feature descriptor for each feature point is calculated, which can achieve scale invariance and illumination invariance of the feature points.

[0053] Feature optimization: Utilizing the prior category information obtained from semantic segmentation, the extracted SIFT feature descriptors are weighted and optimized. Specifically, the semantic segmentation result provides an initial category label for each obstacle region, i.e., the category label corresponding to the connected component. Then, based on the initial category label, the corresponding weight vector is obtained from the pre-trained feature weight library, and this weight vector is element-wise multiplied with the SIFT feature descriptor to obtain the optimized feature vector. The feature weight library is obtained by statistically analyzing the SIFT feature descriptors of different categories of obstacles in the training set, and each category corresponds to a weight vector to highlight the key features of that category of obstacle.

[0054] Category Classification: An SVM classifier is used to classify the optimized feature vectors to achieve accurate obstacle category identification. Specifically, the SVM classifier employs a one-to-one multi-classification strategy, constructing 21 binary SVM models for 7 categories. The training process of the SVM models is as follows: First, a training set is collected, with each sample consisting of the optimized feature vector of an obstacle region and its corresponding true category label; then, a grid search method is used to optimize the parameters of the SVM models, including the penalty coefficient. and kernel function parameters Then, cross-validation is used to train the SVM model to improve its generalization ability. After the SVM model is trained, it is tested on the test set. During the test, it can accurately identify the categories of obstacles such as pedestrians, motor vehicles, non-motor vehicles, scattered objects, and construction facilities.

[0055] 3D obstacle localization combines stereo images captured by the left and right cameras, calculates depth information using a stereo matching algorithm, and then converts pixel coordinates into world coordinates to achieve 3D obstacle localization, as detailed below: Depth Map Calculation: The SGBM stereo matching algorithm is used to process the stereo image pairs acquired by the left and right cameras and calculate the depth map. The SGBM stereo matching algorithm calculates the disparity of corresponding pixels in the left and right images, and then combines the camera's baseline length and focal length to calculate the pixel's depth information. Specifically, the left and right images are first converted to grayscale and histogram equalized; then the matching window size (9×9) and the disparity range of 0-128 pixels are set; the matching cost of each pixel in the left and right images is calculated using Census transform, and then the matching cost is optimized using a dynamic programming algorithm to obtain the initial disparity map. The initial disparity map is then processed to remove incorrect matches and perform disparity smoothing to obtain the final disparity map; then, according to the formula... The calculation can convert a disparity map into a depth map, where the formula contains... For depth, For camera focal length, The baseline lengths of the left and right cameras. For parallax.

[0056] Pixel coordinates to world coordinates: Based on the intrinsic and extrinsic parameter matrices of the left-eye camera, convert the center pixel coordinates of the obstacle region bounding box. Convert to world coordinates And the camera's intrinsic parameter matrix for: ,in and The focal length of the camera (in pixels). and Principal point coordinates (in pixels); camera extrinsic matrix Including rotation matrix Translation vector This function is used to convert coordinates in the camera coordinate system to coordinates in the world coordinate system. The specific conversion steps are as follows: Convert pixel coordinates... Convert to normalized coordinates in camera coordinate system The formula is: ; ; The depth of the pixel is obtained from the depth map. And the coordinates in the camera coordinate system for: ; ; ; The coordinates in the camera coordinate system are converted to coordinates in the world coordinate system using the extrinsic parameter matrix. The formula is: .

[0057] Obstacle size calculation: Based on the pixel coordinates of the obstacle region's bounding box and the depth map, the actual size of the obstacle is calculated. Taking a motor vehicle as an example, the width is calculated as the difference between the pixel coordinates of the left and right sides of the bounding box. According to the formula, the width is Height calculation: the difference in pixel coordinates between the top and bottom edges of the bounding box. According to the formula, the height is The length is estimated based on the type of motor vehicle and a predefined proportional relationship, or calculated through matching of multi-view images.

[0058] S6: Dynamic Optimization and Result Output: Based on the complexity of the current scene and image quality, the detection parameters and process are dynamically adjusted to optimize detection performance. The obstacle detection results are stored and transmitted to external devices, as detailed below: Scene complexity assessment can be graded based on the number, density, and occlusion degree of obstacles in the image, as well as the type of road environment. Specific assessment indicators and grading rules are as follows: Number of obstacles: The number of obstacles obtained after extracting the obstacle region is counted. ,when When the number of obstacles is small; when At that time, the number of obstacles is medium. At that time, there were many obstacles.

[0059] Obstacle density: calculated by the total area of ​​the obstacle region. Area of ​​the road region corresponding to the image The ratio of these values ​​gives the obstacle density as... ,when At that time, the density was low. At medium density, It is a high-density time.

[0060] Occlusion level: For each obstacle region, calculate the proportion of its area that is occluded by other obstacles. ,when Low shading is defined as <30% and ≤30% as high shading. When ≤60%, it is considered moderate occlusion; when A value greater than 60% is considered high occlusion; then, the average occlusion level of all obstacles is taken as the occlusion level of the scene.

[0061] Road environment type: The road environment type is determined by the road area features in the semantic segmentation results and other environmental features in the image. It can be divided into four categories: highway, urban road, rural road and construction road. Among them, highway and urban road are conventional environments, while rural road and construction road are complex environments.

[0062] Based on the above evaluation indicators, the complexity of a scenario can be divided into three levels: simple scenario, general scenario, and complex scenario, as detailed below: Simple scenario: , as well as <30%, and the road environment is a highway.

[0063] Typical scenario: , and 30%≤ ≤60%, and the road environment is urban road.

[0064] Complex scenarios: , as well as If the percentage is greater than 60%, and the road environment is rural road or construction road, it still falls under the category of complex scenarios.

[0065] Based on the above classification of scenario complexity, the key parameters of the improved semantic segmentation network are dynamically adjusted, including the convolutional kernel size, learning rate, and batch size, as follows: Simple scenario: Focusing on real-time detection, a smaller convolutional kernel and a higher learning rate are used, with a larger batch size. Specific parameters: convolutional kernel size is set to 3×3, learning rate is set to 0.01, and batch size is set to 32. Using a small 3×3 convolutional kernel reduces computation and improves processing speed; a higher learning rate speeds up model inference; and a larger batch size fully utilizes the parallel computing power of the GPU, thereby improving overall processing efficiency.

[0066] Typical scenario: Focusing on balancing detection accuracy and real-time performance, a medium-sized convolutional kernel and a moderate learning rate are used, with a medium batch size. Specific parameters: convolutional kernel size is set to 5×5, learning rate is set to 0.001, and batch size is set to 16. The 5×5 convolutional kernel has stronger feature extraction capabilities than the 3×3 convolutional kernel, which can improve detection accuracy. The moderate learning rate and batch size can maintain a certain processing speed while ensuring accuracy.

[0067] For complex scenarios: focusing on detection accuracy, a larger convolutional kernel and a lower learning rate are used, with a smaller batch size. Specific parameters are: convolutional kernel size set to 7×7, learning rate set to 0.0001, and batch size set to 8. The large 7×7 convolutional kernel can extract richer contextual features, which can improve the detection accuracy of occlusion and small target obstacles. The lower learning rate can make the model more stable during inference and reduce false detections. The smaller batch size can reduce the computational pressure and ensure that the model can run stably.

[0068] Based on image quality score Based on the classification results of scene complexity, the detection process is dynamically adjusted to optimize detection performance, as detailed below: when In simple scenarios, the image quality is high and the scene is simple, so there is no need for complex image enhancement processing. The enhancement processing steps of the adaptive image enhancement module can be skipped, and the preprocessed image can be directly transmitted to the improved semantic segmentation module, which not only reduces processing steps but also improves detection speed.

[0069] when In general scenarios, the image quality is good and the scene complexity is moderate. The conventional detection process is adopted, namely: from "multi-source image acquisition and synchronization" to "image preprocessing" and then to "adaptive image".

[0070] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0071] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A road obstacle detection method based on visual semantic segmentation, characterized in that, Includes the following steps: S1: Multi-source image acquisition and synchronization: Multi-view and multi-resolution image data of road scene are acquired synchronously through the left eye camera, right eye camera, forward-looking high-definition camera and infrared camera of the multi-source image sensing module, and the temporal consistency of multi-source images is ensured through the image synchronization control unit. S2: Image preprocessing: The original image acquired in S1 is subjected to image calibration, image alignment, noise removal, and image size standardization to eliminate image distortion and noise interference, thereby obtaining standardized image data; S3: Adaptive Image Enhancement: The preprocessed image is evaluated for quality, and an image enhancement strategy is adaptively selected based on the evaluation results to enhance the image in a targeted manner, thereby improving the image details and quality in obstacle areas; S4: Improved semantic segmentation: Employs a semantic segmentation network based on attention mechanism and multi-scale feature fusion to perform pixel-level semantic segmentation on the enhanced image and output segmentation results for different categories, including roads, obstacles and background; S5: Obstacle Recognition and Localization: Post-process the semantic segmentation results and extract the obstacle region. Then, construct the feature vector of the obstacle, identify the obstacle category through a classifier, and combine the depth information of the stereo image to realize the three-dimensional localization of the obstacle. S6: Dynamic Optimization and Result Output: Based on the complexity of the current scene and the image quality, dynamically adjust the detection parameters and process to optimize detection performance, and store and transmit the obstacle detection results to external devices.

2. The road obstacle detection method based on visual semantic segmentation according to claim 1, characterized in that: In S1, the image synchronization control unit achieves synchronous acquisition of the left eye camera, right eye camera, forward-looking high-definition camera and infrared camera through hardware triggering. Multi-source image acquisition also includes initial settings for camera parameters, including exposure time, gain, and white balance, which can be automatically adjusted according to the current lighting conditions.

3. The road obstacle detection method based on visual semantic segmentation according to claim 2, characterized in that: In S2, image calibration includes intrinsic parameter calibration and extrinsic parameter calibration, as detailed below: The intrinsic parameter calibration adopts the Zhang Zhengyou calibration method, which calculates the camera's intrinsic parameter matrix and distortion coefficient by taking multiple chessboard images, in order to eliminate lens distortion; External parameter calibration obtains the relative positional relationships between the left eye camera, right eye camera, forward-looking high-definition camera, and infrared camera through stereo calibration, and the relative positional relationships include rotation matrix and translation vector; Image alignment is achieved based on extrinsic calibration results, that is, by using a homography transformation algorithm to align multiple source images to the same coordinate system.

4. The road obstacle detection method based on visual semantic segmentation according to claim 3, characterized in that: In S2, noise removal is achieved using an adaptive median filtering algorithm, and the size of the filtering window can be dynamically adjusted according to the density of noise in the image. Image size normalization adjusts all preprocessed images to a uniform resolution, and bilinear interpolation is used for image scaling to ensure the integrity of image edge details.

5. The road obstacle detection method based on visual semantic segmentation according to claim 4, characterized in that: In S3, image quality is evaluated from four dimensions: brightness uniformity, contrast, sharpness, and signal-to-noise ratio, as detailed below: Brightness uniformity is evaluated by calculating the mean and variance of the image brightness; contrast is evaluated by calculating the dynamic range of the image grayscale values; sharpness is evaluated by calculating the edge strength of the image using the Laplacian operator; signal-to-noise ratio is evaluated by calculating the ratio of the image signal to noise; and then, based on the evaluation results of the four dimensions, a weighted summation method is used to obtain the image quality score.

6. The road obstacle detection method based on visual semantic segmentation according to claim 5, characterized in that: In S3, the selection rules for the adaptive enhancement strategy are as follows: When the image quality score and brightness uniformity score are less than the corresponding thresholds, a multi-scale enhancement algorithm based on Retinex theory is used. When the image quality score and contrast score are less than the corresponding thresholds, an adaptive histogram equalization algorithm is used. When the image quality score and sharpness score are less than the corresponding thresholds, a deep learning-based image deblurring algorithm is used. When the image quality score and signal-to-noise ratio score are less than the corresponding thresholds, an algorithm combining wavelet thresholding denoising and contrast enhancement is used. If the image quality score is greater than or equal to the corresponding threshold, no additional image enhancement processing will be performed.

7. The road obstacle detection method based on visual semantic segmentation according to claim 6, characterized in that: In S4, the improved semantic segmentation network includes an encoder, a decoder, and an attention fusion unit, as detailed below; The encoding end uses ResNet-50 as the backbone network, which extracts features through several convolutional blocks. Each convolutional block contains multiple convolutional layers, batch normalization layers, and ReLU activation functions. Finally, it downsamples through max pooling layers to output several feature maps of different scales. The decoding end adopts the symmetrical structure of U-Net, and performs several upsampling operations on the smallest scale feature map output by the encoding end. After each upsampling, it is concatenated with the feature map of the corresponding scale of the encoding end, and then the feature information is fused through convolutional layers to gradually improve the resolution of the feature map, and finally outputs a semantic segmentation result with the same size as the original image. The attention fusion unit is located at the feature concatenation point between the encoder and decoder. It includes a spatial attention module and a channel attention module. The spatial attention module focuses on the spatial region where the obstacle is located by calculating the spatial weight matrix of the feature map. The channel attention module can adaptively adjust the weights of different feature channels through the squeeze-excitation operation, thereby highlighting effective feature information.

8. The road obstacle detection method based on visual semantic segmentation according to claim 7, characterized in that: In S5, obstacle region extraction is achieved by performing connected component analysis on the semantic segmentation results, as follows: In the semantic segmentation results, obstacle category pixels are labeled as foreground and other category pixels are labeled as background. An eight-neighbor connected component detection algorithm is used to extract foreground connected components, and the area of ​​each connected component is calculated. By removing connected components with an area smaller than a preset threshold, the final obstacle region can be obtained. Feature extraction uses the SIFT algorithm to extract key feature points in the obstacle region and calculate a 128-dimensional feature descriptor for each feature point. Then, combined with the category prior information obtained from semantic segmentation, the feature descriptors can be weighted and optimized to construct the feature vector of the obstacle. Category recognition uses a support vector machine classifier, which is trained on a pre-labeled obstacle sample dataset. The training samples include multiple obstacle types, and cross-validation is used to optimize the classifier parameters during training. 3D localization combines stereo images acquired by the left and right cameras, calculates the depth map of the obstacle area using the SGBM stereo matching algorithm, and then converts the pixel coordinates of the obstacle area into world coordinates based on the camera's intrinsic and extrinsic parameters and depth information to obtain the obstacle's 3D position information. At the same time, it calculates the bounding box of the obstacle area to obtain the obstacle's size information.

9. The road obstacle detection method based on visual semantic segmentation according to claim 8, characterized in that: In S6, scene complexity assessment can be classified according to the number, density, occlusion degree of obstacles in the image and the type of road environment, and can be divided into simple scene, general scene and complex scene. The dynamically adjusted detection parameters include the kernel size, learning rate, and batch size of the improved semantic segmentation network, and the detection parameters can be adaptively adjusted according to simple, general, and complex scenarios. The process optimization is adjusted based on the results of image quality scores and scene complexity classifications. Specifically, when the image quality score is greater than or equal to the corresponding threshold and the scene is simple, the image enhancement process in S3 is skipped and semantic segmentation in S4 is performed directly. When the image quality score is less than the corresponding threshold and the scene is complex, multi-level image enhancement processing is performed in S3, followed by denoising, contrast enhancement and deblurring processing, and then semantic segmentation in S4.

10. A road obstacle detection system based on visual semantic segmentation, applied to the road obstacle detection method based on visual semantic segmentation as described in any one of claims 1-9, characterized in that, It includes a multi-source image sensing module, an image preprocessing module, an adaptive image enhancement module, an improved semantic segmentation module, an obstacle recognition and localization module, a dynamic optimization decision-making module, and a data storage and interaction module; Multi-source image sensing module: including left eye camera, right eye camera, forward-looking high-definition camera, infrared camera and image synchronization control unit, mainly used to acquire multi-view and multi-resolution image data of road scenes; Image preprocessing module: mainly used for calibration, alignment, noise removal, and size normalization of the original image; Adaptive Image Enhancement Module: Primarily used to assess the quality of preprocessed images and adaptively select enhancement strategies based on the assessment results; Improved semantic segmentation module: used for pixel-level semantic segmentation of the enhanced image; Obstacle recognition and localization module: used to extract obstacle regions, identify obstacle categories, and achieve three-dimensional localization; Dynamic optimization decision module: It can dynamically adjust detection parameters and processes based on scene complexity and image quality; Data storage and interaction module: Used to store various types of data during the detection process and to interact with external devices.

Images