Lighting adaptive road surface detection method and system considering small target region identification

Abstract

The application discloses a light self-adaptive road surface detection method and system considering small target area recognition, and relates to the technical field of intelligent sensing.The application comprises the following steps: receiving real-time image data, building a horizon detection network, inputting the image into the network to obtain an upper half area image containing environmental elements and a lower half area image containing a road surface and obstacles; using the upper half area image to judge the day and night state of the driving scene, so that the system automatically adjusts the model parameters according to the light condition; using the lower half area image to perform semantic segmentation, and intercepting small resolution blocks with road surface information according to the semantic mask; building a multi-block fusion decision network, inputting the blocks into the network, and the network can adaptively allocate the weight of each block feature and perform fusion decision on the prediction probability of each block.The application expands the road surface detection to all-weather scenes, has higher practical application value, and solves the recognition difficulty of small target road surface areas, so that the accuracy of road surface detection is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing technology, specifically to an illumination-adaptive road surface detection method and system that considers the identification of small target areas. Background Technology

[0002] A thorough understanding of road conditions is crucial for ensuring the driving safety of intelligent vehicles. In recent years, over 20,000 traffic accidents occur annually, with loss of tire traction being a common cause. Varying road conditions are a major factor contributing to changes in traction; adverse weather conditions such as rain and snow typically reduce the coefficient of friction between the tires and the road surface, weakening the tires' anti-skid ability and consequently reducing the vehicle's braking performance and stability. While many vehicles are equipped with advanced driver assistance systems (ADAS), such as anti-lock braking systems (ABS) and electronic stability control systems (ESC), these systems can only intervene when the vehicle is already in an unstable state (such as skidding or rollover), significantly increasing the risk of accidents. Active monitoring of road conditions can help active safety control systems adjust their control strategies in a timely manner, ensuring the vehicle remains stable and thus greatly reducing the risk of accidents.

[0003] With the development of intelligent vehicles, an increasing number of vehicles are equipped with sensors such as radar, lidar, infrared cameras, and visible light cameras. These sensors are typically used to identify lanes, potholes, pedestrians, vehicles, traffic lights, etc. In these applications, road surface type detection has become a popular and challenging task driving the planning and decision-making of intelligent vehicle driving behavior. Typical road surface conditions include dry, wet, and icy / snowy surfaces. The purpose of road surface type detection is to enable intelligent vehicles to promptly identify the slippery condition of the road surface and adjust their driving behavior accordingly. Using non-contact sensors to predict the road surface conditions ahead allows the vehicle to adjust its control strategy in advance without requiring specific excitation conditions, ensuring that the vehicle always maintains a stable state. Therefore, this indirect identification method has become one of the most practically valuable research directions in the field.

[0004] In existing technologies, visible light cameras stand out due to their low cost, small size, stable operation, and ability to be integrated with computer vision technology. Using deep learning technology to process visible light images has become the most common technique for solving road detection problems. However, existing methods based on visible light images have limited road information and more interference information in images with many obstacles. The typical texture and color features that image recognition relies on are not obvious. Therefore, it is difficult to effectively detect and recognize small target images where the road area accounts for a small proportion of the field of view. To address this, we propose an illumination-adaptive road detection method and system that considers the recognition of small target areas. Summary of the Invention

[0005] The purpose of this invention is to provide an illumination-adaptive road surface detection method that takes into account the identification of small target areas, so as to solve the problems mentioned in the background art.

[0006] According to a first aspect of the present invention, in order to achieve the above-mentioned objective, the present invention provides the following technical solution: an illumination-adaptive road surface detection method considering small target region identification, comprising the following steps:

[0007] Receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment region and the road surface region, and obtain the upper half image containing environment elements and the lower half image containing road surface, vehicles and pedestrians.

[0008] The day and night status of the driving scene is determined by using the HSV attributes of the upper half of the image containing environmental information, which makes it easier for the overall system to automatically adjust the subsequent model parameters according to the lighting conditions.

[0009] Based on the above-determined lighting conditions, load the corresponding model parameters, perform semantic segmentation on the road surface area image, and extract small-resolution image blocks with complete road surface information based on the obtained semantic mask.

[0010] A multi-patch fusion decision network is constructed. Based on the lighting conditions determined above, the corresponding model parameters are loaded. The captured image patches are input into the multi-patch fusion decision network. The multi-patch fusion decision network adaptively assigns weights to the features of each patch. Based on the assigned weights, the predicted probabilities of each image are weighted to obtain the fused recognition result.

[0011] Furthermore, real-time image data is received, a horizon detection network is built, and the image is input into the horizon detection network to separate the environment region and the road surface region, resulting in an upper half image containing environment elements and a lower half image containing the road surface, vehicles, and pedestrians, as detailed below:

[0012] (1) The horizon detection network uses ResNet-34 as the backbone network and adds two lightweight attention mechanisms, CBAM and ECA, in parallel to the horizon detection network structure;

[0013] (2) Input the image into the horizon detection network. The image is divided along the height axis. The horizons of different images are distributed in different height regions of the image. The position of the horizon is determined according to the distribution of objects in the image, and the image is divided into upper and lower parts according to the horizon.

[0014] Furthermore, the lightweight attention mechanism parallel CBAM includes two sub-modules, namely the channel attention module CAM and the spatial attention module SAM. The original serial CAM and SAM are modified into a parallel structure to improve accuracy while maintaining similar inference speed.

[0015] The feature representation of the output of the previous residual block in the horizon detection network is as follows: ,in The number of channels is a feature. and The optimized features have the same dimensions for their height and width, specifically:

[0016]

[0017] In the formula, Features of the input module For channel attention features, Spatial attention features and For the weights of the MLP, The number of channels in the input image. Scaling factor Indicates adoption Convolution operations with varying kernel sizes, This represents the Sigmoid function. The attention features of the final parallel lightweight attention mechanism CBAM.

[0018] Furthermore, the lightweight attention mechanism ECA is represented as follows:

[0019]

[0020] In the formula, It is the number of channels of the input feature. This indicates that the kernel size is... One-dimensional convolutional layer, and These are hyperparameters. Taking the absolute value and rounding down to the nearest odd number ensures that the convolution kernel size is odd. For the final ECA attention features, This represents the Sigmoid function.

[0021] Furthermore, the HSV attributes of the upper half of the image, which contains environmental information, are used to determine the day / night state of the driving scene, facilitating the overall system to automatically adjust subsequent model parameters according to lighting conditions, as detailed below:

[0022] (1) Convert the upper half of the image containing environmental information after horizon detection to the HSV color space, which contains three attributes: hue, saturation and value. Hue represents the type of color, saturation describes the gray level of a specific color, and value represents the brightness or intensity of a color.

[0023] (2) A tree ensemble model is constructed using gradient boosting decision tree, and the HSV attributes of the image are input into the ensemble model to regress the day and night state of the driving scene;

[0024] (21) The ensemble model uses the efficient gradient boosting algorithm XGBoost, and the objective function is expressed as:

[0025]

[0026] In the formula, The set of parameters representing the model. It is the training error, used to measure the predicted value. Compared with the true value The differences between them It is a regularization term used to control model complexity. It is the number of leaf nodes in the tree. It is the weight of the leaf node. and It is a hyperparameter used to adjust the strength of regularization;

[0027] (22) XGBoost uses a greedy algorithm to construct a new tree To minimize the objective function, assuming there is already a priori... The tree, the first The prediction of the number of trees is The model's predictions are updated as follows:

[0028]

[0029] In the formula, It is in the The updated predicted values, It was before The cumulative predicted value of trees, It is the first Tree pairs of input The predicted output;

[0030] (23) In order to construct the first XGBoost optimizes the second-order approximation of the objective function by performing a second-order Taylor expansion on the objective function:

[0031]

[0032] In the formula, and These are the first and second derivatives of the loss function with respect to the predicted values, respectively.

[0033] (24) Determine the optimal weight of each leaf node by minimizing the second-order approximation of the objective function:

[0034]

[0035] Therefore, the gain of the split node (i.e., the reduction in the objective function) can be expressed as:

[0036]

[0037] The greater the gain, the better the effect after splitting.

[0038] Furthermore, based on the lighting conditions determined above, the corresponding model parameters are loaded, and the road surface area image is semantically segmented. Small-resolution image patches with complete road surface information are then extracted based on the obtained semantic mask, as follows:

[0039] (1) Semantic segmentation: Construct a semantic segmentation network, take the lower half of the image after horizon detection as the input of the semantic segmentation network, load the corresponding model parameters according to the obtained day and night illumination results, and obtain the semantic mask of the road surface area;

[0040] (2) Crop image block: Find the maximum and minimum coordinates of the non-zero pixels in the mask, locate the approximate center point of the road surface area, and crop an image block with a size of [250, 200] centered on this point.

[0041] Furthermore, the semantic segmentation network uses the DeeplabV3+ model, which consists of an encoder and a decoder. The encoder part introduces dilated convolutions for cross-pixel feature extraction.

[0042] The DeeplabV3+ model uses ResNet-50 as the backbone network. After the image passes through the backbone network, one part is directly input to the decoder, and the other part is processed by parallel dilated convolution. The outputs of the two parts are then concatenated to obtain the final semantic segmentation mask.

[0043] Furthermore, a multi-tile fusion decision network is constructed. Based on the aforementioned lighting conditions, corresponding model parameters are loaded. The captured image patches are input into the multi-tile fusion decision network, which adaptively assigns weights to the features of each patch. The fused recognition result is obtained by weighting the predicted probabilities of each image according to the assigned weights, as detailed below:

[0044] (1) Construct a multi-tile fusion decision network, which includes a probability prediction network branch for tile feature extraction and probability prediction, and a confidence network branch for adaptive learning of the reliability of each tile.

[0045] Among them, the probability prediction network branch uses the EfficientNetV2 model, replacing the SE attention mechanism in the original network structure with the parallel lightweight attention mechanism CBAM.

[0046] The confidence network branch consists of a max pooling layer, a 1×1 convolutional layer, a style pooling and style ensemble module, and the last two convolutional layers.

[0047] (2) Input each image patch into the multi-patch fusion decision network, load the corresponding network parameters according to the obtained day and night lighting conditions, the multi-patch fusion decision network adaptively allocates the weights of each patch feature, and obtains the fused recognition result by weighting the prediction probability of each image according to the allocated weights.

[0048] (21) The resolution of the input image patch is kept in [250, 200] without scaling, and the predicted probability of each image patch is obtained: , , …;

[0049] (22) The high-resolution shallow features extracted from EfficientNetV2 are used as input to the confidence network branch to learn the weights assigned to the prediction results of each image patch.

[0050] (23) The confidence network branch will generate shallow feature maps in the probability prediction network branch. Mapped to a probability array This array is used to determine the contribution of each image patch to the final classification result, where , , , and These represent the training batch size, the number of channels in the feature map, the height, the width, and the number of image patches, respectively.

[0051] (24) Feature map First, a max pooling layer and a 1×1 convolutional layer are used to compress the size and number of channels, resulting in... For each image patch, the focus is primarily on spatial features. Next, the style pooling and style integration modules in the network structure... Extract style information and integrate it to obtain... ,in This represents the number of pooling types used; where aggregated features of image mean, maximum, and standard deviation are used for style pooling.

[0052] (25) Then, the style information of all image patches is concatenated along the channel dimension to obtain The input is a two-layer convolutional neural network used to estimate the recalibration weights for each channel. After normalization, the result is... This is used to enable the network to dynamically estimate the relative importance of each style information and adaptively adjust the weights of the prediction probabilities of each image patch. The above process can be expressed as:

[0053]

[0054] Learned weights This is used to measure the confidence level of each image patch. The network then weights and fuses the predicted probabilities of the image patches based on different confidence levels.

[0055]

[0056] In the formula, Represents different types of road surfaces.

[0057] According to a second aspect of the present invention, the present invention provides an illumination-adaptive road surface detection system considering small target region identification, for implementing the above-described illumination-adaptive road surface detection method considering small target region identification, comprising:

[0058] The information separation module is used to receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road surface area, and obtain the upper half image containing environmental elements and the lower half image containing the road surface, vehicles and pedestrians.

[0059] The illumination decision module uses the HSV attributes of the upper half of the image, which contains environmental information, to determine the day and night state of the driving scene, so that the overall system can automatically adjust the subsequent model parameters according to the illumination conditions.

[0060] The region extraction module loads the corresponding model parameters based on the lighting conditions determined above, performs semantic segmentation on the road surface region image, and extracts small-resolution image blocks with complete road surface information based on the obtained semantic mask.

[0061] The fusion recognition module is used to build a multi-patch fusion decision network. Based on the lighting conditions determined above, the corresponding model parameters are loaded, and the captured image patches are input into the multi-patch fusion decision network. The multi-patch fusion decision network adaptively allocates the weights of each patch feature, and the predicted probabilities of each image are weighted according to the allocated weights to obtain the fused recognition result.

[0062] According to a third aspect of the present invention, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores the computer program capable of running on the processor, and the processor loads and executes the computer program, employing the above-described illumination-adaptive road surface detection method considering small target area identification.

[0063] The present invention has at least the following beneficial effects:

[0064] 1. This invention is based on the horizon detection network to segment environmental information and road surface information, and uses road surface information and environmental information for independent detection branches, thereby avoiding information redundancy and mutual interference, and improving the accuracy of road surface area detection.

[0065] 2. This invention proposes an adaptive illumination adjustment method that can adaptively adjust model parameters according to different illumination conditions to achieve all-weather road surface type detection.

[0066] 3. This invention takes into account the small proportion of road surface areas in the field of view and uses a multi-patch result fusion decision method. It proposes a multi-patch fusion decision network (MFDNet), which can suppress the prediction results of low-confidence patches and amplify the prediction results of high-confidence patches, so that the final detection results have good robustness and are more in line with actual driving scenarios.

[0067] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0068] Figure 1 This is a schematic flowchart of the detection method described in this invention;

[0069] Figure 2 This is a schematic diagram of the detection method described in this invention.

[0070] Figure 3 This is a schematic diagram of the horizon detection network in this invention;

[0071] Figure 4 This is a structural diagram of the parallel CBAM attention mechanism in the horizon detection network of this invention;

[0072] Figure 5 This is a structural diagram of the ECA attention mechanism in the horizon detection network of this invention;

[0073] Figure 6 This is a schematic diagram of an embodiment of the XGBoost tree structure in this invention;

[0074] Figure 7 This is a schematic diagram of the road surface area image block cropping method provided by the present invention;

[0075] Figure 8 This is a network structure diagram of the Multi-Patch Fusion Decision Network (MFDNet) proposed in this invention;

[0076] Figure 9 This is a schematic diagram illustrating the experimental results of the present invention in road surface type detection under large target images and small target images, using the non-fusion decision method and the soft voting decision method. Detailed Implementation

[0077] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0078] Example 1:

[0079] Please see Figures 1-9 The present invention provides a technical solution: an illumination-adaptive road surface detection method considering small target area identification, comprising the following steps:

[0080] S1. Information Separation: Receive real-time image data, construct a horizon detection network, input the image into the horizon detection network to separate the environment region from the road surface region, obtaining an upper half image containing environment elements and a lower half image containing the road surface, vehicles, and pedestrians, such as... Figure 3 As shown, the details are as follows:

[0081] Real-time images are acquired from vehicle-mounted cameras, transforming the regression problem into a classification problem, and the results are output using a simple classification network.

[0082] (S11) A horizon detection network is built using ResNet-34 as the backbone network. Two lightweight attention modules, CBAM and ECA, are added to the horizon detection network in parallel to improve the network performance.

[0083] (S11.1) As Figure 4 The parallel CBAM module shown includes two sub-modules: CAM (Channel Attention Module) and SAM (Spatial Attention Module). The originally sequential CAM and SAM are modified into a parallel structure, slightly improving accuracy while maintaining similar inference speed. The feature representation of the output of the previous residual block in the network is as follows: ,in The number of channels is a feature. and Given the height and width of a feature, the optimized feature has the same size and can be represented as:

[0084]

[0085] In the formula, Features of the input module For channel attention features, Spatial attention features and For the weights of the MLP, The number of channels in the input image. Scaling factor Indicates adoption Convolution operations with varying kernel sizes, This represents the Sigmoid function. For the final parallel CBAM attention features;

[0086] (S11.2) As Figure 5 As shown, the lightweight attention mechanism ECA is represented as follows:

[0087]

[0088] In the formula, It is the number of channels of the input feature. This indicates that the kernel size is... One-dimensional convolutional layer, and These are hyperparameters. Taking the absolute value and rounding down to the nearest odd number ensures that the convolution kernel size is odd. This is the final ECA attention feature;

[0089] It should be noted that the Horizon Detection Network inserts parallel CBAM modules in the first thirteen residual blocks and ECA modules in the last three residual blocks to achieve optimal performance. The improved network structure is called Multi-Attention ResNet (MA-ResNet).

[0090] (S12) Input the image into the MA-ResNet network. The image is divided along the height axis. The horizon of different images is distributed in different height regions of the image. The position of the horizon of the image can be approximately determined according to the distribution of objects in the image, so that the image can be simply divided into upper and lower parts according to the horizon.

[0091] Because most of the cameras in the dataset are positioned at the driver's viewpoint, there is a significant imbalance in the dataset. In most images, the horizon is located in the center, making the network prone to overfitting on a large number of samples and underfitting on a small number. Therefore, a dynamically scaled cross-entropy loss (FocalLoss) is needed as the loss function. This loss function uses a dynamic focusing factor to reduce the weight of easily distinguishable samples during training and quickly shift the training focus to difficult-to-distinguish samples. The formula for the loss function is as follows:

[0092]

[0093] In the formula, The total number of samples, It is the total number of categories. It is a category The balance factor It is a focusing factor. It is a sample In category The real labels on It is a sample Category The predicted probability; as As the number of distinguishable samples increases, the loss contribution of easily distinguishable samples decreases significantly, while the impact of difficult-to-distinguishable samples on network training is amplified;

[0094] S2. Illumination Decision: The day / night state of the driving scene is determined using the HSV attributes of the upper half of the image, which contains environmental information. This allows the overall system to automatically adjust subsequent model parameters based on illumination conditions, as detailed below:

[0095] (S21) Convert the upper half of the image after horizon detection to the HSV color space, which contains three attributes: hue, saturation and value. Hue represents the type of color, saturation describes the gray level of a specific color, and value represents the brightness or intensity of a color.

[0096] (S22) combination Figure 6 Gradient Boosting Decision Tree (GBDT) is used to construct a tree ensemble model, and the HSV attributes of the image are input into the ensemble model to regress the day and night state of the driving scene.

[0097] (S22.1) The ensemble model uses the efficient gradient boosting algorithm XGBoost, and the objective function can be expressed as:

[0098]

[0099] In the formula, The set of parameters representing the model. It is the training error, used to measure the predicted value. Compared with the true value The differences between them; It is a regularization term used to control model complexity; It is the number of leaf nodes in the tree. It is the weight of the leaf node. and It is a hyperparameter used to adjust the strength of regularization;

[0100] (S22.2) At each step, XGBoost uses a greedy algorithm to construct a new tree. To minimize the objective function, assuming there is already a priori... The tree, the first The prediction of the number of trees is The model's predictions are updated as follows:

[0101]

[0102] In the formula, It is in the The updated predicted values, It was before The cumulative predicted value of trees, It is the first Tree pairs of input The predicted output;

[0103] (S22.3) In order to construct the first XGBoost optimizes the second-order approximation of the objective function by performing a second-order Taylor expansion on the objective function:

[0104]

[0105] In the formula, and These are the first and second derivatives of the loss function with respect to the predicted values, respectively.

[0106] (S22.4) The optimal weight for each leaf node is determined by minimizing the second-order approximation of the objective function:

[0107]

[0108] Therefore, the gain of the split node (i.e., the reduction in the objective function) can be expressed as:

[0109]

[0110] This gain formula is used to evaluate whether it is worthwhile to split at this node; the greater the gain, the better the effect after splitting.

[0111] Specifically, XGBoost effectively prevents overfitting and improves the model's generalization ability by iteratively optimizing the objective function, achieving an accuracy of over 98.5% when distinguishing between day and night scenes in images.

[0112] In the subsequent steps, the model parameters are trained for different day and night lighting conditions at each stage, and the corresponding model parameters are loaded based on the lighting decision results here;

[0113] S3. Region Extraction: Based on the lighting conditions determined above, the corresponding model parameters are loaded, and semantic segmentation is performed on the road surface region image. Small-resolution image patches with complete road surface information are then extracted based on the obtained semantic mask, as detailed below:

[0114] (S31) Semantic segmentation: Construct a semantic segmentation network, take the lower half of the image after horizon detection as the input of the semantic segmentation network, load the corresponding model parameters according to the above illumination decision results, and obtain the semantic mask of the road surface area;

[0115] (S31.1) The semantic segmentation network uses the DeeplabV3+ model, which consists of two parts: an encoder and a decoder. The encoder part introduces dilated convolution to extract features across pixels. Dilated convolution increases the receptive field without losing information, so that each convolution output can contain a wider range of information.

[0116] The DeeplabV3+ model uses ResNet-50 as the backbone network. After the image passes through the backbone network, one part is directly input to the decoder, while the other part undergoes parallel dilated convolution processing. The results of these two branches are concatenated to obtain the final semantic segmentation mask.

[0117] (S31.2) The loss function adopts the pixel-level cross-entropy loss function, which is expressed as:

[0118]

[0119] In the formula, The total number of pixels in the image. The total number of categories, Is with category Related weights, It is a binary indicator, if the pixel If the classification is correct, then It is 1 if it is 1, otherwise it is 0. It is a pixel Predicted as category The probability of;

[0120] (S32) Cropping patches: When the road surface area occupies a very small proportion of the image, these small target images contain a lot of irrelevant information. In the classification task, directly adjusting the size of the small target images and inputting them into the network will significantly affect the accuracy of classification. In order to improve the accuracy of road surface recognition in small target images, small resolution image patches are selected according to the pixel distribution of the mask to focus on the road surface information.

[0121] Combination Figure 7 The road surface area in the mask can be regarded as an irregular shape. By finding the maximum and minimum coordinates of the non-zero pixels, the approximate center point of the road surface area is located, and an image block of size [250, 200] is cropped with this point as the center.

[0122] S4. Fusion Recognition: A multi-tile fusion decision network is constructed. Based on the lighting conditions determined above, the corresponding model parameters are loaded. The captured image patches are input into the multi-tile fusion decision network, which adaptively assigns weights to the features of each patch. The fused recognition result is obtained by weighting the predicted probabilities of each image according to the assigned weights, as detailed below:

[0123] (S41) combination Figure 8 A multi-patch fusion decision network (MFDNet) is constructed, which mainly consists of two parts: a probability prediction network branch for patch feature extraction and probability prediction, and a confidence network branch for adaptive learning of the reliability of each patch; each image patch is input into the multi-patch fusion decision network, and the corresponding network parameters are loaded according to the illumination decision results mentioned above;

[0124] (S41.1) The probabilistic prediction network branch uses the EfficientNetV2 model, replacing the SE attention module in the original network structure with the aforementioned parallel CBAM module to improve classification accuracy. The resolution of the input image patch is kept in [250, 200] without scaling, and the predicted probability of each image patch is obtained: , , …; The loss function used is the cross-entropy loss function, expressed as:

[0125]

[0126] In the formula, It is the total number of samples. It is the total number of categories. It is a sample In category The real labels on It is a pixel Predicted as category The probability of;

[0127] High-resolution shallow features extracted from EfficientNetV2 are used as input to the confidence network branch to learn the weights assigned to the prediction results for each image patch; the confidence network uses the shallow feature maps from the probabilistic prediction network. Mapping to a probability array This array is used to determine the contribution of each image patch to the final classification result. , , , and These represent the training batch size, the number of channels in the feature map, the height, the width, and the number of image patches, respectively.

[0128] Feature map First, a max pooling layer and a 1×1 convolutional layer are used to compress the size and number of channels, resulting in... For each image patch, the focus is primarily on spatial features; next, the style pooling and style integration modules in the network structure... Extract style information and integrate it to obtain... ,in This represents the number of pooling types used; where aggregated features of the image mean, maximum, and standard deviation are used for style pooling; subsequently, the style information of all image patches is concatenated along the channel dimension to obtain... The input is a two-layer convolutional neural network used to estimate the recalibration weights for each channel. After normalization, the result is... This enables the network to dynamically estimate the relative importance of each style information and adaptively adjust the weights of the prediction probabilities for each image patch. This process can be expressed as:

[0129]

[0130] Learned weights This is used to measure the confidence level of each image patch. The network then weights and fuses the predicted probabilities of the image patches based on different confidence levels.

[0131]

[0132] In the formula, Representing different types of road surfaces;

[0133] In this model, the prediction results of low-confidence patches are suppressed, while the prediction results of high-confidence patches are amplified, ensuring that the final result is mainly derived from the contributions of relatively reliable patches.

[0134] The present invention will be further described below with reference to specific embodiments:

[0135] During model training, in order to avoid the sample bias that may exist in a single dataset and the limitations of driving scenarios, the experiment integrated images from the BDD100K, Oxford Robotcar and CADC datasets collected by vehicle cameras to create a three-class dataset covering daytime and nighttime driving scenarios. This dataset was used to identify dry, wet and snowy road surfaces under various lighting conditions, thereby training an all-weather detection model with higher recognition accuracy and better robustness.

[0136] Figure 9This diagram illustrates the experimental results of the method described in this embodiment in road type detection, comparing it with methods using non-fusion decision-making and soft-voting decision-making, under both large and small target images. When the road surface area in the image is large and clearly identifiable, road detection is relatively simple, and therefore the advantages of the method described in this embodiment are not very obvious. However, when the road surface area is small and difficult to identify, a large amount of interference information mixed in the image may affect the recognition effect. In this case, the method described in this embodiment has a significant advantage. Although such small target images account for a small proportion of the dataset, they are a significant factor affecting driving safety in actual driving scenarios. In this case, the method described in this embodiment can effectively suppress these interference areas in the image and amplify the prediction results of reliable areas, thereby making the final detection results more accurate.

[0137] In summary, this embodiment can adaptively adjust model parameters according to different lighting conditions to achieve all-weather road type detection. It also considers the case where the road area occupies a small proportion of the field of view, ensuring that the network can always focus on the road area even in high-resolution images. At the same time, it ensures that images with a large road area have a sufficient receptive field. To this end, a multi-patch result fusion decision method is used, and a multi-patch fusion decision network (MFDNet) is proposed. This network can suppress the prediction results of low-confidence patches and amplify the prediction results of high-confidence patches, so that the final detection results have good robustness and are more in line with actual driving scenarios.

[0138] Example 2:

[0139] This embodiment provides an illumination-adaptive road surface detection system that considers small target region identification, used to implement the above-mentioned illumination-adaptive road surface detection method that considers small target region identification, including:

[0140] The information separation module is used to receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road surface area, and obtain the upper half image containing environmental elements and the lower half image containing the road surface, vehicles and pedestrians.

[0141] The illumination decision module uses the HSV attributes of the upper half of the image, which contains environmental information, to determine the day / night state of the driving scene, facilitating the overall system to automatically adjust subsequent model parameters according to illumination conditions;

[0142] The region extraction module loads the corresponding model parameters based on the aforementioned illumination decision, performs semantic segmentation on the road surface region image, and extracts small-resolution image blocks with complete road surface information based on the obtained semantic mask.

[0143] The fusion recognition module is used to build a multi-patch fusion decision network. Based on the aforementioned illumination decision results, it loads the corresponding model parameters, inputs the captured image patches into the multi-patch fusion decision network, and the multi-patch fusion decision network adaptively allocates the weights of each patch feature. Based on the allocated weights, it weights the predicted probabilities of each image to obtain the fused recognition result.

[0144] Specifically, the aforementioned information separation module, illumination decision module, region extraction module, and fusion recognition module can be embedded into a computer processing system. The computer, based on the illumination-adaptive road surface detection method considering small target region recognition provided above, calls each of the above modules to complete the task of all-weather road surface type detection. The aforementioned information separation module, illumination decision module, region extraction module, and fusion recognition module can perform operations according to the specific steps given in the illumination-adaptive road surface detection method considering small target region recognition.

[0145] It should be noted that the division of the various modules in the above system is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the information separation module can be a separate processing element or integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and its functions can be called and executed by a processing element of the device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0146] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0147] Example 3:

[0148] The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it employs the above-mentioned illumination-adaptive road surface detection method that considers the identification of small target areas.

[0149] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server, and the terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0150] Furthermore, the processor can be a central processing unit (CPU). Of course, depending on the actual use, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be used. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.

[0151] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0152] For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on," "mounted on," "fixed to," or "set on" another element, it may be directly on the other element or there may be an intermediate element present. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible embodiments.

[0153] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0154] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

Claims

1. An illumination-adaptive road surface detection method considering small target region identification, characterized in that, Includes the following steps: Real-time image data is received, a horizon detection network is built, and the image is input into the horizon detection network to separate the environment region and the road surface region, resulting in an upper half image containing environment elements and a lower half image containing the road surface, vehicles, and pedestrians, as detailed below: The Horizon Detection Network uses ResNet-34 as its backbone network and adds two lightweight attention mechanisms, CBAM and ECA, in parallel to the Horizon Detection Network structure. The image is input into the horizon detection network. The image is divided along the height axis. The horizon of different images is distributed in different height regions of the image. The position of the horizon of the image is determined according to the distribution of objects in the image, and the image is divided into upper and lower parts according to the horizon. The day and night status of the driving scene is determined by using the HSV attributes of the upper half of the image containing environmental information, which makes it easier for the overall system to automatically adjust the subsequent model parameters according to the lighting conditions. Based on the above-determined lighting conditions, load the corresponding model parameters, perform semantic segmentation on the road surface area image, and extract small-resolution image blocks with complete road surface information based on the obtained semantic mask. A multi-patch fusion decision network is constructed. Based on the aforementioned lighting conditions, corresponding model parameters are loaded. The captured image patches are input into the multi-patch fusion decision network, which adaptively assigns weights to the features of each patch. The fused recognition result is obtained by weighting the predicted probabilities of each image according to the assigned weights, as detailed below: (1) Construct a multi-tile fusion decision network, which includes a probability prediction network branch for tile feature extraction and probability prediction, and a confidence network branch for adaptive learning of the reliability of each tile. Among them, the probability prediction network branch uses the EfficientNetV2 model, replacing the SE attention mechanism in the original network structure with the parallel lightweight attention mechanism CBAM. The confidence network branch consists of a max pooling layer, a 1×1 convolutional layer, a style pooling and style ensemble module, and the last two convolutional layers. (2) Input each image patch into the multi-patch fusion decision network, load the corresponding network parameters according to the obtained day and night lighting conditions, the multi-patch fusion decision network adaptively allocates the weights of each patch feature, and obtains the fused recognition result by weighting the prediction probability of each image according to the allocated weights. (21) The resolution of the input image patch is kept in [250, 200] without scaling. The predicted probability of each image patch is obtained in the probability prediction branch: , , …; (22) The high-resolution shallow features extracted from EfficientNetV2 are used as input to the confidence network branch to learn the weights assigned to the prediction results of each image patch. (23) The confidence network branch will generate shallow feature maps in the probability prediction network branch. Mapping to a probability array This array is used to determine the contribution of each image patch to the final classification result, where , , , and These represent the training batch size, the number of channels in the feature map, the height, the width, and the number of image patches, respectively. (24) Feature map First, a max pooling layer and a 1×1 convolutional layer are used to compress the size and number of channels, resulting in... For each image patch, the focus is primarily on spatial features. Next, the style pooling and style integration modules in the network structure... Extract style information and integrate it to obtain... ,in This represents the number of pooling types used; where aggregated features of image mean, maximum, and standard deviation are used for style pooling. (25) Then, the style information of all image patches is concatenated along the channel dimension to obtain The input is a two-layer convolutional neural network used to estimate the recalibration weights for each channel. After normalization, the result is... This is used to enable the network to dynamically estimate the relative importance of each style information and adaptively adjust the weights of the prediction probabilities of each image patch. The above process can be expressed as: Learned weights This is used to measure the confidence level of each image patch. The network then weights and fuses the predicted probabilities of the image patches based on different confidence levels. In the formula, Represents different types of road surfaces.

2. The illumination-adaptive road surface detection method considering small target region identification according to claim 1, characterized in that: The lightweight attention mechanism parallel CBAM includes two sub-modules: the channel attention module CAM and the spatial attention module SAM. The original serial CAM and SAM are modified into a parallel structure to improve accuracy while maintaining similar inference speed. The feature representation of the output of the previous residual block in the horizon detection network is as follows: ,in The number of channels is a feature. and The optimized features have the same dimensions for their height and width, specifically: In the formula, Features of the input module For channel attention features, Spatial attention features and For the weights of the MLP, The number of channels in the input image. Scaling factor Indicates adoption Convolution operations with varying kernel sizes, This represents the Sigmoid function. Attention features for the final parallel lightweight attention mechanism, Parallel CBAM.

3. The illumination-adaptive road surface detection method considering small target region identification according to claim 2, characterized in that: The lightweight attention mechanism ECA is represented as follows: In the formula, It is the number of channels of the input feature. This indicates that the kernel size is... One-dimensional convolutional layer, and These are hyperparameters. Taking the absolute value and rounding down to the nearest odd number ensures that the convolution kernel size is odd. For the final ECA attention features, This represents the Sigmoid function.

4. The illumination-adaptive road surface detection method considering small target region identification according to claim 3, characterized in that, The system utilizes the HSV attributes of the upper half of the image, which contains environmental information, to determine the day / night state of the driving scene. This facilitates the overall system in automatically adjusting subsequent model parameters based on lighting conditions, as detailed below: (1) Convert the upper half of the image containing environmental information after horizon detection to the HSV color space, which contains three attributes: hue, saturation and lightness. Hue represents the type of color, saturation describes the gray level of a specific color, and lightness represents the brightness or intensity of a color. (2) A tree ensemble model is constructed using gradient boosting decision tree, and the HSV attributes of the image are input into the ensemble model to regress the day and night state of the driving scene; (21) The ensemble model uses the efficient gradient boosting algorithm XGBoost, and the objective function is expressed as: In the formula, The set of parameters representing the model. It is the training error, used to measure the predicted value. Compared with the true value The differences between them It is a regularization term used to control model complexity. It is the number of leaf nodes in the tree. It is the weight of the leaf node. and It is a hyperparameter used to adjust the regularization strength; (22) XGBoost uses a greedy algorithm to construct a new tree To minimize the objective function, assuming there is already a priori... The tree, the first The prediction of the number of trees is The model's predictions are updated as follows: In the formula, It is in the The updated predicted values, It was before The cumulative predicted value of the trees, It is the first Tree pairs of input The predicted output; (23) In order to construct the first XGBoost optimizes the second-order approximation of the objective function by performing a second-order Taylor expansion on the objective function: In the formula, and These are the first and second derivatives of the loss function with respect to the predicted values, respectively. (24) Determine the optimal weight of each leaf node by minimizing the second-order approximation of the objective function: Therefore, the gain of the split node can be expressed as: The greater the gain, the better the effect after splitting.

5. The illumination-adaptive road surface detection method considering small target region identification according to claim 4, characterized in that, Based on the lighting conditions determined above, the corresponding model parameters are loaded, and semantic segmentation is performed on the road surface image. Small-resolution image patches with complete road surface information are then extracted based on the obtained semantic mask, as detailed below: (1) Semantic segmentation: Construct a semantic segmentation network, take the lower half of the image after horizon detection as the input of the semantic segmentation network, load the corresponding model parameters according to the illumination decision result, and obtain the semantic mask of the road surface area; (2) Crop image block: Find the maximum and minimum coordinates of the non-zero pixels in the mask, locate the approximate center point of the road surface area, and crop an image block with a size of [250, 200] centered on this point.

6. The illumination-adaptive road surface detection method considering small target region identification according to claim 5, characterized in that, The semantic segmentation network uses the DeeplabV3+ model, which consists of an encoder and a decoder. The encoder part introduces dilated convolution to extract features across pixels. The DeeplabV3+ model uses ResNet-50 as the backbone network. After the image passes through the backbone network, one part is directly input to the decoder, and the other part is processed by parallel dilated convolution. The outputs of the two parts are then concatenated to obtain the final semantic segmentation mask.

7. An illumination-adaptive road surface detection system considering small target region identification, used to implement the illumination-adaptive road surface detection method considering small target region identification as described in any one of claims 1 to 6, characterized in that, include: The information separation module is used to receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road surface area, and obtain the upper half image containing environmental elements and the lower half image containing the road surface, vehicles and pedestrians. The illumination decision module uses the HSV attributes of the upper half of the image, which contains environmental information, to determine the day and night state of the driving scene, so that the overall system can automatically adjust the subsequent model parameters according to the illumination conditions. The region extraction module loads the corresponding model parameters based on the lighting conditions determined above, performs semantic segmentation on the road surface region image, and extracts small-resolution image blocks with complete road surface information based on the obtained semantic mask. The fusion recognition module is used to build a multi-patch fusion decision network. Based on the lighting conditions determined above, the corresponding model parameters are loaded, and the captured image patches are input into the multi-patch fusion decision network. The multi-patch fusion decision network adaptively allocates the weights of each patch feature, and the predicted probabilities of each image are weighted according to the allocated weights to obtain the fused recognition result.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs the illumination-adaptive road surface detection method considering small target area identification as described in any one of claims 1 to 6.

Images