Illumination adaptive road surface detection method and system considering small target area recognition

Abstract

The invention discloses an illumination adaptive road surface detection method and system considering small target area identification, and relates to the technical field of intelligent perception. The method comprises the following steps: receiving real-time image data, building a horizon line detection network, and inputting the image into the network to obtain an upper half region image containing environmental elements and a lower half region image containing a road surface and obstacles; the day and night state of the driving scene is judged by using the upper half area image, so that the system automatically adjusts model parameters according to illumination conditions; performing semantic segmentation by using the lower half region image, and intercepting a small-resolution image block with road surface information amount according to a semantic mask; a multi-block fusion decision network is built, blocks are input into the network, and the network can adaptively distribute weights of features of the blocks and perform fusion decision on prediction probabilities of the blocks. According to the invention, the road surface detection is expanded to an all-weather scene, the method has higher practical application value, the identification difficulty of a small target road surface area is overcome, and the accuracy of road surface detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent sensing technology, and in particular to a method and system for adaptive illumination road surface detection that considers small target area recognition. Background Art

[0002] A deep understanding of road conditions is essential to ensure the driving safety of smart vehicles. In recent years, there are more than 20,000 vehicle traffic accidents each year, and loss of tire traction is one of the common causes. Variable road conditions are the main factor leading to changes in traction. Bad weather such as rainy and snowy days usually reduces the adhesion coefficient between the tire and the road, weakens the tire's anti-skid ability, and thus reduces the vehicle's braking performance and stability. Most current vehicles are equipped with advanced driver assistance systems (ADAS), such as anti-lock braking systems (ABS) and electronic stability control systems (ESC). However, these systems can only intervene when the vehicle is already in an unstable state (such as skidding or overturning), which significantly increases the risk of accidents. Active monitoring of road conditions can help active safety control systems adjust control strategies in a timely manner to ensure that the vehicle always remains stable, thereby greatly reducing the risk of accidents.

[0003] With the development of smart cars, more and more vehicles are equipped with sensors such as radar, lidar, infrared cameras and visible light cameras. These sensors are usually used to identify lanes, potholes, pedestrians, vehicles, traffic lights, etc. In these applications, the detection of road surface type has become a popular and challenging task to promote the planning and decision-making of intelligent vehicle driving behavior. Typical road conditions include dry, slippery and icy roads. The purpose of road surface type detection is to enable smart cars to identify the slippery state of the road in time and adjust driving behavior accordingly. Using non-contact sensors to predict the road conditions ahead enables the vehicle to adjust its control strategy in advance, and does not require specific incentive conditions, which can ensure that the vehicle always remains in a stable state. Therefore, this indirect identification method has become the most practical research direction in the field.

[0004] Among the existing technologies, visible light cameras stand out due to their low cost, small size, stable operation and ability to be integrated with computer vision technology. Using deep learning technology to process visible light images has become the most common technology for solving road surface detection problems. However, in the case of many obstacles, the existing methods based on visible light images have less road surface information and more interference information in the pictures, and the typical texture and color features that image recognition relies on are not obvious. Therefore, it is difficult to effectively detect and identify small target images whose road surface area accounts for a small proportion of the field of view. To this end, we propose a light adaptive road surface detection method and system that considers the recognition of small target areas. Summary of the invention

[0005] The purpose of the present invention is to provide an illumination adaptive road surface detection method considering small target area recognition, so as to solve the problems raised in the above background technology.

[0006] According to a first aspect of the present invention, in order to achieve the above-mentioned purpose, the present invention provides the following technical solution: a method for adaptive illumination road surface detection considering small target area recognition, comprising the following steps:

[0007] Receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road area, and obtain the upper half of the image containing environmental elements and the lower half of the image containing the road surface, vehicles and pedestrians;

[0008] The HSV attributes of the upper half of the image containing environmental information are used to determine the day and night state of the driving scene, so that the overall system can automatically adjust subsequent model parameters according to lighting conditions;

[0009] According to the above-determined illumination conditions, the corresponding model parameters are loaded, the road surface area image is semantically segmented, and a small-resolution image block with complete road surface information is intercepted according to the obtained semantic mask;

[0010] Build a multi-block fusion decision network, load the corresponding model parameters according to the above-mentioned lighting conditions, input the captured image blocks into the multi-block fusion decision network, and the multi-block fusion decision network adaptively assigns the weights of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result.

[0011] Furthermore, real-time image data is received, a horizon detection network is built, and the image is input into the horizon detection network to separate the environment area and the road area, thereby obtaining the upper half of the image containing the environment elements and the lower half of the image containing the road surface, vehicles, and pedestrians, as follows:

[0012] (1) The horizon detection network uses ResNet-34 as the backbone network, and adds two lightweight attention mechanisms, CBAM and ECA, to the horizon detection network structure in parallel;

[0013] (2) The image is input into the horizon detection network. The image is divided along the height axis. The horizons of different images are distributed in different height areas of the image. The position of the image horizon is determined according to the distribution of objects in the image, and the image is divided into upper and lower parts according to the horizon.

[0014] Furthermore, the lightweight attention mechanism parallel CBAM includes two submodules, namely the channel attention module CAM and the spatial attention module SAM, which modify the original serial CAM and SAM into a parallel structure to improve the accuracy while maintaining similar reasoning speed;

[0015] The feature representation of the output of the previous residual block in the horizon detection network is Where C is the number of channels of the feature, H and W are the height and width of the feature, and the optimized features have the same size, which is specifically expressed as:

[0016]

[0017] M P_CBAM (F)= σ(M c (F)+ M s (F)) (3)

[0018] Where F is the characteristic of the input module, M c is the channel attention feature, M s is the spatial attention feature, and is the weight of MLP, C is the number of channels of the input image, r is the scaling factor, and f 7×7 represents the convolution operation with a 7×7 convolution kernel size, σ represents the Sigmoid function, and M P_CBAM The attention features of the final parallel lightweight attention mechanism CBAM.

[0019] Furthermore, the lightweight attention mechanism ECA is expressed as:

[0020]

[0021] In the formula, C is the number of channels of the input feature, f k×k represents a one-dimensional convolutional layer with a kernel size of k. Υ and b are hyperparameters. The absolute values ​​are taken and rounded down to the nearest odd number to ensure that the kernel size is an odd number. M ECA is the final ECA attention feature, and σ represents the Sigmoid function.

[0022] Furthermore, the HSV attributes of the upper half of the image containing environmental information are used to determine the day and night state of the driving scene, so that the overall system can automatically adjust the subsequent model parameters according to the lighting conditions, as follows:

[0023] (1) Convert the upper half of the image containing environmental information after horizon detection into the HSV color space, which contains three attributes: hue, saturation, and value. Hue indicates the type of color, saturation describes the grayscale of a specific color, and value indicates the brightness or intensity of the color.

[0024] (2) A gradient boosting decision tree is used to build a tree ensemble model, and the HSV attributes of the image are input into the ensemble model to regress the day and night state of the driving scene;

[0025] (21) The integrated model uses the efficient gradient boosting algorithm XGBoost, and the objective function is expressed as:

[0026]

[0027] In the formula, Represents the set of parameters of the model, is the training error, which is used to measure the predicted value and the true value y i The difference between Ω(f k ) is a regularization term used to control model complexity, T is the number of leaf nodes in the tree, ω j is the weight of the leaf node, γ and λ are hyperparameters used to adjust the regularization strength;

[0028] (22) XGBoost uses a greedy algorithm to build a new tree f t To minimize the objective function, assuming that the first t-1 trees are available, the prediction of the tth tree is f t (x i ), the model’s predicted value is updated to:

[0029]

[0030] In the formula, is the predicted value after the update in round t, is the cumulative prediction value of the first t-1 trees, f t (x i ) is the t-th tree for input x i The predicted output of

[0031] (23) In order to construct the t-th tree, XGBoost optimizes the second-order approximation of the objective function, that is, performing a second-order Taylor expansion on the objective function:

[0032]

[0033] In the formula, and They are the first and second derivatives of the loss function with respect to the predicted value;

[0034] (24) The optimal weight of each leaf node is determined by minimizing the second-order approximation of the objective function:

[0035]

[0036] Therefore, the gain of splitting a node (i.e., the reduction in the objective function) can be expressed as:

[0037]

[0038] The greater the gain, the better the effect after the split.

[0039] Furthermore, the corresponding model parameters are loaded according to the above-judged illumination conditions, the road area image is semantically segmented, and a small-resolution image block with complete road information is intercepted according to the obtained semantic mask, as follows:

[0040] (1) Semantic segmentation: Build a semantic segmentation network, use the lower half of the image after horizon detection as the input of the semantic segmentation network, load the corresponding model parameters according to the obtained day and night illumination results, and obtain the semantic mask of the road area;

[0041] (2) Cropping the image patch: Find the maximum and minimum coordinates of the non-zero pixels in the mask, locate the approximate center point of the road surface area, and crop the image patch of size [250, 200] with this point as the center.

[0042] Furthermore, the semantic segmentation network uses the DeeplabV3+ model, which includes an encoder and a decoder. The encoder introduces a hole convolution for cross-pixel feature extraction.

[0043] The DeeplabV3+ model uses ResNet-50 as the backbone network. After the image passes through the backbone network, part of it is directly input into the decoder, and the other part is processed by parallel hole convolution. The output results of the above two parts are spliced ​​to obtain the final semantic segmentation mask.

[0044] Furthermore, a multi-block fusion decision network is built, and the corresponding model parameters are loaded according to the above-judged illumination conditions. The intercepted image blocks are input into the multi-block fusion decision network. The multi-block fusion decision network adaptively assigns the weights of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result, as follows:

[0045] (1) Building a multi-tile fusion decision network, which includes a probability prediction network branch for extracting tile features and predicting probabilities, and a confidence network branch for adaptively learning the reliability of each tile;

[0046] Among them, the probability prediction network branch uses the EfficientNetV2 model, replacing the SE attention mechanism in the original network structure with a parallel lightweight attention mechanism CBAM;

[0047] The confidence network branch consists of a max pooling layer, a 1×1 convolutional layer, a style pooling and style integration module, and the last two convolutional layers;

[0048] (2) Each image block is input into the multi-block fusion decision network, and the corresponding network parameters are loaded according to the obtained day and night lighting conditions. The multi-block fusion decision network adaptively assigns the weight of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result;

[0049] (21) The resolution of the input image block is kept at [250, 200] without scaling, and the prediction probability of each image block is obtained: P 1 , P 2 , P 3 …;

[0050] (22) The high-resolution shallow features extracted from EfficientNetV2 are used as the input of the confidence network branch to learn the weights assigned to the prediction results of each image block;

[0051] (23) The confidence network branch converts the shallow feature map in the probability prediction network branch Mapping to an array of probabilities This array is used to determine the contribution of each image block to the final classification result, where B, C, H, W, and N represent the training batch size, the number of channels, height, width, and number of image blocks of the feature map, respectively;

[0052] (24) Feature map f i First, it passes through a maximum pooling layer and a 1×1 convolution layer to compress the size and number of channels to obtain f i ', for each image block, we mainly focus on the spatial features. Next, the style pooling and style integration modules in the network structure are transformed from f i 'Extract style information and integrate it to get Where d represents the number of pooling types used; the aggregate features of the image mean, maximum value, and standard deviation are used for style pooling;

[0053] (25) Then the style information of all image blocks is concatenated in the channel dimension to obtain And input a two-layer convolutional neural network to estimate the recalibrated weights of each channel. After normalization, the result is It is used to enable the network to dynamically estimate the relative importance of each style information and adaptively adjust the weight of the prediction probability of each image block. The above process can be expressed as:

[0054]

[0055] The learned weight W i It is used to measure the confidence of each image block. The network performs weighted fusion on the predicted probability of the image block based on different confidence levels:

[0056]

[0057] Where m represents different types of road surfaces.

[0058] According to a second aspect of the present invention, the present invention provides a light adaptive road surface detection system considering small target area recognition, which is used to implement the light adaptive road surface detection method considering small target area recognition, comprising:

[0059] An information separation module is used to receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road area, and obtain the upper half of the image containing environmental elements and the lower half of the image containing the road surface, vehicles and pedestrians;

[0060] The illumination judgment module uses the HSV attributes of the upper half of the image containing environmental information to judge the day and night state of the driving scene, so that the overall system can automatically adjust subsequent model parameters according to the illumination conditions;

[0061] The region extraction module loads the corresponding model parameters according to the above-determined illumination conditions, performs semantic segmentation on the road region image, and extracts a small-resolution image block with complete road information according to the obtained semantic mask;

[0062] The fusion recognition module is used to build a multi-block fusion decision network. The corresponding model parameters are loaded according to the above-mentioned lighting conditions, and the captured image blocks are input into the multi-block fusion decision network. The multi-block fusion decision network adaptively assigns the weights of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result.

[0063] According to a third aspect of the present invention, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the above-mentioned illumination adaptive road surface detection method considering small target area recognition is adopted.

[0064] The present invention has at least the following beneficial effects:

[0065] 1. The present invention divides environmental information and road surface information based on a horizon detection network, and uses the road surface information and environmental information for independent detection branches respectively, thereby avoiding information redundancy and mutual interference, and facilitating improving the accuracy of road surface area detection.

[0066] 2. The present invention proposes an adaptive illumination adjustment method, which can adaptively adjust model parameters according to different illumination conditions to achieve all-weather road type detection.

[0067] 3. The present invention takes into account the situation that the road surface area accounts for a relatively small proportion in the field of view, uses a multi-tile result fusion decision method, and proposes a multi-tile fusion decision network (MFDNet), which can suppress the prediction results of low-confidence tiles and amplify the prediction results of high-confidence tiles, so that the final detection results have good robustness and are more in line with actual driving scenarios.

[0068] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of the process of the detection method of the present invention;

[0070] Figure 2 A schematic diagram of the structure of the detection method of the present invention

[0071] Figure 3 It is a structural schematic diagram of the horizon detection network in the present invention;

[0072] Figure 4 This is a diagram of the parallel CBAM attention mechanism structure in the horizon detection network of the present invention;

[0073] Figure 5 This is a structural diagram of the ECA attention mechanism in the horizon detection network of the present invention;

[0074] Figure 6 A schematic diagram of an embodiment of the XGBoost tree structure in the present invention;

[0075] Figure 7 A schematic diagram of a road area image block cropping method provided by the present invention;

[0076] Figure 8 This is a network structure diagram of the multi-block fusion decision network (MFDNet) proposed in the present invention;

[0077] Fig. 9 It is a schematic diagram of the test results of the present invention in road type detection under large target images and small target images, compared with the method without fusion decision and the method based on soft voting decision. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0079] Embodiment 1:

[0080] See also Figure 1-Figure 9 The present invention provides a technical solution: a light-adaptive road surface detection method considering small target area recognition, comprising the following steps:

[0081] S1. Information separation: Receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road area, and obtain the upper half of the image containing environmental elements and the lower half of the image containing the road surface, vehicles and pedestrians, such as Figure 3 As shown, the details are as follows:

[0082] Get real-time images from the on-board camera, transform the regression problem into a classification problem, and use a simple classification network to output the results;

[0083] (S11) Use ResNet-34 as the backbone network to build a horizon detection network, and add two lightweight attention modules in parallel CBAM and ECA to the horizon detection network to improve the network effect;

[0084] (S11.1) Figure 4 The parallel CBAM module shown in Figure 1 consists of two submodules, namely CAM (channel attention module) and SAM (spatial attention module). The original serial CAM and SAM are modified into a parallel structure, which slightly improves the accuracy while maintaining similar inference speed. The feature representation of the output of the previous residual block in the network is Where C is the number of channels of the feature, H and W are the height and width of the feature, and the optimized features have the same size, which can be expressed as:

[0085]

[0086] M P_CBAM (F)= σ(M c (F)+ M s (F)) (3)

[0087] Where F is the characteristic of the input module, M c is the channel attention feature, M s is the spatial attention feature, and is the weight of MLP, C is the number of channels of the input image, r is the scaling factor, and f 7×7 represents the convolution operation with a 7×7 convolution kernel size, σ represents the Si gmod function, and M P_CBAM is the final parallel CBAM attention feature;

[0088] (S11.2) Figure 5As shown, the lightweight attention mechanism ECA is expressed as:

[0089]

[0090] In the formula, C is the number of channels of the input feature, f k×k represents a one-dimensional convolutional layer with a kernel size of k. Υ and b are hyperparameters. The absolute values ​​are taken and rounded down to the nearest odd number to ensure that the kernel size is an odd number. M ECA is the final ECA attention feature;

[0091] It should be noted that the horizon detection network inserts parallel CBAM modules in the first thirteen residual blocks and ECA modules in the last three residual blocks to achieve the best performance. The improved network structure is called Multi-Attention ResNet (MA-ResNet);

[0092] (S12) inputting the image into the MA-ResNet network, dividing the image along the height axis, the horizons of different images are distributed in different height regions of the image, and the position of the image horizon can be approximately determined according to the distribution of objects in the image, thereby simply dividing the image into upper and lower parts according to the horizon;

[0093] Since the cameras in the dataset are mostly installed at the driving perspective, there is a significant sample imbalance in the dataset. The horizon in most images is located in the middle of the image. The network is prone to overfitting on a large number of samples, while it is underfitting on a small number of samples. It is necessary to use the dynamically scaled cross entropy loss FocalLoss as the loss function. It uses a dynamic focusing factor to reduce the weight of easily distinguishable samples during training and quickly shift the training focus to difficult-to-distinguish samples. The formula of the loss function is as follows:

[0094]

[0095] In the formula, N is the total number of samples, C is the total number of categories, α c is the balancing factor of category C, γ is the focusing factor, and y i,c is the true label of sample i in category C, p i,c is the predicted probability that sample i belongs to category C; as γ increases, the loss contribution of easy-to-distinguish samples is significantly reduced, and the impact of difficult-to-distinguish samples on network training is amplified;

[0096] S2. Lighting judgment: The HSV attributes of the upper half of the image containing environmental information are used to judge the day and night state of the driving scene, so that the overall system can automatically adjust the subsequent model parameters according to the lighting conditions, as follows:

[0097] (S21) converting the upper half of the image after horizon detection into the HSV color space, which includes three attributes: hue, saturation, and value. Hue indicates the type of color, saturation describes the grayscale of a specific color, and value indicates the brightness or intensity of the color.

[0098] (S22) Combination Figure 6 , a Gradient Boosting Decision Tree (GBDT) is used to build a tree integration model, and the HSV attributes of the image are input into the integration model to regress the day and night state of the driving scene;

[0099] (S22.1) The integrated model uses the efficient gradient boosting algorithm XGBoost, and the objective function can be expressed as:

[0100]

[0101] In the formula, Represents the set of parameters of the model, is the training error, which is used to measure the predicted value and the true value y i The difference between Ω(f k ) is a regularization term used to control model complexity; T is the number of leaf nodes in the tree, ω j is the weight of the leaf node, γ and λ are hyperparameters used to adjust the regularization strength;

[0102] (S22.2) At each step, XGBoost uses a greedy algorithm to build a new tree f t To minimize the objective function, assuming that the first t-1 trees are available, the prediction of the tth tree is f t (x i ), the model’s predicted value is updated to:

[0103]

[0104] In the formula, is the predicted value after the update in round t, is the cumulative prediction value of the first t-1 trees, f t (x i ) is the t-th tree for input x i The predicted output of

[0105] (S22.3) In order to construct the t-th tree, XGBoost optimizes the second-order approximation of the objective function, that is, performs a second-order Taylor expansion on the objective function:

[0106]

[0107] In the formula, and They are the first and second derivatives of the loss function with respect to the predicted value;

[0108] (S22.4) Determine the optimal weight of each leaf node by minimizing the second-order approximation of the objective function:

[0109]

[0110] Therefore, the gain of splitting a node (i.e., the reduction in the objective function) can be expressed as:

[0111]

[0112] This gain formula is used to evaluate whether it is worth splitting the node. The greater the gain, the better the effect after the split.

[0113] Specifically, XGBoost effectively prevents overfitting and improves the generalization ability of the model by iteratively optimizing the objective function, achieving an accuracy of more than 98.5% when distinguishing day and night scenes in images;

[0114] In the subsequent steps, each stage trains the model parameters for different lighting conditions during the day and night, and loads the corresponding model parameters according to the lighting judgment results here;

[0115] S3. Region extraction: Load the corresponding model parameters according to the above-determined illumination conditions, perform semantic segmentation on the road area image, and extract small-resolution image blocks with complete road information based on the obtained semantic mask, as follows:

[0116] (S31) semantic segmentation: constructing a semantic segmentation network, taking the lower half of the image after horizon detection as the input of the semantic segmentation network, loading corresponding model parameters according to the result of the aforementioned illumination judgment, and obtaining a semantic mask of the road surface area;

[0117] (S31.1) The semantic segmentation network uses the Deep l abV3+ model, which consists of two parts: an encoder and a decoder. The encoder introduces a dilated convolution for cross-pixel feature extraction. The dilated convolution increases the receptive field without losing information, so that each convolution output can contain a wider range of information.

[0118] The Deep l abV3+ model uses ResNet-50 as the backbone network. After the image passes through the backbone network, part of it is directly input to the decoder, while the other part undergoes parallel dilated convolution processing. The results of these two branches are spliced ​​to obtain the final semantic segmentation mask.

[0119] (S31.2) The loss function uses the pixel-level cross entropy loss function, expressed as:

[0120]

[0121] Where N is the total number of pixels in the image, C is the total number of categories, and w c is the weight associated with category C, y i,c is a binary indicator, if pixel i is classified correctly, then y i,c is 1, otherwise it is 0, p i,c is the probability that pixel i is predicted to be of category C;

[0122] (S32) Crop image blocks: When the road surface area accounts for a very small proportion of the image, these small target images contain a lot of irrelevant information. In the classification task, directly adjusting the size of the small target image and inputting it into the network will significantly affect the classification accuracy. In order to improve the accuracy of road surface recognition in small target images, a small resolution image block is selected according to the pixel distribution of the mask to focus on the road surface information.

[0123] Combination Figure 7 ,The road surface area in the mask can be regarded as an irregular shape. By finding the maximum and minimum coordinates of non-zero pixels, the approximate center point of the road surface area is located, and the image block with a size of [250,200] is cropped with this point as the center;

[0124] S4. Fusion recognition: Build a multi-block fusion decision network, load the corresponding model parameters according to the above-mentioned lighting conditions, input the intercepted image blocks into the multi-block fusion decision network, and the multi-block fusion decision network adaptively assigns the weight of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result, as follows:

[0125] (S41) Combination Figure 8 A multi-tile fusion decision network (MFDNet) is built, which mainly consists of two parts: a probability prediction network branch for extracting tile features and predicting probabilities, and a confidence network branch for adaptively learning the reliability of each tile; each image block is input into the multi-tile fusion decision network, and the corresponding network parameters are loaded according to the results of the aforementioned illumination judgment;

[0126] (S41.1) The probability prediction network branch uses the EfficientNetV2 model. The SE attention module in the original network structure is replaced by the aforementioned parallel CBAM module to improve the classification accuracy. The resolution of the input image block is kept at [250, 200] without scaling. The prediction probability of each image block is obtained: P 1 , P 2 , P 3 …; the loss function adopts the cross entropy loss function, which is expressed as:

[0127]

[0128] In the formula, N is the total number of samples, C is the total number of categories, and y i,c is the true label of sample i in category c, p i,c is the probability that pixel i is predicted to be of category C;

[0129] The high-resolution shallow features extracted from EfficientNetV2 are used as the input of the confidence network branch to learn the weights assigned to the prediction results of each image block; the confidence network uses the shallow feature maps in the probability prediction network as input to the confidence network branch. Mapping to an array of probabilities This array is used to determine the contribution of each image block to the final classification result. Here, B, C, H, W, and N represent the training batch size, the number of channels, height, width, and number of image blocks of the feature map, respectively.

[0130] Feature map f i First, a maximum pooling layer and a 1×1 convolution layer are used to compress the size and number of channels to obtain f i ', for each image block, we mainly focus on the spatial features; next, the style pooling and style integration modules in the network structure are transformed from f i 'Extract style information and integrate it to get Where d represents the number of pooling types used; the aggregated features of the image mean, maximum value, and standard deviation are used for style pooling; then the style information of all image blocks is concatenated in the channel dimension to obtain And input a two-layer convolutional neural network to estimate the recalibrated weights of each channel. After normalization, the result is This enables the network to dynamically estimate the relative importance of each style information and adaptively adjust the weight of the prediction probability of each image block. The above process can be expressed as:

[0131]

[0132] The learned weight W iIt is used to measure the confidence of each image block. The network performs weighted fusion on the predicted probability of the image block based on different confidence levels:

[0133]

[0134] In the formula, m represents different types of road surfaces;

[0135] In this model, the prediction results of low-confidence patches are suppressed, while the prediction results of high-confidence patches are amplified, ensuring that the final result is mainly contributed by relatively reliable patches.

[0136] Next, the present invention is further described in conjunction with specific embodiments:

[0137] In the process of model training, in order to avoid the sample bias and driving scene limitations that may exist in a single dataset, the experiment integrated images of the BDD100K, Oxford Robotcar and CADC datasets collected by the on-board camera to create a three-category dataset covering daytime and nighttime driving scenes. This dataset is used to identify dry, slippery and snowy roads under various lighting conditions, thereby training an all-weather detection model with higher recognition accuracy and better robustness.

[0138] Fig. 9 This is a schematic diagram of the test results of the method described in this embodiment in road type detection under large target images and small target images, compared with the method without fusion decision and the method based on soft voting decision; when the road surface area in the image is large and clear and easy to identify, the road surface detection is relatively simple, so the advantage of the method described in this embodiment is not obvious; however, when the road surface area is small and difficult to identify, a large amount of interference information mixed in the image may affect the recognition effect, and the method described in this embodiment has significant advantages at this time; although this type of small target image accounts for a small proportion in the data set, it is a situation that affects driving safety and cannot be ignored in actual driving scenarios. At this time, the method described in this embodiment can effectively suppress these interference areas in the image and amplify the prediction results of reliable areas, so that the final detection result is more accurate.

[0139] In summary, this embodiment can adaptively adjust the model parameters according to different lighting conditions to achieve all-weather road type detection, and takes into account the situation that the road surface area accounts for a small proportion in the field of view, ensuring that the network can always focus on the road surface area even in high-resolution images, while ensuring that images with a large proportion of the road surface have sufficient receptive fields. For this purpose, a multi-tile result fusion decision method is used, and a multi-tile fusion decision network (MFDNet) is proposed. The network can suppress the prediction results of low-confidence tiles and amplify the prediction results of high-confidence tiles, so that the final detection results have good robustness and are more in line with actual driving scenarios.

[0140] Embodiment 2:

[0141] This embodiment provides a light-adaptive road surface detection system considering small target area recognition, which is used to implement the light-adaptive road surface detection method considering small target area recognition, including:

[0142] An information separation module is used to receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road area, and obtain the upper half of the image containing environmental elements and the lower half of the image containing the road surface, vehicles and pedestrians;

[0143] The illumination judgment module uses the HSV attributes of the upper half of the image containing environmental information to judge the day and night state of the driving scene, so that the overall system can automatically adjust subsequent model parameters according to the illumination conditions;

[0144] The region extraction module loads the corresponding model parameters according to the above-mentioned illumination judgment result, performs semantic segmentation on the road region image, and intercepts the small-resolution image block with complete road information according to the obtained semantic mask;

[0145] The fusion recognition module is used to build a multi-block fusion decision network. According to the results of the aforementioned illumination judgment, the corresponding model parameters are loaded, and the captured image blocks are input into the multi-block fusion decision network. The multi-block fusion decision network adaptively assigns the weights of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result.

[0146] Specifically, the above-mentioned information separation module, illumination judgment module, region extraction module and fusion recognition module can be embedded in a computer processing system. The computer calls the above-mentioned modules to complete the task of realizing all-weather road type detection based on the above-mentioned illumination adaptive road surface detection method considering small target area recognition; the above-mentioned information separation module, illumination judgment module, region extraction module and fusion recognition module can perform operations according to the specific steps given in the illumination adaptive road surface detection method considering small target area recognition.

[0147] It should be noted that it should be understood that the division of the various modules of the above system is only the division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated, and these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the information separation module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The functions of the above signal processing module, and the implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0148] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a certain module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program codes. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0149] Embodiment three:

[0150] The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores a computer program capable of running on the processor. When the processor loads and executes the computer program, the above-mentioned illumination adaptive road surface detection method considering small target area recognition is adopted.

[0151] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory. For example, the terminal device can also include input and output devices, network access devices and buses.

[0152] Furthermore, the processor may adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. may also be adopted. The general-purpose processor may adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0153] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0154] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on", "installed on", "fixed on" or "set on" another element, it can be directly on the other element or there can also be a centered element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be a centered element at the same time. The terms "vertical", "horizontal", "up", "down", "left", "right" and similar expressions used herein are only for illustrative purposes and are not intended to be the only implementation method.

[0155] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0156] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. An illumination-adaptive road surface detection method considering small target area recognition, characterized in that: The following steps are involved: Receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road area, and obtain the upper half of the image containing environmental elements and the lower half of the image containing the road surface, vehicles and pedestrians; The HSV attributes of the upper half of the image containing environmental information are used to determine the day and night state of the driving scene, so that the overall system can automatically adjust subsequent model parameters according to lighting conditions; According to the above-determined illumination conditions, the corresponding model parameters are loaded, the road surface area image is semantically segmented, and a small-resolution image block with complete road surface information is intercepted according to the obtained semantic mask; Build a multi-block fusion decision network, load the corresponding model parameters according to the above-mentioned lighting conditions, input the captured image blocks into the multi-block fusion decision network, and the multi-block fusion decision network adaptively assigns the weights of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result.

2. The illumination adaptive road surface detection method considering small target area recognition according to claim 1, characterized in that: Receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road area, and obtain the upper half of the image containing environmental elements and the lower half of the image containing the road surface, vehicles and pedestrians, as follows: (1) The horizon detection network uses ResNet-34 as the backbone network, and adds two lightweight attention mechanisms, CBAM and ECA, to the horizon detection network structure in parallel; (2) The image is input into the horizon detection network. The image is divided along the height axis. The horizons of different images are distributed in different height areas of the image. The position of the image horizon is determined according to the distribution of objects in the image, and the image is divided into upper and lower parts according to the horizon.

3. The illumination adaptive road surface detection method considering small target area recognition according to claim 2, characterized in that: The lightweight attention mechanism parallel CBAM includes two submodules, namely the channel attention module CAM and the spatial attention module SAM. The original serial CAM and SAM are modified into a parallel structure to improve the accuracy while maintaining similar reasoning speed; The feature representation of the output of the previous residual block in the horizon detection network is Where C is the number of channels of the feature, H and W are the height and width of the feature, and the optimized features have the same size, which is specifically expressed as: M P_CBAM (F)= σ(M c (F)+ M s (F)) (3) Where F is the characteristic of the input module, M c is the channel attention feature, M s is the spatial attention feature, and is the weight of MLP, C is the number of channels of the input image, r is the scaling factor, and f 7×7 represents the convolution operation with a 7×7 convolution kernel size, σ represents the Sigmoid function, and M P_CBAM The attention features of parallel CBAM are finally parallelized with the lightweight attention mechanism.

4. The illumination adaptive road surface detection method considering small target area recognition according to claim 2, characterized in that: The lightweight attention mechanism ECA is expressed as: In the formula, C is the number of channels of the input feature, f k×k represents a one-dimensional convolutional layer with a kernel size of k. Υ and b are hyperparameters. The absolute values ​​are taken and rounded down to the nearest odd number to ensure that the kernel size is an odd number. M ECA is the final ECA attention feature, and σ represents the Sigmoid function.

5. The illumination adaptive road surface detection method considering small target area recognition according to claim 4, characterized in that: The HSV attributes of the upper half of the image containing environmental information are used to determine the day and night state of the driving scene, so that the overall system can automatically adjust the subsequent model parameters according to the lighting conditions, as follows: (1) Convert the upper half of the image containing environmental information after horizon detection into the HSV color space, which contains three attributes: hue, saturation, and value. Hue indicates the type of color, saturation describes the grayscale of a specific color, and value indicates the brightness or intensity of the color. (2) A gradient boosting decision tree is used to build a tree ensemble model, and the HSV attributes of the image are input into the ensemble model to regress the day and night state of the driving scene; (21) The integrated model uses the efficient gradient boosting algorithm XGBoost, and the objective function is expressed as: In the formula, Represents the set of parameters of the model, is the training error, which is used to measure the predicted value and the true value y i The difference between Ω(f k ) is a regularization term used to control model complexity, T is the number of leaf nodes in the tree, ω j is the weight of the leaf node, γ and λ are hyperparameters used to adjust the regularization strength; (22) XGBoost uses a greedy algorithm to build a new tree f t To minimize the objective function, assuming that the first t-1 trees are available, the prediction of the tth tree is f t (x i ), the model’s predicted value is updated to: In the formula, is the predicted value after the update in round t, is the cumulative prediction value of the first t-1 trees, f t (x i ) is the t-th tree for input x i The predicted output of (23) In order to construct the t-th tree, XGBoost optimizes the second-order approximation of the objective function, that is, performing a second-order Taylor expansion on the objective function: In the formula, and They are the first and second derivatives of the loss function with respect to the predicted value; (24) The optimal weight of each leaf node is determined by minimizing the second-order approximation of the objective function: Therefore, the gain of splitting a node (i.e., the reduction in the objective function) can be expressed as: The greater the gain, the better the effect after the split.

6. The illumination adaptive road surface detection method considering small target area recognition according to claim 5, characterized in that: According to the above-judged lighting conditions, the corresponding model parameters are loaded, the road area image is semantically segmented, and the small-resolution image blocks with complete road information are intercepted according to the obtained semantic mask, as follows: (1) Semantic segmentation: Build a semantic segmentation network, use the lower half of the image after horizon detection as the input of the semantic segmentation network, load the corresponding model parameters according to the illumination judgment result, and obtain the semantic mask of the road surface area; (2) Cropping the image patch: Find the maximum and minimum coordinates of the non-zero pixels in the mask, locate the approximate center point of the road surface area, and crop the image patch of size [250, 200] centered on this point.

7. The illumination adaptive road surface detection method considering small target area recognition according to claim 6, characterized in that: The semantic segmentation network uses the DeeplabV3+ model, which includes an encoder and a decoder. The encoder introduces a hole convolution for cross-pixel feature extraction. The DeeplabV3+ model uses ResNet-50 as the backbone network. After the image passes through the backbone network, part of it is directly input into the decoder, and the other part is processed by parallel hole convolution. The output results of the above two parts are spliced ​​to obtain the final semantic segmentation mask.

8. The illumination adaptive road surface detection method considering small target area recognition according to claim 6, characterized in that: Build a multi-block fusion decision network, load the corresponding model parameters according to the above-judged lighting conditions, input the intercepted image blocks into the multi-block fusion decision network, and the multi-block fusion decision network adaptively assigns the weights of each block feature, and weights the prediction probability of each image according to the assigned weights to obtain the fused recognition results, as follows: (1) Building a multi-tile fusion decision network, which includes a probability prediction network branch for extracting tile features and predicting probabilities, and a confidence network branch for adaptively learning the reliability of each tile; Among them, the probability prediction network branch uses the EfficientNetV2 model, replacing the SE attention mechanism in the original network structure with a parallel lightweight attention mechanism CBAM; The confidence network branch consists of a max pooling layer, a 1×1 convolutional layer, a style pooling and style integration module, and the last two convolutional layers; (2) Each image block is input into the multi-block fusion decision network, and the corresponding network parameters are loaded according to the obtained day and night lighting conditions. The multi-block fusion decision network adaptively assigns the weight of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result; (21) The resolution of the input image block is kept at [250, 200] without scaling, and the prediction probability of each image block is obtained in the probability prediction branch: P1, P2, P3, ...; (22) The high-resolution shallow features extracted from EfficientNetV2 are used as the input of the confidence network branch to learn the weights assigned to the prediction results of each image block; (23) The confidence network branch converts the shallow feature map in the probability prediction network branch Mapping to an array of probabilities This array is used to determine the contribution of each image block to the final classification result, where B, C, H, W, and N represent the training batch size, the number of channels, height, width, and number of image blocks of the feature map, respectively; (24) Feature map f i First, it passes through a maximum pooling layer and a 1×1 convolution layer to compress the size and number of channels to obtain f i ', for each image block, we mainly focus on the spatial features. Next, the style pooling and style integration modules in the network structure are transformed from f i 'Extract style information and integrate it to get Where d represents the number of pooling types used; the aggregate features of the image mean, maximum value, and standard deviation are used for style pooling; (25) Then the style information of all image blocks is concatenated in the channel dimension to obtain And input a two-layer convolutional neural network to estimate the recalibrated weights of each channel. After normalization, the result is It is used to enable the network to dynamically estimate the relative importance of each style information and adaptively adjust the weight of the prediction probability of each image block. The above process can be expressed as: The learned weight W i It is used to measure the confidence of each image block. The network performs weighted fusion on the predicted probability of the image block based on different confidence levels: Where m represents different types of road surfaces.

9. An illumination adaptive road surface detection system considering small target area recognition, used to implement the illumination adaptive road surface detection method considering small target area recognition according to any one of claims 1 to 8, characterized in that: include: An information separation module is used to receive real-time image data, build a horizon detection network, input the image into the horizon detection network to separate the environment area and the road area, and obtain the upper half of the image containing environmental elements and the lower half of the image containing the road surface, vehicles and pedestrians; The illumination judgment module uses the HSV attributes of the upper half of the image containing environmental information to judge the day and night state of the driving scene, so that the overall system can automatically adjust subsequent model parameters according to the illumination conditions; The region extraction module loads the corresponding model parameters according to the above-determined illumination conditions, performs semantic segmentation on the road region image, and extracts a small-resolution image block with complete road information according to the obtained semantic mask; The fusion recognition module is used to build a multi-block fusion decision network. The corresponding model parameters are loaded according to the above-mentioned lighting conditions, and the captured image blocks are input into the multi-block fusion decision network. The multi-block fusion decision network adaptively assigns the weights of each block feature, and weights the prediction probability of each image according to the assigned weight to obtain the fused recognition result.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the illumination adaptive road surface detection method considering small target area recognition according to any one of claims 1 to 8 is adopted.

Images