An Obstacle Detection Method and Device under Complex Weather Conditions

Abstract

The present invention discloses an obstacle detection method and device under complex weather conditions, including: acquiring an image under a complex weather environment; performing enhanced preprocessing on the image by using a multi-scale retinex with color restoration (MSRCR) algorithm; inputting the preprocessed image into a trained obstacle detection model based on an improved YOLOv3 network; determining an obstacle detection result under complex weather according to the output of the obstacle detection model based on the improved YOLOv3 network; wherein the construction method of the obstacle detection model includes: inserting a spatial pyramid pooling (SPP) module into the convolutional block (Convolutional Set) of YOLOv3; replacing the Leaky-ReLU activation function in the convolutional layer of the original YOLOv3 network with an exponential linear unit (ELU) activation function; acquiring a dataset for obstacle detection under different weather conditions; using the K-Means++ algorithm to re-cluster the ground truth boxes in the dataset to obtain appropriate anchor boxes; training the obstacle detection model with the processed dataset to obtain a trained obstacle detection model based on the improved YOLOv3 network.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to an obstacle detection method and device under complex weather conditions. Background Art

[0002] In recent years, with the rapid development of the social economy, the number of domestic automobiles has increased rapidly. While vehicles bring great convenience to life, problems such as frequent traffic accidents also pose challenges to public traffic safety. The demand for accurate detection of obstacles by vehicles is becoming increasingly urgent. Currently, the implemented autonomous driving systems and most of the automatic assisted driving systems can sense the environment around the vehicle and make corresponding judgments and actions according to the environment. For the acquisition of environmental information, vehicles mainly rely on sensor devices (such as depth cameras, lidar, ultrasonic waves, etc.). Compared with the environmental information obtained by other sensor devices, cameras can not only obtain richer scene information, but also have advantages such as low cost and easy integration.

[0003] With the development of deep learning technology, object detection algorithms based on deep learning have been widely used in object detection. Among them, two-stage object detection algorithms represented by Faster-RCNN have greatly improved the accuracy of object detection, but their detection speed is slow and cannot meet the real-time requirements. Single-stage object detection algorithms represented by the YOLO algorithm have a relatively high detection speed, but the detection accuracy for small objects is relatively low. In 2018, the Redmon J team proposed the YOLOv3 algorithm, which uses a deep residual network to extract image features and realizes multi-scale prediction.

[0004] Due to the interference of weather environmental factors such as possible rainfall, snowfall, sand and dust, haze, strong light, and night, problems such as distortion, blurring, and low contrast will occur in road images, which will affect the obstacle detection for images. Coupled with some defects of the YOLOv3 network itself in detecting road obstacles under complex weather conditions, if the YOLOv3 algorithm is directly used, the detection and classification accuracy of obstacles will be significantly reduced. Summary of the Invention

[0005] To address the deficiencies in the prior art, the present invention provides an obstacle detection method under complex weather conditions, proposing an obstacle detection method that integrates a multi-scale retinex with color restoration (MSRCR) algorithm and an improved YOLOv3 object detection algorithm. First, the original image is preprocessed by the MSRCR algorithm for image enhancement to improve the contrast and detail quality of the image. Then, the YOLOv3 network is optimized by introducing the SPP module, using the ELU activation function to increase the robustness to noise, selecting an appropriate dataset, re-clustering the anchor boxes in the original YOLOv3 network on the dataset using the K-Means++ algorithm, and training the improved YOLOv3 network using the training dataset. Finally, the enhanced and preprocessed image is input into the trained detection network for detection and classification. The present invention uses the improved YOLOv3 network to detect obstacles in the image after image enhancement, improving the detection accuracy.

[0006] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, an obstacle detection method under complex weather conditions is provided, including:

[0008] Obtaining an image under a complex weather environment;

[0009] Performing enhanced preprocessing on the image using the multi-scale retinex with color restoration (MSRCR) algorithm to obtain a preprocessed image;

[0010] Inputting the preprocessed image into a trained obstacle detection model based on the improved YOLOv3 network;

[0011] Determining the obstacle detection result under complex weather according to the output of the obstacle detection model based on the improved YOLOv3 network;

[0012] Wherein the construction method of the obstacle detection model based on the improved YOLOv3 network includes:

[0013] Inserting the SPP module into the Convolutional Set of YOLOv3 to achieve the fusion of features at different scales, extract more detailed information, and thus improve the detection accuracy; using the ELU activation function to replace the Leaky-ReLU activation function in the convolutional layer of the original YOLOv3 network to increase the robustness to noise;

[0014] Obtaining a dataset for obstacle detection under different weather conditions to enhance the robustness of the trained model;

[0015] Using the K-Means++ algorithm to re-cluster the ground truth boxes in the dataset to obtain appropriate anchor boxes, accelerating the convergence speed of sample training and improving the localization accuracy;

[0016] The obstacle detection model is trained with the processed data set to obtain a trained obstacle detection model based on the improved YOLOv3 network.

[0017] In some embodiments, the multi-scale retinex with color restoration (MSRCR) algorithm is used to perform enhanced preprocessing on the image, including:

[0018] Using the weighted sum of several different Gaussian scale parameters, the Gaussian filtering result is used as the estimated illumination image, and a color restoration factor is added to restore the color of the reflected image to avoid color degradation. The calculation formula is as follows:

[0019]

[0020]

[0021]

[0022] Where FMSRCR(x, y) is the enhanced result processed by the MSRCR algorithm, x and y respectively represent the abscissa and ordinate of the image pixels, i represents the serial number of the color component, j represents the serial number of the scale parameter, Ci is the color restoration factor of the i-th color component, N is the number of scale parameters, ωj is the weight corresponding to different scales, I i (x, y) represents the i-th color component of the input image, Gj(x, y) represents the Gaussian filter at the scale of σj, and σj represents the scale parameter of the j-th Gaussian surround. is the convolution symbol, α is the nonlinear intensity adjustment parameter, and β is the gain factor.

[0023] Furthermore, N = 3, representing three scales: small, medium, and large.

[0024] In some embodiments, there are a total of four branches in the SPP module: the first branch is that the input is directly connected to the output branch, the second branch is the max pooling with a pooling kernel of 5×5, the third branch is the max pooling with a pooling kernel of 9×9, and the fourth branch is the max pooling with a pooling kernel of 13×13. The pooling stride is 1 each time, and padding is performed before pooling to keep the size and depth of the finally obtained feature map unchanged; finally, the SPP module realizes the fusion of features of different scales.

[0025] In some embodiments, the Leaky-ReLU activation function in the convolutional layer of the original YOLOv3 network is replaced with the ELU activation function, including:

[0026] The ELU activation function, the calculation formula is:

[0027]

[0028] Let \(u\) and \(m\) represent the horizontal and vertical coordinates of the function respectively. The gradient of the ELU activation function is non-zero for all negative values, and there is no problem of neuron death. That is, when using the ELU activation function in the case of abnormal input, large gradients will not be generated during backpropagation, so it will not lead to neuron death and vanishing gradients, and can shorten the training time and improve the accuracy in training the network; when \(m\) in ELU is 0 or negative, the exponential function is used, and as the parameter becomes smaller, the function gradually converges to a negative value; convergence means there is a small derivative value, reducing the changes and information propagated to the next layer; therefore, the ELU activation function is more robust to noise and can reduce the impact of image noise on the detection results.

[0029] In some embodiments, to obtain a dataset for obstacle detection under different weather conditions and improve the robustness of the training model, the following steps are included: select the representative KITTI dataset and CODA dataset, supplement the KITTI dataset with the modified CODA dataset corresponding labels to obtain a dataset for obstacle detection under different weather conditions; reserve a test set from the obtained dataset, and use the remaining dataset as the training set to increase the robustness of the training model.

[0030] Further, supplementing the KITTI dataset with the modified CODA dataset corresponding labels includes:

[0031] Classify the obstacle labels into categories such as car, van, truck, pedestrian, pedestrian(sitting), cyclist, tram, and misc for detection and classification;

[0032] Expand the complex weather part of the CODA dataset through data augmentation methods such as Mixup, Cutmix, and Cutout, and modify the corresponding labels to conform to the classification of the said obstacle labels;

[0033] Uniformly resize the images in the dataset to 416×416 pixels, which is convenient for the subsequent training of the obstacle detection model based on the improved YOLOv3 network.

[0034] In some embodiments, training the obstacle detection model with the processed dataset includes:

[0035] Train the improved YOLOv3 network on the training set part of the dataset and adjust the parameters accordingly; use the mean average precision (mAP) of each category to evaluate the performance of the model, and the calculation formula is as follows:

[0036]

[0037] mAP refers to the average value of APs for each category. APa is the area under the precision-recall curve for the a-th category. k represents the total number of all categories, and a represents the serial number of the category. The larger the value of the average precision mAP, the higher the overall recognition accuracy of the model. The reserved test set is used to verify the effect, and an obstacle detection model based on the improved YOLOv3 network is obtained.

[0038] In a second aspect, the present invention provides an obstacle detection device in complex weather, including a processor and a storage medium;

[0039] The storage medium is used to store instructions;

[0040] The processor is used to operate according to the instructions to execute the steps of the method according to the first aspect.

[0041] In a third aspect, the present invention provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method according to the first aspect are implemented.

[0042] The advantages of the present invention are as follows: The method provided by the present invention aims at the problem of inaccurate detection of obstacles in road images in complex environments such as rainfall, snowfall, sand and dust, haze, strong light, and night, and proposes an obstacle detection method that combines a multi-scale retinex with color restoration (MSRCR) algorithm and an improved YOLOv3 object detection algorithm. The image to be detected is preprocessed by the MSRCR algorithm for image enhancement to improve the contrast and detail quality of the image; then the YOLOv3 network is optimized. The SPP module is introduced to fuse features of different scales to improve the detection accuracy, and the ELU activation function is used to increase the robustness to noise. A dataset that conforms to the scenario is obtained, and the K-Means++ clustering algorithm is used to re-cluster the true boxes in the dataset to obtain more appropriate anchor box sizes to make the target positioning more accurate, and then the network is trained; finally, the enhanced image is input into the trained improved YOLOv3 network to obtain the detection result. The present invention improves the accuracy and robustness of obstacle detection in complex weather through the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the method in an embodiment of the present invention.

[0044] Figure 2 It is an overall structure diagram of the YOLOv3 network based on an embodiment of the present invention.

[0045] Figure 3 It is a structure diagram of the SPP module in an embodiment of the present invention.

[0046] Figure 4 It is a schematic diagram of the position of the SPP module in an embodiment of the present invention.

[0047] Figure 5 It is a schematic diagram of a convolutional layer after replacing the activation function in an embodiment of the present invention. Specific embodiments

[0048] In order to make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

[0049] In the description of the present invention, the meaning of several is more than one, the meaning of multiple is more than two, and understandings such as greater than, less than, exceeding, etc. do not include the present number, and understandings such as above, below, within, etc. include the present number. If the first and second are described only for the purpose of distinguishing technical features, they cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0050] In the description of the present invention, the descriptions with reference to terms such as "an embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0051] The YOLOv3 algorithm is an improvement on YOLOv1 and YOLOv2. It has the advantages of high detection accuracy, accurate positioning, fast speed, etc. Especially when introducing the multi-scale prediction method, it can achieve the detection of small targets and has good robustness to the environmental scene. Therefore, it is still a research hotspot at present. In order to extract deeper feature information, YOLOv3 uses the Darknet-53 network as the backbone network. The Darknet-53 network uses a large number of 1×1 and 3×3 convolutional layers and residual units to obtain more meaningful semantic information from the upsampled features and obtain finer-grained information from the previous feature maps.

[0052] The purpose of image enhancement technology is to improve the image quality, present useful information in the image, make the image more in line with human visual perception, and make it easier to obtain the effect of machine recognition. The color restoration multi-scale Retinex algorithm (MSRCR) is an improvement and enhancement based on the single-scale Retinex algorithm (SSR) and the multi-scale Retinex algorithm (MSR). It can significantly improve the color cast effect of the existing two algorithms and expand the dynamic range of the image, so that the details of the bright and dark areas of the image can be well presented. Using image enhancement technology to preprocess the image can detect obstacles in the image more accurately.

[0053] Example 1

[0054] A method for obstacle detection in complex weather, including:

[0055] Obtain an image in a complex weather environment;

[0056] Use the multi-scale retina enhancement algorithm MSRCR with color restoration to perform enhanced preprocessing on the image to obtain a preprocessed image;

[0057] Input the preprocessed image into a trained obstacle detection model based on the improved YOLOv3 network;

[0058] According to the output of the obstacle detection model based on the improved YOLOv3 network, determine the obstacle detection result in complex weather;

[0059] Wherein the construction method of the obstacle detection model based on the improved YOLOv3 network includes:

[0060] Insert the SPP module into the Convolutional Set of YOLOv3 to achieve the fusion of features at different scales, extract more detailed information, and thus improve the detection accuracy; use the ELU activation function to replace the Leaky-ReLU activation function in the convolutional layer of the original YOLOv3 network to increase the robustness to noise;

[0061] Obtain a dataset for obstacle detection under different weather conditions to improve the robustness of the training model;

[0062] Use the K-Means++ algorithm to re-cluster the ground truth boxes in the dataset to obtain appropriate anchor boxes, accelerate the convergence speed of sample training, and improve the positioning accuracy;

[0063] Train the obstacle detection model with the processed dataset to obtain a trained obstacle detection model based on the improved YOLOv3 network.

[0064] In some embodiments, a method for obstacle detection in complex weather, such as Figure 1As shown below, the detailed steps are as follows:

[0065] S1: Preprocess the image using the MSRCR image enhancement algorithm to reduce the problems of low contrast and blurred details in the image under complex weather conditions, making the image easier to extract features;

[0066] Specifically, it includes:

[0067] S11: Select the MSRCR algorithm to achieve image feature enhancement. The MSRCR algorithm uses the weighted sum of several different Gaussian scale parameters, takes the Gaussian filtering result as the estimated illumination image, and adds a color restoration factor to restore the color of the reflected image. This avoids the problem of color degradation, and the enhanced image has better color guarantee. The calculation formula is as follows:

[0068]

[0069]

[0070]

[0071] x and y respectively represent the horizontal and vertical coordinates of the image pixels, i represents the serial number of the color component, j represents the serial number of the scale parameter, Ii(x, y) represents the i-th color component of the input image, ωj is the weight corresponding to different scales, N is the number of scale parameters, generally N = 3, representing small, medium, and large scales, Gj(x, y) represents the Gaussian filter when the scale is σj, is the convolution symbol, Ci is the color restoration factor of the i-th color component, σj represents the scale parameter of the j-th Gaussian surround, α is the non-linear intensity adjustment parameter, β is the gain factor, and FMSRCR(x, y) is the enhancement result processed by the MSRCR algorithm;

[0072] S12: Preprocess the image using the above MSRCR algorithm to improve the overall brightness and saturation, reduce the noise in the image, and enable the detection model to classify and locate obstacles more accurately;

[0073] S2: The YOLOv3 model is mature and has stable practical application effects, meeting the requirements of the obstacle detection method. Therefore, YOLOv3 is selected as the detection network for obstacle detection in complex weather. The network structure of YOLOv3, as Figure 2 shown, uses the Darknet53 feature extraction network, and the activation function uses Leaky-ReLU. However, considering the problems of blurred images, low contrast, and color fading under complex weather, the original network needs to be optimized to achieve the recognition and location of obstacle types as much as possible.

[0074] For the YOLOv3 network, an SPP module is introduced to achieve feature fusion at different scales, obtain more features, and improve detection accuracy. The ELU activation function is used to replace the Leaky-ReLU to increase the robustness to noise, enabling it to accurately detect blurred and low-contrast images in complex weather conditions. The CODA and KITTI datasets with rich weather scenes and suitable for obstacle detection scenarios are selected as the training set and test set to enhance the robustness of the training model. The K-Means++ algorithm is used to re-cluster the real boxes of the acquired dataset to obtain anchor boxes more suitable for this scenario, overcoming the limitations of the original algorithm using K-Means clustering and the size differences of the anchor boxes obtained by clustering different datasets, and improving the positioning accuracy of the prediction boxes. Finally, the improved YOLOv3 network is trained using the selected training set to obtain a trained obstacle detection network model for complex weather conditions;

[0075] Specifically, it includes:

[0076] S21: The recognition accuracy is improved by introducing the SPP module. The SPP module is inserted into the ConvolutionalSet of the YOLOv3 convolutional block. The module structure of the SPP is as Figure 3 shown. The insertion position of the SPP module in the Convolutional Set is as Figure 4 shown. There are a total of four branches in the SPP module. The first branch is that the input is directly connected to the output branch. The second branch is the maximum pooling with a pooling kernel of 5×5. The third branch is the maximum pooling with a pooling kernel of 9×9. The fourth branch is the maximum pooling with a pooling kernel of 13×13. The pooling stride is 1 each time. In order to keep the size and depth of the finally obtained feature map unchanged, padding needs to be performed before pooling. Finally, the SPP module realizes the fusion of features at different scales.

[0077] The SPP module extracts local and global features through the idea of a spatial pyramid, improving the receptive field of the model. After the feature map in the SPP module fuses local and global features, it can enrich the expression ability of the feature map, which is beneficial for the situation where the target sizes in the dataset to be detected vary greatly, greatly improving the detection accuracy. Since in the classification network of the fully connected layer, it is strictly required that the input resolution matches the feature dimension of the fully connected layer. The SPP module can convert the FeatherMap with any resolution into a designed feature vector with the same dimension as the fully connected layer, avoiding the phenomenon of image distortion caused by operations such as cropping and scaling of the image area, and improving the detection accuracy;

[0078] S22: Replace the Leaky-ReLU activation function in the convolutional layer of the original YOLOv3 network with the ELU activation function, as Figure 5As shown. An appropriate activation function determines the model's ability to solve complex tasks. The activation function used in YOLOv3 is Leaky-ReLU, and its calculation formula is:

[0079]

[0080] where u and m represent the horizontal and vertical coordinates of the function respectively. However, the robustness of Leaky-ReLU is very poor. Therefore, the activation function used in each network convolutional layer is changed to ELU, and its calculation formula is:

[0081]

[0082] When m takes a positive value, the activation function ELU is the same as Leaky-ReLU. The difference is that when m is 0 or negative in ELU, the exponential function is used. As the parameter becomes smaller, the function gradually converges to a negative value. Convergence means having a small derivative value, reducing the changes and information propagated to the next layer. Therefore, the ELU activation function is more robust to noise.

[0083] S23: The obstacle detection algorithm based on deep learning needs to learn features from the dataset. Therefore, the selected dataset must be representative and extensive, covering complex weather conditions and road scenes. Therefore, the representative KITTI dataset and CODA dataset are selected. The KITTI dataset conforms to the definition and classification of road obstacles, but lacks complex weather scenes. Therefore, the CODA dataset is used to supplement the KITTI dataset to obtain a dataset that better conforms to the applicable scenario of the present invention. The obstacles are classified into car, van, truck, pedestrian, pedestrian(sitting), cyclist, tram, and misc for detection and classification. The complex weather part of the CODA dataset is expanded through data augmentation methods such as Mixup, Cutmix, and Cutout, and the corresponding labels are modified to conform to the above classification. Finally, the images of the dataset are uniformly resized to 416×416 pixels to facilitate the training of the subsequent YOLOv3 model;

[0084] S24: The clustering method is changed from the K-Means algorithm to the K-Means++ algorithm. The K-Means algorithm randomly selects k data points as the initial clustering centroids, and the result is easily affected by the initial value selection and can only find a local optimal solution. The K-Means++ algorithm is selected to solve the influence of the K-Means initial value selection on the clustering result. This algorithm will select data points one by one as centroids as much as possible to ensure the global optimal solution. First step, according to the value of K, randomly select a data point from the data set as the first initial centroid, and the value of K is the number of anchor points. The second step is to calculate the distances from other data points to the previous centroid. The third step is to select the data point farthest from the existing centroids as the next centroid. Similarly, if n initial centroids have been selected, then select the data point farthest from the current n centroids as the (n + 1)-th initial centroid. This process continues until K initial centroids are selected. The fourth step is to calculate the distances from all data points to each centroid and divide them into the nearest centroid as a class. The fifth step is to recalculate the clustering centroids for each class. Repeat the fourth and fifth steps until the change in the centroids is less than a threshold. The purpose of YOLOv3 clustering is to make the anchor boxes as close as possible to the ground truth boxes, so the distance calculation formula is:

[0085] d(box, centropd) = 1 - IOU(box, centroid)

[0086] box is the size of the rectangle, centroid is the center of the rectangle, and IOU is the intersection over union of the two rectangles.

[0087] Cluster the ground truth boxes in the data set, and finally obtain 9 anchor boxes corresponding to the new data set. The anchor boxes of YOLOv3 are nine boxes with different sizes obtained by using K-Means clustering on the ground truth boxes of the COCO data set, which can avoid the model blindly searching during training and contribute to the rapid convergence of the model. Use the K-Means++ algorithm to cluster the nine anchor boxes in the YOLOv3 network. The network outputs of three different depths correspond to three different sizes of feature maps, and each feature map corresponds to three anchor boxes;

[0088] S25: Train the detection model. Set up the experimental environment. On the deep learning framework, train the improved YOLOv3 network on the training set part of the obtained data set, and adjust the parameters accordingly according to the actual situation to obtain better results. Use the mean average precision (mAP) to evaluate the performance of the algorithm. mAP is the average of APs, and the calculation formula is as follows:

[0089]

[0090] mAP refers to the average of APs for each class, and AP ais the area under the precision and recall curves for the a-th category. k represents the total number of all categories, and a represents the serial number of the category. The AP value can describe the area of the precision-recall curve, and the calculation formula is as follows:

[0091] precision = TP / (TP + FP)

[0092] recall = TP / (TP + FN)

[0093] precision represents precision, recall represents recall, TP represents True Positives, FP represents False Positives, and FN represents False Negatives. The larger the value of the mean average precision mAP, the higher the overall recognition accuracy of the model. Finally, the test set is used to verify the detection effect of the model;

[0094] S3: Input the preprocessed enhanced images into the trained improved network to obtain detection and classification results, with the accuracy and robustness improved, realizing an obstacle detection method under complex weather conditions.

[0095] Specifically, it includes:

[0096] S31: Images of various scenarios under complex weather conditions are enhanced by MSRCR and then input into the previously trained improved YOLOv3 object detection network. Specifically, it detects and locates and classifies several predefined obstacles such as car, van, truck, pedestrian, pedestrian (sitting), cyclist, tram, and misc to obtain detection results, realizing an obstacle detection method under complex weather conditions.

[0097] Embodiment 2

[0098] In the second aspect, this embodiment provides an obstacle detection device under complex weather conditions, including a processor and a storage medium;

[0099] The storage medium is used to store instructions;

[0100] The processor is used to operate according to the instructions to execute the steps of the method according to Embodiment 1.

[0101] Embodiment 3

[0102] In the third aspect, this embodiment provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the steps of the method according to Embodiment 1.

[0103] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0105] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0107] As is known by common technical knowledge, the present invention can be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the above-disclosed embodiments are illustrative in all aspects and not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are encompassed by the present invention.

Claims

1. A method for obstacle detection under complex weather conditions, characterized in that, Including: Obtaining an image in a complex weather environment; Performing enhanced preprocessing on the image using the multi-scale retinex with color restoration (MSRCR) algorithm to obtain a preprocessed image; Inputting the preprocessed image into a trained obstacle detection model based on the improved YOLOv3 network; Determining the obstacle detection result in complex weather according to the output of the obstacle detection model based on the improved YOLOv3 network; Wherein the construction method of the obstacle detection model based on the improved YOLOv3 network includes: Inserting an SPP module into the Convolutional Set of YOLOv3 to achieve the fusion of features at different scales, extract more detailed information, and thus improve the detection accuracy; using the ELU activation function to replace the Leaky-ReLU activation function in the convolutional layer of the original YOLOv3 network to increase the robustness to noise; Obtaining a dataset for obstacle detection under different weather conditions to enhance the robustness of the training model; Using the K-Means++ algorithm to re-cluster the ground truth boxes in the dataset to obtain appropriate anchor boxes, accelerating the convergence speed of sample training and improving the localization accuracy; Training the obstacle detection model with the processed dataset to obtain a trained obstacle detection model based on the improved YOLOv3 network; Wherein, performing enhanced preprocessing on the image using the multi-scale retinex with color restoration (MSRCR) algorithm includes: Using the weighted sum of several different Gaussian scale parameters, taking the Gaussian filtering result as the estimated illumination image, and adding a color restoration factor to restore the color of the reflected image to avoid color degradation. The calculation formula is as follows: Among them, F MSRCR (x, y) is the enhancement result processed by the MSRCR algorithm. x and y represent the abscissa and ordinate of the image pixels respectively. i represents the serial number of the color component, j represents the serial number of the scale parameter, C i is the color restoration factor of the i-th color component, N is the number of scale parameters, ω j is the weight corresponding to different scales, I i (x, y) represents the i-th color component of the input image, G j (x, y) represents the Gaussian filter at the scale of σ j , σ j represents the scale parameter of the j-th Gaussian surround, is the convolution symbol, α is the nonlinear intensity adjustment parameter, and β is the gain factor.

2. The method for obstacle detection under complex weather conditions according to claim 1, characterized in that, N = 3, representing three scales: small, medium, and large.

3. The method for obstacle detection under complex weather conditions according to claim 1, characterized in that, There are a total of four branches in the SPP module: the first branch directly connects the input to the output branch, the second branch is the max pooling with a pooling kernel of 5×5, the third branch is the max pooling with a pooling kernel of 9×9, and the fourth branch is the max pooling with a pooling kernel of 13×13. The pooling stride is 1 each time, and padding is performed before pooling to keep the size and depth of the finally obtained feature map unchanged; ultimately, the SPP module realizes the fusion of features at different scales.

4. The method for obstacle detection under complex weather conditions according to claim 1, characterized in that, Using the ELU activation function to replace the Leaky-ReLU activation function in the convolutional layer of the original YOLOv3 network includes: The ELU activation function, the calculation formula is: u and m respectively represent the horizontal and vertical coordinates of the function. The gradient of the ELU activation function is non-zero for all negative values, and there is no problem of neuron death. That is, when using the ELU activation function for abnormal inputs, a large gradient will not be generated during backpropagation, so it will not cause neuron death and gradient disappearance, and can shorten the training time and improve the accuracy in training the network; when m in ELU is 0 or a negative value, the exponential function is used, and as the parameter becomes smaller, the function gradually converges to a negative value; convergence means having a small derivative value, reducing the change and information propagated to the next layer; therefore, the ELU activation function is more robust to noise and can reduce the impact of image noise on the detection result.

5. The method for obstacle detection under complex weather conditions according to claim 1, characterized in that, Obtain a dataset for obstacle detection under different weather conditions to improve the robustness of the training model, including: selecting representative KITTI dataset and CODA dataset, supplementing the KITTI dataset after modifying the corresponding labels of the CODA dataset to obtain a dataset for obstacle detection under different weather conditions; reserving a test set from the obtained dataset, and using the remaining dataset as the training set to increase the robustness of the training model.

6. The method for obstacle detection under complex weather conditions according to claim 5, characterized in that, Supplement the KITTI dataset after modifying the corresponding labels of the CODA dataset, including: Classify the obstacle labels into categories such as car, van, truck, pedestrian, pedestrian(sitting), cyclist, tram, and misc for detection and classification; Expand the complex weather part of the CODA dataset through data augmentation methods such as Mixup, Cutmix, and Cutout, and modify the corresponding labels to conform to the classification of the obstacle labels; Uniformly resize the images in the dataset to 416×416 pixels to facilitate the subsequent training of the obstacle detection model based on the improved YOLOv3 network.

7. The method for obstacle detection under complex weather conditions according to claim 1, characterized in that, Train the obstacle detection model with the processed dataset, including: Train the improved YOLOv3 network on the training set part of the dataset, and adjust the parameters accordingly according to the actual situation; use the average value mAP of the APs of each category to evaluate the performance of the model, and the calculation formula is as follows: mAP refers to the average value of APs for each category. AP a is the area under the precision and recall curve for the a-th category, k represents the total number of all categories, and a represents the serial number of the category; the larger the value of the average precision mAP, the higher the overall recognition accuracy of the model. The reserved test set is used to verify the effect, and the trained obstacle detection model based on the improved YOLOv3 network is obtained.

8. An obstacle detection device under complex weather conditions, characterized in that Including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images