A foggy-day driving obstacle detection method and device

Abstract

The application discloses a foggy-day driving obstacle detection method and device, belongs to the technical field of intelligent transportation, and aims to solve the technical problem that the existing foggy-day target detection algorithm can cause image to lose detailed information, and thus leads to low detection accuracy. The method comprises the following steps: performing edge feature extraction on each sunny-day image in a preset data set to obtain an edge feature map of each sunny-day image; constructing a training data set based on the preset data set and the edge feature map; constructing an edge-convolution feature fusion network, inputting each sunny-day image in the training data set and the corresponding edge feature map into the edge-convolution feature fusion network, and obtaining a corresponding feature fusion map; constructing each sunny-day image in the training data set and the corresponding feature fusion map into an optimization data set, training an obstacle detection model, optimizing the trained obstacle detection model, and performing obstacle target detection on a to-be-detected foggy-day image after fog removal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, and in particular to a foggy-day driving obstacle detection method and device. BACKGROUND

[0002] In recent years, the development of assisted driving technology has provided strong guarantee for driving safety, but there are still many problems in the face of bad weather environment. Automatic driving needs to accurately perceive the driving environment, and obstacle detection based on visual sensors is one of the main means of environmental perception. However, affected by bad weather, especially in foggy conditions, the image quality decreases, and the information loss after image dehazing is serious, which increases the complexity of detection and even leads to traffic accidents.

[0003] Driving obstacle detection in foggy conditions mainly includes two methods: combining image dehazing algorithm and target detection algorithm for detection, and training and detecting by fusing features of sunny and foggy images. The degradation of image quality in foggy conditions is the most direct reason affecting target detection, so combining dehazing algorithm with target detection algorithm is the most commonly used method. However, a large amount of color, texture and other detailed information is lost after image dehazing, which restricts the improvement of obstacle detection performance. Another method of training and detecting by fusing features of sunny and foggy images improves the detection speed, but the detection accuracy is low. SUMMARY

[0004] The embodiments of the present application provide a foggy-day driving obstacle detection method and device, which are used to solve the following technical problem: the existing foggy-day target detection algorithm will cause the image to lose detailed information, resulting in low detection accuracy.

[0005] The embodiments of the present application adopt the following technical solutions:

[0006] On the one hand, the embodiments of the present application provide a foggy-day driving obstacle detection method, which comprises: performing edge feature extraction on each sunny image in a preset data set to obtain an edge feature map of each sunny image; wherein the sunny image contains an obstacle target; based on the preset data set and the edge feature map, a training data set is constructed, and an obstacle detection model is trained through the training data set; an edge-convolution feature fusion network is constructed, and each sunny image in the training data set and the corresponding edge feature map are input into the edge-convolution feature fusion network to obtain a corresponding feature fusion map; each sunny image in the training data set and the corresponding feature fusion map are constructed into an optimized data set, and the obstacle detection model is optimized; and the obstacle target in a to-be-detected foggy image after dehazing processing is detected by the optimized obstacle detection model.

[0007] In an implementable embodiment, edge feature extraction is performed on each sunny day image in the preset data set to obtain an edge feature map of each sunny day image, specifically including: extracting all sunny day images in the KITTI data set to constitute the preset data set; performing edge feature extraction on each sunny day image in the preset data set by an edge feature extraction model containing a sobel operator to obtain a corresponding edge feature map.

[0008] In an implementable embodiment, a training data set is constructed based on the preset data set and the edge feature map, specifically including: calculating a global optimal threshold ODS of the edge feature map; performing binaryzation processing on each edge feature map according to the global optimal threshold ODS to obtain a binary edge map; wherein an element value greater than the global optimal threshold ODS in the edge feature map is set to 1, and an element value less than or equal to the global optimal threshold ODS is set to 0; counting the number of elements with an element value of 1 in each binary edge map to obtain an edge feature point evaluation value; if the edge feature point evaluation value is greater than a preset threshold, the corresponding edge feature map is retained, and the retained edge feature map and the corresponding sunny day image constitute the training data set.

[0009] In an implementable embodiment, an edge-convolution feature fusion network is constructed, specifically including: constructing the Focus structure, the CBL structure and the CSP1_X structure in the YOLOv5s network into a plurality of feature extraction layers in a preset order, and constructing the plurality of feature extraction layers and a pooling layer into a Backbone network; constructing a cascade module between the fifth feature extraction layer and the seventh feature extraction layer of the Backbone network; the Backbone network and the cascade module constitute the edge-convolution feature fusion network.

[0010] In an implementable embodiment, the method further includes: constructing a residual module in each CSP1_X structure in the Backbone network; wherein the residual module contains X residual blocks and an add operation layer.

[0011] In an implementable embodiment, each sunny day image in the training data set and the corresponding edge feature map are input into the edge-convolutional feature fusion network to obtain a corresponding feature fusion map, specifically including: inputting each sunny day image in the training data set and the corresponding edge feature map into the edge-convolutional feature fusion network in pairs; extracting the convolutional features of the obstacles in the input sunny day image through the CBL structure; performing an add superposition operation on the extracted convolutional features and the input edge feature map through the residual module in the CSP1_X structure to obtain a composite feature map; and concatenating and splicing the composite feature map output by the fifth feature extraction layer in the Backbone network with the up-sampling of the composite feature map output by the seventh feature extraction layer to obtain the feature fusion map.

[0012] In an implementable embodiment, each sunny day image in the training data set and the corresponding feature fusion map are constructed into an optimization data set, and the obstacle detection model is optimized, specifically including: training the obstacle detection model through the optimization data set; and performing parameter optimization on the weight parameters of each layer network in the obstacle detection model through a warm-up learning rate adjustment strategy during the training process until the learning rate of each layer network reaches the test requirement, thereby obtaining an optimized obstacle detection model.

[0013] In an implementable embodiment, before training the obstacle detection model through the training data set, the method further includes: building the obstacle detection model based on a YOLOv5s network and a pytorch framework; setting the iteration number, initial learning rate, periodic learning rate, momentum parameter and weight decay parameter of each layer network of the obstacle detection model; wherein the iteration number is uniformly set to 300, the initial learning rate is set to 0.01, the periodic learning rate is set to 0.2, the momentum parameter is set to 0.937, and the weight decay parameter is set to 0.0005.

[0014] In an implementable embodiment, the obstacle target detection is performed on the to-be-detected foggy day image after fog removal processing through the optimized obstacle detection model, specifically including: collecting a to-be-detected foggy day image during driving in a foggy day through a vehicle-mounted device; performing fog removal processing on the to-be-detected foggy day image through a gated context aggregation network (GCANet) to obtain a corresponding de-fogging image; inputting the de-fogging image into the optimized obstacle detection model for detection; if an obstacle target is detected, labeling the obstacle target in the de-fogging image through a label box to obtain an obstacle target detection result image.

[0015] In another aspect, the embodiments of the present application also provide a foggy-day driving obstacle detection device. The device comprises at least one processor, and a memory connected in communication with the at least one processor. The memory stores instructions executable by the at least one processor, so that the at least one processor can execute the foggy-day driving obstacle detection method according to any of the above embodiments.

[0016] Compared with the prior art, the foggy-day driving obstacle detection method and device provided by the present application has the following beneficial effects:

[0017] 1. By comparing and analyzing the differences in edge features of images before and after defogging by DCP, AOD-Net and GCA Net, it is found that the GCA Net algorithm can still retain good edge features after defogging the image. Therefore, the GCA Net algorithm is selected to defog the foggy-day image. From the perspective of changes in image features after defogging, a new idea is opened up to solve the problem of limited detection accuracy caused by information loss after defogging.

[0018] 2. In order to make full use of the retained edge features after defogging the image, a model training method of edge and convolution feature fusion is proposed. By fusing the features to adjust the parameter weight of the obstacle detection model, the direct connection between the edge features in the image and the obstacle detection model is established, and on the basis of the traditional detection method after defogging, the detection accuracy of driving obstacles is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art without creative labor. In the drawings:

[0020] Figure 1 A flowchart of a foggy-day driving obstacle detection method provided by the embodiments of the present application;

[0021] Figure 2 A schematic diagram of a Backbone network structure provided by the embodiments of the present application;

[0022] Figure 3 A schematic diagram of an edge-convolution feature fusion network structure provided by the embodiments of the present application;

[0023] Figure 4 A schematic diagram of a foggy-day driving obstacle detection device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0024] In order for those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0025] The embodiment of the present application provides a foggy driving obstacle detection method, as shown in the figure, the foggy driving obstacle detection method specifically comprises steps S101-S106: Figure 1

[0026] S101, edge feature extraction is performed on each sunny day image in the preset data set to obtain an edge feature map of each sunny day image.

[0027] Specifically, the main purpose of automatic driving obstacle detection is to detect most vehicles and pedestrians, as well as electric vehicles and bicycles and other non-motor vehicles, which requires training data to meet the special requirements of its task. In the present application, KITTI data set is selected as the basic training data, and the images in KITTI data set contain vehicle, pedestrian, non-motor vehicle and other obstacle targets. First, all sunny day images in KITTI data set are extracted to form a preset data set.

[0028] Further, the edge feature extraction model containing the sobel operator is used to extract the edge features of each sunny day image in the preset data set to obtain the corresponding edge feature map.

[0029] S102, based on the preset data set and the edge feature map, a training data set is constructed, and an obstacle detection model is trained through the training data set.

[0030] Specifically, the global optimal threshold value ODS of the edge feature map is calculated by the existing optimal dataset scale (ODS) calculation method. According to the calculated global optimal threshold value ODS, each edge feature map is binarized to obtain a binary edge map. In the edge feature map, the element value greater than the global optimal threshold value ODS is set to 1, and the element value less than or equal to the global optimal threshold value ODS is set to 0.

[0031] Further, the number of elements with element value of 1 in each binary edge map is counted to obtain an edge feature point evaluation value. If the edge feature point evaluation value is greater than a preset threshold value, the corresponding edge feature map is retained, and the retained edge feature map and the corresponding sunny day image form a training data set.

[0032] ​In one embodiment, the edge feature map obtained after feature extraction is a matrix, and the elements in the matrix take values ​​between 0 and 1. Based on the calculated ODS, elements in the matrix with values ​​less than or equal to ODS are set to 0, and elements with values ​​greater than ODS are set to 1, thereby retrieving the edge features. Figure Two Binarization. Then, count the number of 1s in the binarized image. If the number of 1s is greater than a preset threshold, it proves that the edge feature map corresponding to the binary image contains relatively rich feature points. Then, select this edge feature map and the corresponding original sunny day image and add them to the training dataset.

[0033] Furthermore, an obstacle detection model was built based on the YOLOv5s network and the PyTorch framework. The iteration count, initial learning rate, cyclic learning rate, momentum parameter, and weight decay parameter for each layer of the obstacle detection model were set. The built obstacle detection model was then trained using a training dataset.

[0034] In one embodiment, the number of iterations was uniformly set to 300, the initial learning rate was set to 0.01, the periodic learning rate was set to 0.2, the momentum parameter was set to 0.937, and the weight decay parameter was set to 0.0005. 5985 sunny daytime images rich in edge feature points and their corresponding edge images were selected and input into the obstacle detection model for training.

[0035] S103. Construct an edge-convolution feature fusion network.

[0036] Specifically, unlike other object detection tasks, vehicle obstacle detection requires higher time efficiency. This means that in addition to high detection accuracy, fast detection speed is also required. Therefore, by comparing the characteristics of various feature extraction networks, the Focus structure and CSP1_X structure in the YOLOv5s network are selected to extract the convolutional features of obstacles. In this application, the Focus structure, CBL structure, and CSP1_X structure in the YOLOv5s network are constructed into several feature extraction layers in a preset order, and these feature extraction layers are combined with a pooling layer to form a Backbone network. Then, a cascaded module is constructed between the fifth and seventh feature extraction layers of the Backbone network. The Backbone network and the cascaded module constitute an edge-convolutional feature fusion network.

[0037] As a feasible implementation, a residual module is constructed in each CSP1_X structure in the Backbone network; wherein the residual module contains X residual blocks and an add operation layer.

[0038] In one embodiment, Figure 2 A schematic diagram of a Backbone network structure provided in an embodiment of this application is shown below.Figure 2 As shown in the figure, the input of the Backbone network is a sunny image and a corresponding edge feature map, then the processed input is input into the CBL structure for convolution feature extraction, then the convolution feature and the edge feature map are input into the CSP1_X structure for superposition, then the repeated processing of the CBL structure and the CSP1_X structure is continued, and finally the complete Backbone network is formed after the pooling layer processing. Among them, as shown in the figure, Figure 2 As shown in the figure, the CSP1_X structure includes a CBL structure, a residual module and other structures, wherein the residual module includes X residual blocks and an add operation, and the convolution feature and the edge feature map are superimposed through the add operation to complete the composition of the feature map. In the Backbone network, each layer of the CSP1_X structure outputs a composite feature map, and in this application, the composite feature maps output by the fifth layer and the seventh layer of the corresponding CSP1_X structure are used for subsequent feature fusion processing.

[0039] In one embodiment, Figure 3 An edge-convolution feature fusion network provided in an embodiment of the present application is shown in the figure, Figure 3 As shown in the figure, the composite feature map output by the seventh layer of the Backbone network is spliced with the composite feature output by the fifth layer after upsampling through the concat operation. The CSP2_X structure is also introduced, and the CSP2_X structure includes 2X residual blocks.

[0040] The edge and convolution feature fusion in the present application specifically includes two implementation approaches of add and concat. The add operation is embodied in the process of extracting the convolution feature in the entire Backbone network and exists in the residual module. The corresponding convolution feature and the edge feature map are superimposed through the add operation to complete the composition of the feature map, and then the next layer operation is performed. On the premise of not changing the dimension, the information amount of the feature image is increased, that is, the information under each feature is increased. The concat belongs to the splicing of the feature map. For some output feature map layers (such as the fifth layer) in the Backbone network, the concat is spliced with the upsampling of the subsequent convolution layer, thereby expanding the dimension of the feature map and realizing feature fusion. Unlike the add operation, the concat operation realizes the fusion of multi-scale features and residual convolution layer information, increases the description features of a certain target in the image, and is not a single target feature. Through the add operation, the convolution features and edge feature information of each target in the sunny environment are increased, and then the target features under different environments are spliced through the concat to realize image feature fusion.

[0041] S104, input each sunny image in the training data set and the corresponding edge feature map into the edge-convolution feature fusion network to obtain a corresponding feature fusion image.

[0042] Specifically, each sunny day image in the training data set is input into the edge-convolution feature fusion network in pairs with the corresponding edge feature map. Through the CBL structure, the convolution features of the obstacles in the input sunny day image are extracted. Through the residual module in the CSP1_X structure, the extracted convolution features are added to the input edge feature map to obtain a composite feature map. The composite feature map output by the fifth feature extraction layer in the Backbone network is cascaded and spliced with the up-sampling of the composite feature map output by the seventh feature extraction layer to obtain a feature fusion map.

[0043] S105, each sunny day image in the training data set and the corresponding feature fusion map are constructed into an optimization data set, and the obstacle detection model is optimized.

[0044] Specifically, each sunny day image in the training data set and the corresponding feature fusion map are constructed into an optimization data set, and the obstacle detection model is trained through the optimization data set. In the training process, the weight parameters of each layer network in the obstacle detection model are parameter-optimized through the warm-up learning rate adjustment strategy until the learning rate of each layer network reaches the training requirement, and an optimized obstacle detection model is obtained. The training requirement can be a learning rate threshold set by the developer.

[0045] S106, the obstacle target detection is performed on the fog-processed to-be-detected foggy day image through the optimized obstacle detection model.

[0046] Specifically, in the automatic driving or unmanned driving scene, real-time images during driving are collected through a vehicle-mounted device. If the real-time image is a foggy day image, the foggy day image is first processed through the GCANet to obtain a corresponding de-fogging image. Then the de-fogging image is input into the optimized obstacle detection model for detection. If an obstacle target is detected, the obstacle target is labeled in the de-fogging image through a label box to obtain an obstacle target detection result image.

[0047] The application selects the GCANet algorithm for de-fogging processing. The key of the algorithm is to use a smooth dilated convolution instead of a dilated convolution, which not only solves the grid artifact problem, but also proposes a new fusion network to fuse features at different levels, does not depend on prior knowledge, and retains more image information after de-fogging, significantly improving the de-fogging effect of the image.

[0048] In addition, the application embodiment also provides a foggy day driving obstacle detection device, as shown in Figure 4 The device specifically comprises:

[0049] at least one processor; and a memory connected with the at least one processor in communication; wherein

[0050] The memory stores instructions capable of being executed by the at least one processor, so as to enable the at least one processor to perform:

[0051] Edge feature extraction is performed on each sunny image in the preset data set to obtain an edge feature map of each sunny image; wherein the sunny image contains an obstacle target;

[0052] Based on the preset data set and the edge feature map, a training data set is constructed, and an obstacle detection model is trained through the training data set;

[0053] An edge-convolution feature fusion network is constructed, and each sunny image in the training data set and the corresponding edge feature map are input into the edge-convolution feature fusion network to obtain a corresponding feature fusion map;

[0054] Each sunny image in the training data set and the corresponding feature fusion map are constructed into an optimization data set, and the obstacle detection model is optimized;

[0055] Through the optimized obstacle detection model, an obstacle target is detected in a to-be-detected foggy image after fog removal processing.

[0056] The foggy weather driving obstacle detection method and device provided in the application fuse obstacle edge features and convolution features when training obstacle detection model parameters, so that the foggy image can further utilize the edge features reserved after GCANet dehazing, thereby meeting the high-precision requirements of auxiliary driving and automatic driving obstacle detection while ensuring real-time performance. The application takes YOLOv5 network as a basic network, uses sunny pictures and corresponding edge feature pictures for obstacle detection model training, realizes edge feature and convolution feature fusion, and performs driving obstacle detection in a heavy fog traffic environment. The method can better recognize the image edge information remaining after dehazing, greatly improves the detection accuracy while ensuring time efficiency. This has important practical significance for improving driving obstacle detection in bad weather and improves the safety of automatic driving.

[0057] Each embodiment in the application is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments. Especially, the device embodiment is basically similar to the method embodiment, so the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0058] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of this application, other aspects of the application will become apparent from consideration of the drawings and following detailed description, it being understood that such changes in the details are within the scope of this application. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by those skilled in the art in light of the teachings and

[0059] The specific embodiments described herein have many advantages over conventional methods. Although only a limited number of embodiments of the application are described herein, it is understood that the application is capable of using numerous other possible arrangements without departing from the scope of the present application. Accordingly, the specific embodiments described herein are illustrative only and are not intended to limit the scope of the present application in any manner.

Claims

1. A foggy-day driving obstacle detection method, characterized by, The method comprises: Edge feature extraction is performed on each sunny image in a preset data set to obtain an edge feature map of each sunny image, wherein the sunny image contains an obstacle target; Based on the preset data set and the edge feature map, a training data set is constructed, and an obstacle detection model is trained through the training data set; An edge-convolution feature fusion network is constructed, and each sunny image in the training data set and the corresponding edge feature map are input into the edge-convolution feature fusion network to obtain a corresponding feature fusion map, specifically comprising: The Focus structure, the CBL structure and the CSP1_X structure in the YOLOv5s network are constructed into a plurality of feature extraction layers in a preset order, and the plurality of feature extraction layers and a pooling layer are constructed into a Backbone network; A cascade module is constructed between the fifth feature extraction layer and the seventh feature extraction layer in the Backbone network, and a residual module is constructed in each CSP1_X structure in the Backbone network; wherein the residual module contains X residual blocks and an add operation layer; The Backbone network and the cascade module constitute the edge-convolution feature fusion network; Each sunny image in the training data set and the corresponding edge feature map are input into the edge-convolution feature fusion network in pairs; The convolution features of the input sunny image are extracted through the CBL structure; The extracted convolution features and the input edge feature map are subjected to an add superposition operation through the residual module in the CSP1_X structure to obtain a composite feature map; The composite feature map output by the fifth feature extraction layer in the Backbone network is cascaded and spliced with the up-sampling of the composite feature map output by the seventh feature extraction layer to obtain the feature fusion map; Each sunny image in the training data set and the corresponding feature fusion map are constructed into an optimized data set, and the obstacle detection model is optimized; The obstacle target detection is performed on the to-be-detected foggy image after the haze removal processing through the optimized obstacle detection model.

2. The method according to claim 1, wherein Edge feature extraction is performed on each sunny image in a preset data set to obtain an edge feature map of each sunny image, specifically comprising: All sunny images in the KITTI data set are extracted to constitute the preset data set; Edge feature extraction is performed on each sunny image in the preset data set through an edge feature extraction model containing a sobel operator to obtain a corresponding edge feature map.

3. The method according to claim 2, wherein Based on the preset data set and the edge feature map, a training data set is constructed, specifically comprising: The global best threshold ODS of the edge feature map is calculated; Each edge feature map is subjected to binaryzation processing according to the global best threshold ODS to obtain a binary edge map; wherein the element value greater than the global best threshold ODS in the edge feature map is set to 1, and the element value less than or equal to the global best threshold ODS is set to 0; Count the number of elements with a value of 1 in each binary edge map to obtain an edge feature point evaluation value; If the edge feature point evaluation value is greater than a preset threshold, the corresponding edge feature map is retained, and the retained edge feature map and the corresponding sunny day image form the training data set.

4. The method according to claim 1, wherein Each sunny day image in the training data set and the corresponding feature fusion map are constructed into an optimization data set, and the obstacle detection model is optimized, specifically including: The obstacle detection model is trained through the optimization data set; During the training process, the weight parameters of each layer network in the obstacle detection model are optimized through the warm-up learning rate adjustment strategy until the learning rate of each layer network meets the test requirements, and the optimized obstacle detection model is obtained.

5. The method of claim 1, wherein the method further comprises: Before training the obstacle detection model through the training data set, the method further includes: The obstacle detection model is built based on the YOLOv5s network and the pytorch framework; The iteration number, initial learning rate, periodic learning rate, momentum parameter and weight decay parameter of each layer network of the obstacle detection model are set; The iteration number is uniformly set to 300, the initial learning rate is set to 0.01, the periodic learning rate is set to 0.2, the momentum parameter is set to 0.937, and the weight decay parameter is set to 0.0005.

6. The method of claim 1, wherein the method further comprises: Through the optimized obstacle detection model, the obstacle target detection is performed on the dehazed foggy day image to be detected, specifically including: Through the vehicle-mounted device, the foggy day image to be detected during driving in foggy weather is collected; Through the gated context aggregation network GCANet, the foggy day image to be detected is dehazed to obtain the corresponding dehazed image; The dehazed image is input into the optimized obstacle detection model for detection; If an obstacle target is detected, the obstacle target is labeled in the dehazed image through a label box to obtain an obstacle target detection result image.

7. A foggy-day driving obstacle detection device characterized by comprising: The device includes: At least one processor; and The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, so that the at least one processor can execute the obstacle detection method for driving in foggy weather according to any one of claims 1-6.

Images