A method for identifying night visibility levels based on feature extraction

Abstract

The present invention discloses a method for identifying the night visibility level based on feature extraction, which includes: obtaining a night color image; extracting the luminance histogram data of the night color image, and obtaining a luminance feature vector through three-layer MLP processing; extracting the saturation image data S of the night color image, merging the saturation image data S with the night color image data RGB to form four-channel data, and obtaining a feature vector of SRGB through convolutional neural network processing; after merging and connecting the luminance feature vector of the night color image and the SRGB feature vector, performing class judgment through a discriminator to achieve the identification of the night visibility level. The present invention uses a feature fusion method to extract multiple features for classifier training, realizes feature complementarity, and realizes the identification of the night visibility level in the case of fog. Since the visibility at night is not high and the image degradation is serious, by effectively extracting different features of the night fog image, the accuracy rate of visibility level classification will be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of road visibility detection, and more specifically, to a method for identifying night visibility levels based on feature extraction. Background Art

[0002] With the rapid development of economic construction, road transportation has become increasingly busy. Road traffic safety issues have become social problems, causing serious losses to people's lives and property. Major road traffic accidents caused by decreased visibility occur frequently, which has attracted the high attention of governments around the world.

[0003] Currently, video image-based visibility detection methods at home and abroad include camera model calibration method, template matching method, dark channel prior method, double brightness difference method, deep learning method, etc. The commonly used dark channel prior method obtains the transmittance from the target object to the camera point according to the dark channel prior theory, and uses the transmittance to derive the atmospheric extinction coefficient, and then estimates the visibility. Current research shows that the transmittance obtained by this method is not accurate enough, and the real-time performance of the optimized algorithm is not very good. The double brightness difference method calculates the visibility using the ratio of the background brightness differences of two objects at different distances near the horizon and their corresponding horizontal sky. The advantage of this method is that visibility detection can be carried out even at night, and the disadvantage is that artificial targets need to be set up. Fog scene images refer to natural scene pictures taken in foggy weather. From the perspective of the image spectrum, the degraded foggy weather images lose a large amount of high-frequency signals. Therefore, obtaining image features similar to the above through artificially designed feature engineering can be used as an important indicator for judging whether there is fog. However, such methods have relatively large defects in robustness and are easily affected by non-uniform fog and moving targets. Moreover, there are relatively many studies on foggy weather visibility detection during the day, while foggy weather visibility detection at night is a relatively less mentioned aspect at the present stage. In fact, the visibility of the atmosphere itself is not high at night. When there is fog, serious traffic accidents are more likely to occur due to low visibility, resulting in irreparable consequences. In recent years, with the promotion of new generation technologies such as artificial intelligence algorithms, 5G, and big data, image perception and understanding technologies based on deep learning algorithms have been widely applied in various fields.

[0004] In the prior art, a method for detecting night visibility and classifying and warning visibility levels is disclosed, which belongs to the technical field of intelligent vehicle safety auxiliary driving. This method constructs a night color image classifier, different night visibility classifiers, a night color image visibility detection model based on the Allard's law for completely dark night, and a non-completely dark night visibility warning level classification model based on supervised learning, calculates the visibility, and combines the driving speed and the distance to the vehicle ahead for warning. During actual driving, road video images during vehicle travel are collected. According to the offline-trained image classification library, it is determined in real time whether the image is a night image and a completely dark image. If it is a completely dark image, the night visibility is calculated in real time. If it is a non-completely dark image, the night visibility warning level is classified based on supervised learning, and the driving speed and the distance to the vehicle ahead are combined for warning to achieve night visibility warning and reduce the impact of low night visibility on the driver's vision. This solution cannot well handle the visibility assessment problems in foggy and non-foggy nights. Summary of the Invention

[0005] The present invention provides a method for identifying night visibility levels based on feature extraction, which realizes the identification of night visibility levels.

[0006] To solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A method for identifying night visibility levels based on feature extraction includes the following steps:

[0008] Obtain a night color image;

[0009] Extract the luminance histogram data of the night color image and process it to obtain the luminance feature vector of the night color image;

[0010] Extract the saturation image data S of the night color image, and process the saturation image data S together with the night color image data to obtain the SRGB feature vector of the night color image, where the SRGB feature vector refers to saturation S + color RGB feature vector;

[0011] After combining and connecting the luminance feature vector and the SRGB feature vector of the night color image, perform category judgment through a discriminator to realize the identification of night visibility levels.

[0012] Preferably, the colored night color images include non-foggy night images and foggy night images, and the foggy images are divided into three levels: light fog, medium fog, and heavy fog according to the visibility distance.

[0013] Preferably, the processing to obtain the luminance feature vector of the night color image is specifically as follows:

[0014] Input the luminance histogram data of the night color image into a multi-layer perceptron (MLP), and the MLP outputs the luminance feature vector of the night color image.

[0015] Preferably, the MLP includes an input layer, a hidden layer, and an output layer, which are fully connected between different layers. The training parameters of the MLP are: dropout is 0.5, the activation function is relu, and there are no other parameters. Finally, an n*1-dimensional luminance feature vector is obtained.

[0016] Preferably, the extraction of the saturation image data of the night color image is specifically as follows:

[0017] S = max(R, G, B) - min(R, G, B)

[0018] In the formula, S is the saturation image data, and R, G, and B are the values corresponding to the red, green, and blue channels of the night color image respectively.

[0019] Preferably, before jointly processing the saturation image data S and the night color image data, the saturation image data S and the night color image data are respectively subjected to size normalization to keep the sizes of the saturation image data and the night color image consistent and meet the requirements of the input image resolution of the subsequent convolutional neural network model.

[0020] Preferably, the joint processing of the saturation image data and the night color image is specifically as follows:

[0021] In addition to the RGB three channels of the night color image, the input data adds the saturation image data S as the fourth channel, and the input data is input into a convolutional neural network (CNN), and the CNN outputs the SRGB feature vector of the night color image.

[0022] Preferably, the convolutional neural network includes 4 convolutional layers and 2 fully connected layers. The input of the convolutional neural network is 4-channel image data after size normalization. After feature extraction by the convolutional neural network, an n*1-dimensional SRGB feature vector is obtained.

[0023] Preferably, the merging and connection of the luminance feature vector and the SRGB feature vector of the night color image is specifically as follows:

[0024] Use the concat method to merge and connect the luminance feature vector and the SRGB feature vector of the night color image to obtain an n*2-dimensional feature vector.

[0025] Preferably, the category judgment is performed through a discriminator, specifically as follows:

[0026] The discriminator is constructed using the global average pooling method. The discriminator includes 3 convolutional layers and 1 global average pooling layer. The output of the discriminator is then passed through the softmax function to calculate the probabilities of the sample corresponding to each category, where the categories include sunny, light fog, moderate fog, and heavy fog.

[0027] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0028] The present invention uses a feature fusion method to simultaneously extract multiple features for classifier training, achieving feature complementarity and reducing the influence of the inherent defects of a single feature, thereby realizing the recognition of the night visibility level in the presence of fog. Since the visibility at night is inherently low and the image degradation is severe, by effectively extracting different features of the night fog image, the accuracy of visibility level classification will be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0030] Figure 2 (a) is a night fog image, Figure 2 (b) is Figure 2 the brightness histogram of (a), Figure 2 (c) is a night fog-free image, Figure 2 (d) is Figure 2 the brightness histogram of (c).

[0031] Figure 3 (a) is a night fog image, Figure 3 (b) is Figure 3 the saturation image of (a), Figure 3 (c) is a night fog-free image, Figure 3 (d) is Figure 3 the saturation image of (c).

[0032] Figure 4 It is a schematic diagram of the night visibility level recognition network framework provided by the embodiment.

[0033] Figure 5 It is a schematic diagram of the MLP three-layer network structure.

[0034] Figure 6 It is a schematic diagram of the CNN network model.

[0035] Figure 7 It is a schematic diagram of images of different night visibility levels.

[0036] Figure 8 It is a schematic diagram of the comparison results of the block experiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The accompanying drawings are only for illustrative purposes and should not be construed as limiting the present patent;

[0038] To better illustrate this embodiment, some components in the accompanying drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product;

[0039] For those skilled in the art, it is understandable that some well-known structures and their descriptions in the accompanying drawings may be omitted.

[0040] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] Embodiment 1

[0042] This embodiment provides a method for identifying the night visibility level based on feature extraction, as Figure 1 shown, including the following steps:

[0043] Obtain a night color image;

[0044] Extract the luminance histogram data of the night color image, and process it to obtain the luminance feature vector of the night color image;

[0045] Extract the saturation image data S of the night color image, and process the saturation image data S together with the night color image data to obtain the SRGB feature vector of the night color image;

[0046] After merging and connecting the luminance feature vector and the SRGB feature vector of the night color image, perform class judgment through a discriminator to achieve the identification of the night visibility level.

[0047] In the problem of image recognition, a feature is usually only sensitive to the changes in some characteristics of the image, and not sensitive to the changes in other characteristics. Therefore, when the difference between two types of images is not significant in a certain feature-sensitive characteristic, the classifier trained based on a single feature cannot output the correct classification. In addition, the complex background noise in the image will also lead to a decline in the quality of the feature data, increasing both the difficulty of classifier training and reducing the accuracy of classification. One way to solve this problem is to use the method of feature fusion, extract multiple features at the same time for classifier training, achieve feature complementarity, and reduce the impact of the inherent defects of a single feature. Since the visibility at night is not high and the image degradation is serious, if different features of the night fog image can be effectively extracted at this time, the accuracy of visibility level classification will be greatly improved. Therefore, this embodiment proposes a method for identifying the night visibility level based on feature extraction, using a deep learning network model to identify the night visibility level, and the deep learning network model includes two parts: night fog image feature extraction and deep learning network model.

[0048] Embodiment 2

[0049] On the basis of Embodiment 1, this embodiment further discloses the following content:

[0050] The colored night-time color images include night-time fog-free images and night-time foggy images, and the foggy images are divided into three levels: light fog, medium fog, and heavy fog according to the visibility distance.

[0051] This embodiment utilizes the saturation feature of the night-time color image and obtains the brightness distribution feature of the night-time color image by using the brightness histogram; by fusing the original RBG image, the saturation feature, and the brightness feature, a night-time visibility level recognition network is constructed, and the network framework is as Figure 4 shown.

[0052] In the night-time visibility level recognition network framework, there are three modules. Module 1 is used to obtain the brightness feature vector, Module 2 is used to fuse the original RBG image and the saturation feature to obtain the SRBG feature vector, and Module 3 is used to obtain the night-time visibility level according to the brightness feature vector and the SRBG feature vector.

[0053] The processing to obtain the brightness feature vector of the night-time color image is specifically as follows:

[0054] Input the brightness histogram data of the night-time color image into the multi-layer perceptron MLP, and the multi-layer perceptron MLP outputs the brightness feature vector of the night-time color image.

[0055] As the fog concentration level deepens, the distribution range of the brightness histogram feature of the image gradually expands on the horizontal axis, as Figure 2 shown, presenting an obvious non-linear feature, and the MLP multi-layer perceptron MLP is a forward-structured artificial neural network, which can effectively handle non-linearly separable problems. Therefore, the MLP network can be used to process the brightness histogram data.

[0056] Since there is only the brightness feature in the feature vector, in order to control the number of parameters and prevent overfitting, this embodiment uses the most typical three-layer MLP, as Figure 5 shown. The multi-layer perceptron MLP includes an input layer, a hidden layer, and an output layer, and they are fully connected between different layers. The training parameters of the multi-layer perceptron MLP are: dropout is 0.5, the activation function is relu, and there are no other parameters. Finally, an n*1-dimensional brightness feature vector is obtained. In this embodiment, n is 1024.

[0057] In the case of fog-free weather, the brightness attenuation of the light source in the night-time color image is very obvious, as Figure 2 (c) shown, and the street light source in the figure has a relatively clear boundary; in the case of foggy weather, the fog will cause the brightness attenuation rate of the light source to slow down, and at the same time, the light intensity of the light source itself decreases, and the attenuation starts earlier, as Figure 2(a) As shown. Through the above analysis, the first feature for classifying night visibility levels - the brightness histogram feature - can be summarized as follows. Figure 2 As shown, it can be seen from the figure that the brightness histogram of the foggy image is widely distributed within the horizontal axis range, as Figure 2 (b) As shown, while the brightness histogram of the fog-free image is relatively concentrated, as Figure 2 (d) As shown.

[0058] The saturation image data of the night color image is extracted as follows:

[0059] S = max(R, G, B) - min(R, G, B)

[0060] In the formula, S is the saturation image data, and R, G, and B are the values corresponding to the red, green, and blue channels of the night color image respectively.

[0061] The saturation of the image reflects the vividness of the colors in the image. From the calculation formula, it can be seen that the saturation is the difference between the maximum and minimum values among the RGB values. Therefore, the saturation reflects the degree of dispersion of the RGB values. The higher the degree of dispersion, the higher the saturation.

[0062] The weather conditions on foggy days will cause the disappearance of the detailed contours in the image and the overall whitening, resulting in a lower degree of dispersion of the RGB values and a consequent decrease in the saturation of the image, as Figure 3 As shown. It can be seen from the figure that there is an obvious difference in the saturation between the foggy image and the fog-free image. Therefore, the second feature for classifying night visibility levels - the saturation feature - can be obtained.

[0063] Before processing the saturation image data S and the night color image data together, the saturation image data S and the night color image data are respectively normalized to keep the size of the saturation image data consistent with that of the night color image and meet the requirements of the input image resolution of the subsequent convolutional neural network model.

[0064] The processing of the saturation image data and the night color image together is as follows:

[0065] To solve the problem of degradation of the night color image, in addition to the RGB three channels of the night color image, the saturation image data S is added as the fourth channel to the input data, and the input data is input into the convolutional neural network, and the convolutional neural network outputs the SRGB feature vector of the night color image.

[0066] Since the image information in the night scene is more single than that in the day image (in most cases, there are only black - gray and white light sources in the image), in order to prevent over - fitting and improve the network processing speed to achieve real - time detection, the convolutional neural network designed in this embodiment includes 4 convolutional layers and 2 fully - connected layers, as Figure 6 shown, the input of the convolutional neural network is 4 - channel image data after size normalization, with a size of 4 * 448 * 448. After feature extraction by the convolutional neural network, a 1024 * 1 - dimensional SRGB feature vector is obtained.

[0067] The merging and connection of the luminance feature vector and the SRGB feature vector of the night - time color image are specifically as follows:

[0068] The luminance feature vector and the SRGB feature vector of the night - time color image are merged and connected by using the concat method to obtain a 1024 * 2 - dimensional feature vector.

[0069] In order to better control the number of parameters, the present invention adopts the GAP (Global average pooling) method to construct a discriminator. The category judgment is performed by the discriminator, specifically as follows:

[0070] The discriminator is constructed by using the global average pooling (GAP) method. The discriminator includes 3 convolutional layers and 1 global average pooling layer. The output of the discriminator is further calculated by the softmax function to obtain the probabilities of the samples corresponding to each category. The categories include clear sky, light fog, moderate fog, and heavy fog.

[0071] Embodiment 3

[0072] This embodiment provides model verification, specifically as follows:

[0073] The hardware development environment of this experiment consists of a single computer with two Intel processors, and the specific parameters are Xeon - E5 CPU, 4 × TITAN RTX GPU, and 64GB RAM. The experiment is implemented based on Pytorch - 1.4.1 and CUDA - 10.0.

[0074] Referring to the national standard GB / T 31445 - 2015, the night - time fog concentration is divided into 4 levels, as shown in Table 1. According to the division of visibility levels in Table 1, a corresponding night - time visibility level image test library is established, as Figure 6 shown. All images in the test library are from actual highway shooting scenes, and the original size of the images is 1920 * 1080. In the entire dataset, the training set is 4000 pictures of 4 categories, 1000 pictures for each category, and the test set is 1600 pictures of 4 categories, 400 pictures for each category.

[0075] Table 1 Classification Criteria for Nighttime Fog Concentration Levels

[0076]

[0077] (1) Block Experiments

[0078] The purpose of this experiment is to verify the effects of each module and the performance advantages of the multi - feature fusion model compared with the single - feature model. In this experiment, the same training set and test set are used, and three classification models are trained simultaneously: the brightness histogram model (Module 1), the RGBS four - channel model (Module 2), and the multi - feature fusion model (Module 1 + 2 + 3). The main training parameters include: (1) The learning rate is 0.001. The decay is 0.0005 per 7 epochs; the SGD optimization method is used with a momentum of 0.9; the loss function is CrossEntropy; the data pre - processing method is: perform random rotation from - 20 to 20 degrees.

[0079] The experimental results are as Figure 8 shown. It can be seen from the figure that the deep - learning network model based on the fusion of RGBS four - channel and brightness histogram features (Module 1 + 2 + 3) effectively combines the advantages of the RGBS four - channel model (Module 2) and the brightness histogram model (Module 1). In the case of only using the RGBS four - channel model (Module 2), the accuracies for light fog, medium fog, and heavy fog are 0.904, 0.947, and 0.926 respectively. While in the case of only using the brightness histogram model (Module 1), the accuracies for light fog, medium fog, and heavy fog are 1, 0.626, and 0.58 respectively. The RGBS four - channel and brightness histogram feature fusion model (Module 1 + 2 + 3) fully absorbs the judgment advantages of both, and the accuracies for light fog, medium fog, and heavy fog are 0.953, 0.92, and 0.93 respectively.

[0080] (2) Performance Comparison Experiment with Classical Algorithms

[0081] The purpose of this experiment is to compare the algorithm proposed in the present invention with classical image classification models. The network models participating in the comparison include Inception V4, Inception_resnet V2, and EfficientNet_B7. The input image sizes of each different model are the standard sizes of their respective networks, obtained by resizing the original images. Table 2 shows the recognition results. It can be seen from Table 2 that compared with classical network models, the present invention has the best classification performance in the problem of nighttime visibility level recognition, and the present invention has a smaller number of parameters and also has an absolute advantage in processing speed.

[0082] Table 2 Performance Comparison of Different Networks for Nighttime Visibility Level Recognition

[0083]

[0084] Like or corresponding reference numerals designate like or corresponding parts;

[0085] The terms used in the drawings to describe the positional relationships are for illustrative purposes only and should not be construed as limiting the present patent;

[0086] Obviously, the above embodiments of the present invention are merely examples given for clearly illustrating the present invention, rather than limiting the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made on the basis of the above description. It is not necessary and impossible to list all implementation manners here. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A method for identifying night visibility levels based on feature extraction, characterized in that, It includes the following steps: Obtain a nighttime color image; Extract the luminance histogram data of the nighttime color image, and process it to obtain the luminance feature vector of the nighttime color image. Specifically: Input the luminance histogram data of the nighttime color image into a multi-layer perceptron MLP, and the multi-layer perceptron MLP outputs the luminance feature vector of the nighttime color image; Extract the saturation image data S of the nighttime color image, and process the saturation image data S together with the nighttime color image data to obtain the SRGB feature vector of the nighttime color image; After merging and connecting the luminance feature vector and the SRGB feature vector of the nighttime color image, perform class judgment through a discriminator to achieve nighttime visibility level recognition; The processing of the saturation image data together with the nighttime color image specifically is: In addition to the RGB three channels of the nighttime color image, the input data adds the saturation image data S as the fourth channel, and input the input data into a convolutional neural network, and the convolutional neural network outputs the SRGB feature vector of the nighttime color image.

2. The method for identifying the night visibility level based on feature extraction according to claim 1, wherein The nighttime color image includes a nighttime fog-free image and a nighttime foggy image, and the foggy images are divided into three levels: light fog, medium fog, and heavy fog according to the visibility distance.

3. The method for identifying night visibility level based on feature extraction according to claim 2, wherein The multi-layer perceptron MLP includes an input layer, a hidden layer, and an output layer, and they are fully connected between different layers. The training parameters of the multi-layer perceptron MLP are: dropout is 0.5, the activation function is relu, there are no other parameters, and finally an n*1-dimensional luminance feature vector is obtained.

4. The method for identifying the night visibility level based on feature extraction according to claim 2, wherein The extraction of the saturation image data of the nighttime color image specifically is: S = max(R, G, B) - min(R, G, B) In the formula, S is the saturation image data, and R, G, and B are the values corresponding to the red, green, and blue channels of the nighttime color image respectively.

5. The method for identifying night visibility level based on feature extraction according to claim 4, wherein Before processing the saturation image data S together with the nighttime color image data, size normalization is respectively performed on the saturation image data S and the nighttime color image data to keep the sizes of the saturation image data and the nighttime color image consistent and meet the requirements of the input image resolution of the subsequent convolutional neural network model.

6. The method for identifying the night visibility level based on feature extraction according to claim 5, characterized in that The convolutional neural network includes 4 convolutional layers and 2 fully connected layers. The input of the convolutional neural network is the 4-channel image data after size normalization. After feature extraction by the convolutional neural network, an n*1-dimensional SRGB feature vector is obtained.

7. The method for identifying the night visibility level based on feature extraction according to claim 1, wherein The merging and connecting of the luminance feature vector and the SRGB feature vector of the nighttime color image specifically is: Use the concat method to merge and connect the luminance feature vector and the SRGB feature vector of the nighttime color image to obtain an n*2-dimensional feature vector.

8. The method for identifying night visibility level based on feature extraction according to claim 7, characterized in that, The class judgment through the discriminator specifically is: Adopt the global average pooling method to construct the discriminator. The discriminator includes 3 convolutional layers and 1 global average pooling layer. The output of the discriminator is then calculated by the softmax function to obtain the probabilities of the samples corresponding to each class. The classes include clear sky, light fog, medium fog, and heavy fog.

Images