A vehicle 24-color recognition method based on smooth neural network with multi-layer features

Abstract

The invention discloses a method for recognizing 24-color long-tail vehicles using a smooth neural network based on multi-layer features. The network includes four parts: feature extraction, multi-scale feature fusion, suggestion box generation, and smooth modulation. The network involves the image processing and target detection technology fields of machine learning, and can improve the vehicle color recognition problem in intelligent traffic management and criminal investigation auxiliary judgment. The specific scheme is as follows: first, a total of 100 hours of urban road monitoring video is collected, and 10091 images are intercepted. Each image contains multi-color vehicles, covering 24 color types, including red, dark red, pink, orange, dark orange, orange-red, yellow, lemon yellow, khaki, green, dark green, grass green, cyan, blue, dark blue, purple, black, white, silver gray, gray, dark gray, champagne, brown, and dark brown. The database is distributed in a long tail; secondly, the image is pre-processed by defogging, lighting adjustment, and other pre-processing operations; then, the scientific nature of the network setting is explained through layer-by-layer visualization and calcination experiments; finally, a model is trained for 24 types of long-tail vehicle color recognition. The present invention can improve the accuracy of vehicle color recognition and can detect and recognize in real time, thus better meeting the need for fine classification of vehicle colors.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and target detection of machine learning, and specifically relates to a vehicle 24-color long-tail recognition method under a smooth neural network based on multi-layer features and a self-built vehicle 24-color reference database. Background Art

[0002] At present, vehicle management mainly makes real-time decisions based on attributes such as license plate, color, model, and lines. If the license plate of a vehicle is replaced, tampered with, blocked, removed, or forged, the camera cannot capture the license plate in real time, and the license plate cannot be relied on to quickly identify the vehicle and make real-time decisions. Vehicle color, as another important attribute for obtaining vehicle information, can assist traffic law enforcement and improve the reliability of vehicle identification.

[0003] Due to the complexity of paint and color matching, vehicles have rich and personalized colors, with at least hundreds of colors. The current vehicle color recognition technology usually has only 8 color categories, so it is urgent to improve the fine classification of vehicle colors. However, changes in natural environments such as rain, fog, and light, accumulation of dust on the body, oxidation of the coating causing the color of the vehicle to degenerate and blur, and the long-tail effect in the data distribution of vehicle colors will lead to errors in color classification of vehicle images, thereby affecting the recognition rate and accuracy.

[0004] At present, the research on vehicle color recognition usually uses the vehicle color dataset established by Chen et al., which covers 8 categories: white, black, gray, red, cyan, blue, yellow, green, a total of 15,601 images, and each image contains only one car. The vehicle color dataset established by Jeong et al. contains 7 colors: black, gray, silver, white, blue, red, and yellow, a total of 6,819 images. The vehicle color dataset established by Tilakaratna et al. contains 13 colors: silver, white, black, gray, sky blue, red, blue, brown, orange, pink, green, yellow, and purple, a total of 7,000 images. Since the existing public vehicle datasets contain fewer types of vehicle colors and the vehicle color in a single image is single, it is necessary to create a dataset with richer colors to improve the accuracy of fine classification of vehicle colors.

[0005] The outstanding research work of scholars at home and abroad can be basically divided into two categories. One is to manually design feature extraction schemes and then use classifiers for recognition, including Chen et al. extracting color histogram features, using spatial pyramid matching and feature context information, and combining support vector machines to recognize vehicle colors; Dule et al. proposed to extract different color features from different color spaces, and used K nearest neighbors, support vector machines and artificial neural networks to study the recognition performance of vehicle colors. However, their databases only have five to eight categories, all of which have insufficient classification problems, and the recognition accuracy still needs to be improved. The other category is the use of deep learning algorithms for vehicle color recognition after 2015, including Hu et al. using deep neural networks to obtain image feature maps and combining traditional classifiers to apply them to vehicle color recognition; Rachmadi et al. used an end-to-end deep neural network structure to integrate feature extraction and classifiers into a unified framework through joint optimization; Fu et al. used multi-scale comprehensive feature fusion convolutional neural networks based on residual learning for vehicle color recognition, and established a system for vehicle color recognition in actual traffic scenarios. However, due to the complexity of the convolutional neural network structure, the susceptibility to overfitting, the small number of data sets and their uneven distribution, complex application scenarios cannot achieve the corresponding real-time performance and accuracy.

[0006] In summary, current similar solutions have failed to solve the problem of fine classification, accurate and efficient vehicle color recognition. Summary of the invention

[0007] In order to overcome the above problems, the present invention has built a larger-scale and color-diverse vehicle color classification database, and proposed a smooth modulation neural network method based on multi-layer feature representation for vehicle 24-color recognition method, which can improve the accuracy of vehicle 24-color recognition.

[0008] The method for identifying a vehicle 24-color long-tail by using a smooth modulation neural network method based on multi-layer feature representation is characterized by comprising the following steps:

[0009] Step 1: Build a 24-color vehicle database, annotate it with experts, and divide it into training data, verification data, and test data in a ratio of 8:1:1;

[0010] Step 2: Dehaze the image with haze;

[0011] Step 3: Linearly adjust the image brightness on the RGB three channels;

[0012] Step 4: Adjust some convolutional cascade blocks of ResNet-50 and use a lightweight feature extraction network to extract important features of color and vehicle type from the image to obtain a feature map with key attributes.

[0013] Step 5: The color features extracted by the deep network and the shallow network in step 4 are fused to construct a multi-scale feature fusion network to fuse feature layers of different depths;

[0014] Step 6: Generate multiple candidate regions of interest on the feature map, use the classifier to distinguish these ROIs into background and foreground, and use the regressor to perform preliminary screening of the locations of these ROIs;

[0015] Step 7: Adjust the Smooth L1 loss function by increasing the hyperparameter β to accurately adjust the target box and different types of targets;

[0016] Step 8: Train the network model to perform vehicle color recognition.

[0017] By adopting the above technology, compared with the prior art, the beneficial effects of the present invention are as follows: the present invention preprocesses the image to overcome the interference of complex environmental noise, obtains a feature map with key attributes through a lightweight feature extraction network and a multi-scale feature fusion network, avoids the limitations of manually designed feature areas, and achieves real-time and accurate vehicle color recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of the method is shown in FIG.

[0019] Figure 2 This is the overall structure diagram of this method;

[0020] Figure 3 It is a 24-color vehicle distribution map;

[0021] Figure 4 This is the flow chart of feature extraction and feature fusion. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but the protection scope of the present invention is not limited thereto.

[0023] The present invention can improve the accuracy of vehicle color recognition and can detect and recognize in real time, thus better meeting the need for fine classification of vehicle colors.

[0024] like Figure 1-2 As shown, a smooth modulation neural network method based on multi-layer feature representation of the present invention comprises the following steps:

[0025] Step 1: Construct a vehicle color database with a total of 10091 images with a resolution of 1920×1080 pixels and a total of 31232 vehicles. Each image contains multi-color vehicles, covering 24 color types, including red, dark red, pink, orange, dark orange, orange-red, yellow, lemon yellow, khaki, green, dark green, grass green, cyan, blue, dark blue, purple, black, white, silver gray, gray, dark gray, champagne, brown, dark brown, including cars, trucks, buses and other vehicle types, and expert annotation, divided into training data, verification data and test data in a ratio of 8:1:1. The distribution of each vehicle color is shown in the following figure. Figure 3 The quantity is shown in Table 1. Preliminary analysis shows that the database has a long tail effect;

[0026] color quantity Proportion color quantity Proportion color quantity Proportion White 11827 37.87％ red 644 2.06％ Lemon Yellow 92 0.29％ black 6270 20.08％ blue 553 1.77％ Dark Orange 90 0.29％ orange color 2431 7.78％ champagne 466 1.49％ Dark green 70 0.22％ Silver Gray 2125 6.80％ Dark Blue 365 1.17％ Orange Red 63 0.20％ Grass Green 1766 5.65％ blue 316 1.01％ Earthy yellow 52 0.17％ Dark Gray 1555 4.98％ dark brown 230 0.74％ green 50 0.16％ Dark red 1263 4.04％ brown 118 0.38％ pink 35 0.11％ grey 736 2.36％ yellow 100 0.32％ Purple 15 0.04％

[0027] Step 2: Defogging the image according to formula (1);

[0028]

[0029] Where J(x) is the haze-free image to be restored, I(x) is the image to be dehazed, A is the global atmospheric light component, t(x) is the transmittance, t0 is the threshold, and this method uses t0 = 0.1 as the standard.

[0030] Step 3: linearly adjust the image brightness on the three GRB channels according to formula (2);

[0031] Val i =α×Col i +β,i=r,g,b (2)

[0032] Among them, Col i Represents the RGB value of the original image, Val i Indicates the adjusted RGB value. α and β are hyperparameters. When the image brightness needs to be enhanced at night, α is set to 1.5 and β is set to 0, so that the feature gap between different colors will be more obvious. When the image exposure needs to be reduced at noon, α is set to 0.8 and β is set to -10, which can reduce the exposure of the entire image.

[0033] Step 4: Extract image features, use layer-by-layer visualization to intuitively determine the number of network convolution modules, adjust some convolution blocks of ResNet-50, and then give a balanced value through comparative analysis. The convolution module is significantly reduced with less accuracy sacrifice, so as to determine the construction of the magnitude network feature extraction module. The structural parameters of the feature extraction network constructed by this method are shown in Table 2, which is divided into 5 convolution blocks and a total of 42 layers of network;

[0034] Table 2

[0035]

[0036] Step 5: The color features extracted by the deep network and the shallow network in step 4 are fused to construct a multi-scale feature fusion network. Due to the different number of channels of feature pixels, the output features of each layer are standardized before fusion. The feature extraction network framework defined by this method is as follows: Figure 4 As shown;

[0037] Step 6: Generate proposal boxes, generate multiple candidate regions of interest on the feature map, and then use the classifier to distinguish these ROIs into background and foreground, and use the regressor to perform preliminary screening of the locations of these ROIs;

[0038] Step 7: Adjust the Smooth L1 loss function by adding the hyperparameter β to optimize the target box and different types of targets, as shown in formula (3):

[0039]

[0040] where y i is the true value, f(x i ) is the predicted value, and the value of β is 0.11;

[0041] Step 8: Train the network model to perform vehicle color recognition.

[0042] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A vehicle 24-color recognition method based on a smooth neural network with multi-layer features, characterized in that The steps include: S1: self-built vehicle color 24-classification dataset and expert annotation; S2: Preprocessing, including rain and fog removal, lighting adjustment, and restoration of fog-free image as shown in formula (1). Linear adjustment of lighting on the RGB three channels as shown in formula (2): Where J(x) is the haze-free image to be restored, I(x) is the image to be dehazed, A is the global atmospheric light component, t(x) is the transmittance, t0 is the threshold, and this method uses t0 = 0.1 as the standard; Val i =α×Col i +β,i=r,g,b (2) Where Col i Represents the RGB value of the original image, Val i Indicates the adjusted RGB value. α and β are hyperparameters. When the image brightness needs to be enhanced at night, α is set to 1.5 and β is set to 0 to make the feature difference between different colors more obvious. When the image exposure needs to be reduced at noon, α is set to 0.8 and β is set to -10 to reduce the exposure of the entire image. S3: Use a lightweight feature extraction network to extract important features of color and vehicle type from the image, and combine it with a multi-scale feature fusion network to obtain a feature map with key attributes; S4: Generate multiple candidate regions of interest on the feature map, then use the classifier to distinguish these ROIs into background and foreground, and use the regressor to perform preliminary screening of the locations of these ROIs; S5: Smooth modulation, the classification loss function is used to distinguish different types of targets, and the regression loss function is used to accurately adjust the target box; by adding the hyperparameter β to adjust the Smooth L1 loss function, the target box and different types of targets are optimized as shown in formula (3): Where y i is the true value, f(x i ) is the predicted value, and the value of β is 0.

11.

2. The method for identifying vehicles in 24 colors using a smooth neural network based on multi-layer features according to claim 1 is characterized in that: In step S1, a total of 100 hours of videos shot by surveillance cameras at multiple roads in Xi'an were collected. The shooting direction was the image in front of the vehicle. A total of 10,091 pictures were captured with a resolution of 1920×1080, and a total of 31,232 vehicles.

3. The vehicle 24-color recognition method based on a smooth neural network with multi-layer features according to claim 1 is characterized in that: In step S1, each image contains multi-color vehicles, covering 24 color types, including red, dark red, pink, orange, dark orange, orange-red, yellow, lemon yellow, khaki, green, dark green, grass green, cyan, blue, dark blue, purple, black, white, silver gray, gray, dark gray, champagne, brown, and dark brown.

4. The method for identifying vehicles in 24 colors using a smooth neural network based on multi-layer features according to claim 1 is characterized in that: The data set in step S1 includes multiple vehicle types such as cars, trucks and buses, and is affected by natural conditions such as rain, fog or lighting. The data distribution has an obvious long-tail effect, among which white vehicles are the largest, with 11,827 vehicles, accounting for 37.87%, and purple vehicles are the least, with 15 vehicles, accounting for 0.04%.

5. The vehicle 24-color recognition method based on a smooth neural network with multi-layer features according to claim 1 is characterized in that: The structural parameters of the feature extraction network constructed in step S3 are divided into 5 convolution blocks; the first convolution block is composed of a convolution kernel of 7*7 size and stride 2, and the number of output channels is 64; the second convolution block is composed of a 3*3 maximum pooling layer with a stride of 2 and 3 residual blocks cascaded, and the number of output channels is 256; the third convolution block is composed of 4 residual blocks cascaded, and the number of output channels is 512; the fourth convolution block is composed of 4 convolution blocks cascaded, and the number of output channels is 1024; the fifth convolution block is composed of 3 convolution blocks cascaded, and the number of output channels is 2048; finally, a fully connected layer is added, and the number of output channels is 1000, for a total of 42 layers of networks.

Images