A vehicle image recognition model establishment and recognition method

Abstract

The present invention discloses a method for establishing a vehicle image recognition model. A commonly used public vehicle image dataset is used to augment and preprocess the vehicle image dataset, construct a multi-branch convolutional neural network, and train it to obtain a multi-branch convolutional neural network for vehicle recognition. The trained multi-branch convolutional neural network is then converted into a single-branch convolutional neural network through structural reparameterization. Features of vehicle images are extracted and classified, ultimately determining the vehicle type. This method improves the multi-branch convolutional neural network by using a spatial pyramid and concatenating the pooled results, enabling the network to extract features of different scales and achieve vehicle image feature extraction and fusion. Furthermore, the optimization function is modified to accelerate the training of the multi-branch convolutional network, improve the accuracy of the vehicle recognition method based on the multi-branch convolutional neural network, and reduce the time required to train the multi-branch convolutional neural network.

Description

Technical Field

[0001] The present invention belongs to the field of image technology and relates to an image classification and recognition method, and in particular to a vehicle image recognition model establishment and recognition method. Background Art

[0002] Vehicle type recognition is the core of intelligent traffic management. In the field of intelligent transportation, the increasing number of vehicle types on public roads makes vehicle classification not only a hot topic in computer vision and pattern recognition, but also plays a crucial role in intelligent transportation. Vehicle type classification aims to distinguish between different types of vehicles within the broad category. For example, based on their overall appearance, vehicles can be categorized as sedans, buses, minibuses, trucks, minivans, and urban off-road vehicles. Vehicle image datasets have little inter-class variation and are affected by background interference, occlusion, and camera angle. This makes vehicle type classification more difficult than general image classification, making it more valuable.

[0003] In recent years, deep learning has made significant progress in fields such as images, text, and speech. Convolutional neural networks, in particular, have been widely used in image processing. CNN-based image classification has gradually become a research hotspot, and vehicle type classification has also made significant progress. A convolutional neural network model for vehicle type classification, based on rearview vehicle images, standardizes rearview vehicle images to a fixed size of 32×32 and feeds them into a CNN for vehicle type classification. This model places strict requirements on the input image size, and changes in image size can reduce classification accuracy and impair model generalization. The proposed vehicle type classification model uses channel pooling to improve convolutional neural networks. It first divides feature maps into several parts and then performs a pooling operation on these feature maps. This compresses the feature maps of each part into a new feature map by selecting the maximum value at the same position from different feature maps. While improving the accuracy of fine-grained vehicle classification tasks while reducing a large number of parameters, this method suffers from slow processing speed. Furthermore, it combines the physical features of the vehicle's rearview image with deep learning-based appearance features for vehicle classification, maintaining a highly physical interpretation of the vehicle image features. While maintaining high feature density, it also performs data dimensionality reduction through principal component analysis or discriminant analysis. This model relies on good physical feature selection; improper physical feature selection directly impacts classification performance. A real-time convolutional neural network for vehicle classification is proposed based on SE and the introduction of Haar wavelets as pooling layers. This model suppresses feature maps irrelevant to classification, enhances useful feature maps, and uses a cross-entropy loss function to improve the network's classification performance. While the network's parameters and number of layers are selected in real time, it still suffers from a large number of parameters and long training time. Summary of the Invention

[0004] In view of the technical problems of the vehicle model recognition method in the prior art such as long training time and low accuracy when recognizing vehicle images, the purpose of the present invention is to provide a vehicle image recognition model establishment and recognition method.

[0005] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0006] A method for establishing a vehicle image recognition model is characterized by performing the following steps:

[0007] Step 1: Select a commonly used public vehicle image dataset; perform preprocessing on each image in the vehicle image dataset, such as cropping, brightness enhancement, contrast enhancement, and horizontal flipping, to obtain the preprocessed vehicle image dataset for training, validation, and testing;

[0008] Step 2: Input the pre-processed vehicle images and corresponding categories into a multi-branch convolutional neural network for training;

[0009] The convolutional neural network parameters are all random numbers;

[0010] The multi-branch convolutional neural network includes a 3×3 convolutional layer, a 1×1 convolutional layer, a residual connection layer, an activation layer, a normalization layer, a spatial pyramid layer and a fully connected layer arranged in sequence;

[0011] The convolutional layer and spatial pyramid layer are used to extract features from the input image; the fully connected layer is used to classify and identify the extracted vehicle image features;

[0012] Step 3: Based on the vehicle image dataset, a multi-branch convolutional neural network is trained. After training, it is converted into a single-branch convolutional neural network. The single-branch convolutional neural network is used to classify and identify vehicle images to obtain the final predicted category, specifically:

[0013] Among them, when training a multi-branch convolutional neural network, formula I is used as the loss function L:

[0014]

[0015] In the formula, m is the number of sample categories; i is the i-th sample, x i is the label corresponding to the i-th sample, y is the output function of the model, that is is the probability that the predicted vehicle of the actual output is category i; CrossEntropyLoss is the loss value between the actual value and the predicted value.

[0016] Step 4: Use stochastic gradient descent with gradient centralization to accelerate the training of multi-branch convolutional neural networks, specifically:

[0017] When accelerating the training of multi-branch convolutional neural networks, Equation II is used as the optimization function:

[0018]

[0019] Where W i is the i-th column vector of the weight matrix W, W j,i is the j-th row and i-th column element of the weight matrix W, L is the loss function, is the gradient matrix corresponding to the weight matrix W, and M is the number of rows of the weight matrix W.

[0020] Step 5: Based on the additivity of convolution, the multi-branch training model is equivalently converted into a single-channel test model with only 3×3 convolution through structural reparameterization technology.

[0021] According to the present invention, the spatial pyramid layer includes multiple parallel pooling layers of windows of different sizes and an average pooling method; the fully connected layer includes a dense convolution layer and a Sigmoid activation layer arranged in sequence.

[0022] Furthermore, the activation function in the activation function layer is a ReLU function.

[0023] Experiments conducted by the applicant have shown that the vehicle image recognition model established by the above method can be used in applications for vehicle image recognition.

[0024] That is, the vehicle image dataset and the category corresponding to the vehicle are used as input, the prediction result of the multi-branch convolutional neural network is used as output, the multi-branch convolutional neural network is trained, and a vehicle image recognition model is obtained.

[0025] Compared with the prior art, the vehicle image recognition model establishment and recognition method of the present invention has the following technical innovations:

[0026] 1. The vehicle image dataset is preprocessed using methods such as cropping, brightness enhancement, contrast enhancement, and horizontal flipping to improve the accuracy of vehicle recognition;

[0027] 2. By adding a spatial pyramid layer to the original feature extraction backbone network, multi-scale feature extraction is performed on each feature image extracted after the feature extraction sub-network. This improves image scale invariance and vehicle type classification accuracy while reducing overfitting. Furthermore, only a single convolutional layer is required, improving vehicle recognition accuracy without significantly increasing model complexity.

[0028] 3. By modifying the optimization function, the gradient is directly centralized, and the output feature space and weight space are regularized at the same time, which can smooth the model's training loss curve and improve the model's classification accuracy; it can also suppress large gradients, making the training process more stable and faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the training structure of a vehicle image recognition model provided in one embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a test structure in a vehicle image recognition model provided in one embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a spatial pyramid structure in a vehicle image recognition model provided in one embodiment of the present invention;

[0032] Figure 4 The preprocessed vehicle image provided in one embodiment of the present invention; wherein (a) is the original image, (b) is the brightness-enhanced image, (c) is the contrast-enhanced image, and (d) is the horizontally flipped image;

[0033] Figure 5 A training loss curve diagram provided in one embodiment of the present invention;

[0034] Figure 6 This is the recognition result of each type of vehicle image in the data set in one embodiment of the present invention.

[0035] Figure 7 is the multi-classification confusion matrix.

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0037] Convolutional Neural Network: Different network architectures are used in the network training and network testing phases. The training phase focuses more on accuracy, while the testing phase focuses more on speed. The training network combines the advantages of multi-branch model training (high performance), while the testing network combines the benefits of single-branch model testing (fast speed and memory saving).

[0038] Spatial pyramid: A spatial pyramid is connected between the convolutional layer and the fully connected layer of the convolutional neural network to achieve multi-scale feature extraction of the input image without significantly increasing the parameters and model complexity, thereby improving the scale invariance of the image.

[0039] Gradient centralization: Directly processing the gradients calculated by the model, regularizing both the output feature space and the weight space, improving the stability of gradient descent and speeding up network training.

[0040] Cross entropy loss: The loss given by the actual output and the expected output is responsible for guiding the parameter adjustment of the neural network.

[0041] Example 1:

[0042] This embodiment provides a method for establishing a vehicle image recognition model, which is performed according to the following steps:

[0043] Step 1: Collect multiple vehicle images to obtain a vehicle image dataset;

[0044] In this embodiment, the collected vehicle image can be collected by on-site shooting, network search, etc. The original vehicle image obtained can be a local area or the entire vehicle, but it is necessary to ensure that the color and structure information in the obtained image are relatively complete.

[0045] In this embodiment, a vehicle dataset captured in real time under road monitoring is used as the original vehicle dataset. The dataset includes vehicles of different time periods, different sizes, and different driving speeds.

[0046] Step 2: Preprocess each image in the vehicle image dataset to obtain a preprocessed vehicle image dataset for training; the preprocessing includes cropping, rotation, mirroring, random brightness enhancement, random contrast enhancement and normalization; Figure 4 The preprocessed vehicle images are given; among them, (a) is the original image, (b) is the brightness enhanced image, (c) is the contrast enhanced image, and (d) is the horizontally flipped image.

[0047] Since the format or size of each image in the original vehicle dataset is not uniform, when using convolutional neural network for training, the number of parameters increases as the image size increases. Excessive parameters will lead to longer network training time and more stringent hardware requirements. Therefore, scaling the image to an appropriate size can greatly speed up the network training without affecting the results. Because convolutional networks are sensitive to rotation, rotation and mirroring can be used to expand the dataset to improve network performance. Random brightness enhancement and random contrast enhancement can also be used to expand the dataset to simulate real-world scenarios to improve the network's generalization ability. The specific steps include:

[0048] Step 2.1: Crop the corresponding vehicle image and scale it to M*M size to obtain the adjusted vehicle image;

[0049] To scale the vehicle image to the appropriate size, you can directly take N M*M image blocks from the original image, with each block partially overlapping. You can also choose to directly scale the entire image to M*M size.

[0050] In this embodiment, a random cropping method is used to directly scale the entire image to a size of 224*224.

[0051] Step 2.2: Mirror, randomly enhance brightness, and randomly enhance contrast of each adjusted original vehicle image to obtain the dataset required for the final training.

[0052] In order to further improve the performance of the network, the original vehicle image is mirrored and then subjected to random brightness enhancement and random contrast enhancement to obtain an expanded vehicle dataset consisting of 3 original images.

[0053] In this embodiment, the vehicles are mirrored, randomly enhanced in brightness, and randomly enhanced in contrast, respectively, to obtain an expanded vehicle dataset of 26,852 images.

[0054] Step 3: Input the pre-processed vehicle images and corresponding categories into a multi-branch convolutional neural network for training;

[0055] The multi-branch convolutional neural network includes a 3×3 convolutional layer, a 1×1 convolutional layer, an activation layer, a normalization layer, a residual connection layer, a spatial pyramid layer and a fully connected layer connected in sequence.

[0056] In this step, the multi-branch convolutional neural network includes a feature extraction network and a recognition network, wherein the feature extraction network is used to propose vehicle features; and the recognition network is used to classify and identify the vehicle features extracted by the feature extraction network.

[0057] Among them, the feature extraction network includes a 3×3 convolution layer, a 1×1 convolution layer, an activation layer, a normalization layer, a residual connection layer, and a spatial pyramid layer connected in sequence, such as Figure 1 This embodiment uses Python and the PyTorch deep learning framework to implement the feature extraction network, where convolution, activation, and normalization are all obtained by calling the corresponding functions of the corresponding framework. Residual and spatial pyramids are used to reduce parameters and avoid information loss.

[0058] The feature extraction network is used to extract vehicle features. The spatial pyramid layer includes multiple parallel pooling layers of windows of different sizes and average pooling methods. The results of different pooling are then spliced ​​together. In other words, Figure 3 As shown in the figure, the output results of the last convolutional layer are simultaneously input into the pooling layers of pooling windows of different sizes for calculation. After the calculation results are spliced, they are input into the fully connected layer for classification.

[0059] The network layer structure of the training network model designed in this embodiment is shown in Table 1.

[0060] Table 1: Network layer structure of SPP-RepVGG training model

[0061]

[0062]

[0063] In Table 1, - indicates that the module does not have this parameter; 3×3 or 1×1 indicates that there are parallel 3×3 and 1×1 convolution kernels in this stage; 2 or 1 indicates that the stride of the first module in this stage is 2, and the stride of the remaining modules is 1.

[0064] The training network structure designed in this embodiment is as follows Figure 1 shown.

[0065] In this embodiment, the activation function in the activation function layer is a ReLU function.

[0066] In this embodiment, the spatial pyramid layer includes three pooling windows of different sizes, and the window size of each pooling window is 1, 3, and 4 respectively.

[0067] According to the above design, a multi-branch convolutional neural network is obtained, and the multi-branch convolutional neural network is trained using data.

[0068] Optionally, in step 3, when training the multi-branch adversarial neural network, formula I is used as the loss function L:

[0069]

[0070] In the formula, m is the number of sample categories; i is the i-th sample, x i is the label corresponding to the i-th sample, and y is the output function of the model, that is, is the probability that the predicted vehicle of the actual output is category i; CrossEntropyLoss is the loss value between the actual value and the predicted value.

[0071] Step 4: Use stochastic gradient descent with gradient centralization to accelerate the training of multi-branch convolutional neural networks, specifically:

[0072] When accelerating the training of multi-branch convolutional neural networks, Equation II is used as the optimization function:

[0073]

[0074] Where W i is the i-th column vector of the weight matrix W, W j,iis the j-th row and i-th column element of the weight matrix W, L is the loss function, is the gradient matrix corresponding to the weight matrix W, and M is the number of rows of the weight matrix W.

[0075] Example 2:

[0076] The vehicle image recognition model established by the method of Example 1 is used to recognize the vehicle image to be recognized, and the following steps AB are performed:

[0077] Step A: preprocessing the vehicle image to be identified by using step 2 in Example 1 to obtain a preprocessed vehicle image to be identified;

[0078] Step B: input the image of the vehicle to be identified into the test model to obtain the identified vehicle type.

[0079] In this embodiment, Figure 5 The image to be identified is input into the test model to obtain the vehicle type.

[0080] The network layer structure of the designed test model is shown in Table 2:

[0081] Table 2: Network layer structure of the SPP-RepVGG test model

[0082] Module Convolution kernel size step length Output size enter - - 224×224×3 Stage 1 3×3 2 or 1 112×112×48 2×Stage2 3×3 2 or 1 56×56×48 4×Stage3 3×3 2 or 1 28×28×96 14×Stage4 3×3 2 or 1 14×14×192 Stage5 3×3 2 7×7×1280 SPP - - 1×33280 FC - - 1×6

[0083] In Table 2, - means that the module does not have this parameter; 2 or 1 means that the step size of the first module in this stage is 2, and the step size of the remaining modules is 1. The network structure of the test designed in this embodiment is as follows Figure 2 shown.

[0084] In this embodiment, the recognition method provided in this embodiment (referred to as this method in Table 3) is compared with the ResNet-18, VGG-16 and RepVGG-A0 methods in the prior art. The comparison of six indicators for measuring recognition effect, namely floating point operations (FLOPs), number of model parameters (M), model operation speed (s / epoch), loss value, TOP1 accuracy and multi-classification confusion matrix, is shown in Table 3. Figure 6 As shown in Figure 7, the smaller the floating-point operation value, the smaller the computing power consumed by the model and the simpler the model. The model operation speed determines the time consumed by model training, which is measured by the time taken for each training epoch (s / epoch). The loss value reflects the degree of closeness between the actual value and the predicted value, which is measured by the cross-entropy loss. The smaller the cross-entropy loss, the better the robustness of the model.

[0085] Table 3: Comparison between this method and traditional methods

[0086] Model Loss value FLOPs(G) Model parameter number (M) Training time (s / epoch) VGG-16 0.243 15.4158 70.3056 724.3125 ResNet-18 0.162 1.8191 11.6895 304.9083 RepVGG-A0 0.159 1.3620 7.0361 324.2686 SPP-RepVGG 0.141 1.3621 7.2281 299.3559

Claims

1. A method for establishing a vehicle image recognition model, characterized in that: Follow these steps: Step 1: Select a commonly used public vehicle image dataset; preprocess each image in the vehicle image dataset to obtain a preprocessed vehicle image dataset for training, verification, and testing; the preprocessing includes cropping, brightness enhancement, contrast enhancement, horizontal flipping, and normalization; Step 2: Input the pre-processed vehicle images and corresponding categories into a multi-branch convolutional neural network for training; The parameters of the multi-branch convolutional neural network are all random numbers; The multi-branch convolutional neural network includes a 3×3 convolutional layer, a 1×1 convolutional layer, a residual connection layer, an activation layer, a BN normalization layer, a spatial pyramid layer and a fully connected layer connected in sequence; The convolutional layer and spatial pyramid layer are used to extract features from the input image; the fully connected layer is used to classify and identify the extracted vehicle image features; Step 3: Based on the vehicle image dataset, train a multi-branch convolutional neural network to obtain the final prediction category, specifically: When training a multi-branch convolutional neural network, formula I is used as the loss function L: In the formula, m is the number of sample categories; i is the i-th sample, x i is the label corresponding to the i-th sample, and y is the output function of the model, that is, is the probability that the predicted vehicle is of category i; CrossEntropyLoss is the loss value between the actual value and the predicted value; Step 4: Use stochastic gradient descent with gradient centralization to accelerate the training of multi-branch convolutional neural networks, specifically: When accelerating the training of multi-branch convolutional neural networks, Equation II is used as the optimization function: Where W i is the i-th column vector of the weight matrix W, W j,i is the j-th row and i-th column element of the weight matrix W, L is the loss function, is the gradient matrix corresponding to the weight matrix W, and M is the number of rows of the weight matrix W; Step 5: Based on the additivity of convolution, the multi-branch training model is equivalently converted into a single-channel test model with only 3×3 convolution through structural reparameterization technology.

2. The method according to claim 1, wherein The spatial pyramid layer includes multiple parallel pooling layers of windows of different sizes and an average pooling method; the fully connected layer includes a dense convolution layer and a Sigmoid activation layer arranged in sequence.

3. The method according to claim 1, wherein The activation function in the activation layer is the ReLU function.

4. The vehicle image recognition model established by the method according to any one of claims 1 to 3 is used for the application of vehicle image recognition to be identified.

5. The use according to claim 4, characterized in that The vehicle image dataset and the corresponding vehicle category are used as input, the prediction results of the multi-branch convolutional neural network are used as output, and the multi-branch convolutional neural network is trained to obtain a vehicle image recognition model.

Images