A vehicle face recognition method for vehicles with obscured license plates

Abstract

This invention discloses a vehicle face recognition method for vehicles with obscured license plates, solving the technical problems of existing methods for recognizing obscured license plates. The invention obtains color prediction information and image vector information based on a color prediction module, a global perception module, and a detail perception module. Then, based on the color prediction information, it locates the corresponding folder and calculates the cosine distance between the vectors and the images in the folder, sorting them from largest to smallest, and selecting the top five as the recognition results. Compared to traditional methods, this invention adds an auxiliary process of vehicle color classification, using the global perception module and the detail perception module for recognition. The global perception module performs multiple normalization calculations from both batch and channel perspectives, i.e., searching and recognizing vehicles with obscured license plates in a vehicle image database from multiple dimensions including color appearance, global information, and detail information. This improves search capability and the model's representation of the content of obscured vehicles, successfully identifying vehicles with corresponding unobscured license plates.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and pattern recognition technology, and specifically to a method for vehicle face recognition for vehicles with obscured license plates. Background Technology

[0002] Cars are a common mode of transportation for most households. However, in complex traffic environments, some drivers may engage in illegal behaviors such as speeding and overloading on certain roads, obscuring their license plates to evade police tracking, and then restoring them to normal operation after a period of time. This phenomenon makes traffic control and monitoring difficult and hard to resolve. Although there are numerous cameras on the road capable of monitoring and tracking illegal vehicles, most cars share similar characteristics such as model and color. Finding the corresponding target image from the massive image database of these non-overlapping cameras is extremely difficult and time-consuming to do manually. Therefore, a method is needed that uses deep learning techniques to identify obscured vehicles in the image database by recognizing their unobscured faces, thus identifying vehicles committing violations. However, existing deep learning-based methods extract an effective global feature representation for each vehicle face image, lacking consideration of detailed feature information. Furthermore, features extracted through neural networks struggle to capture information about appearance variations, and the resulting vehicle face recognition data, after sorting, often differs significantly from the actual vehicle in the test image. Summary of the Invention

[0003] The purpose of this invention is to overcome the above-mentioned deficiencies in the prior art and provide a vehicle face recognition method for vehicles with obscured license plates.

[0004] The objective of this invention can be achieved by adopting the following technical solutions:

[0005] A vehicle face recognition method for vehicles with obscured license plates, the vehicle face recognition method comprising the following steps:

[0006] S1. Input a specific car face image P from the dataset into the backbone network to obtain the feature map F. p ∈R C×W×H Where C represents the number of feature map channels, W and H are the width and height of the feature map, and R represents the real number field;

[0007] S2, transfer feature map F p The color prediction is input into the color prediction module to predict the vehicle color, and the predicted vehicle color result i is obtained. The color prediction result i is used in step S6.

[0008] S3, transfer feature map F p The input is fed into the global perception module for prediction, resulting in a vector. Where D is the dimension of the column vector, and vector Q1 contains global information of the car face image;

[0009] S4. Transfer feature map F p The input is fed into the detail-aware module for prediction, resulting in a vector. Vector Q2 contains detailed information about the car's front image;

[0010] S5. Concatenate vectors Q1 and Q2 to obtain vector Q. r ∈R D×1 Vector Q r It contains both global and detailed information about the vehicle's front end;

[0011] S6. Based on the predicted vehicle color result i from step S2, convert vector Q... r Stored in a folder named i;

[0012] S7. Repeat steps S1 to S6 for all car face images in the image library of the dataset to obtain the vector set Q. i ={Q i,k |1≤k≤I n}, where I n Q represents the number of vectors in folder i. i,k This refers to the vector corresponding to the k-th image in folder i;

[0013] S8. Input the image of the vehicle face to be tested and execute step S1 sequentially to obtain the feature map F. q ;

[0014] S9, Input Feature Map F q After performing step S2, the color prediction result is obtained as i. q The color prediction result i is used in step S11;

[0015] S10, Input Feature Map F q By sequentially performing steps S3, S4, and S5, the concatenated vector Q is obtained. q ∈R D×1 Vector Q q It contains global and detailed information about the vehicle's front image to be tested;

[0016] S11, change vector Q q With folder i q The cosine distance is calculated for all vectors in the set, resulting in the set. I nq For folder i q The number of vectors in the middle. It refers to folder i q The vector corresponding to the g-th image in the image, and the color prediction and classification processing make the recognition of car face images more targeted;

[0017] S12, Set Sort the images by distance values ​​from largest to smallest, and select the images corresponding to the top 5 vectors with the highest distance values ​​as the prediction results.

[0018] Furthermore, the backbone network structure is connected sequentially from the input layer to the output layer as follows: convolutional layer conv1_1, ReLU layer conv1_1_relu, convolutional layer conv1_2, ReLU layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, ReLU layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, ReLU layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, and ReLU layer con v3_1_relu, convolutional layer conv3_2, ReLU layer conv3_2_relu, convolutional layer conv3_3, ReLU layer conv3_3_relu, pooling layer max_pooling3, convolutional layer conv4_1, ReLU layer conv4_1_relu, convolutional layer conv4_2, ReLU layer conv4_2_relu, convolutional layer conv4_3, ReLU layer conv4_3_relu, pooling layer max_pooling4, convolutional layer conv5_1, ReLU layer conv5_ 1_relu, convolutional layer conv5_2, Relu layer conv5_2_relu, convolutional layer conv5_3, Relu layer conv5_3_relu, pooling layer max_pooling5, convolutional layer fc6, Relu layer fc6_relu, convolutional layer fc7, Relu layer fc7_relu, convolutional layer conv6_1, Relu layer conv6_1_relu, convolutional layer conv6_2, Relu layer conv6_2_relu, convolutional layer conv7_1, Relu layer conv7_1_relu u, convolutional layer conv7_2, ReLU layer conv7_2_relu, convolutional layer conv8_1, ReLU layer conv8_1_relu, convolutional layer conv8_2, ReLU layer conv8_2_relu, convolutional layer conv9_1, ReLU layer conv9_1_relu, convolutional layer conv9_2, ReLU layer conv9_2_relu, convolutional layer conv10_1, ReLU layer conv10_1_relu, convolutional layer conv10_2, ReLU layer conv10_2_relu.

[0019] Furthermore, the color prediction module structure is connected sequentially from the input layer to the output layer as follows: pooling layer global_pooling_1, fc layer fc_1, and fc layer fc_2.

[0020] Furthermore, the global perception module structure is connected sequentially from the input layer to the output layer as follows: pooling layer global_pooling_2, multi-dimensional normalization layer conv1_1_in_bn, convolutional layer conv11_1, ReLU layer conv11_1_relu, convolutional layer conv11_2, and ReLU layer conv11_2_relu; wherein, the multi-dimensional normalization layer conv1_1_in_bn is composed of BN layer conv11_1_bn and IN layer conv11_1_in.

[0021] Furthermore, the detail-aware module structure is connected sequentially from the input layer to the output layer as follows: feature detail compression layer horizontal_pooling_1, BN layer conv12_1_bn, convolutional layer conv12_1, ReLU layer conv12_1_relu, convolutional layer conv12_2, and ReLU layer conv12_2_relu.

[0022] Furthermore, step S2 is as follows:

[0023] S21, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_1 to obtain the feature map E. a ∈R C×1 Used for feature map F p Information compression;

[0024] S22, Transfer feature map E a The input is fed into the fully connected layers fc_1 and fc_2, and the vector E is obtained through the softmax function. f ∈R N ×1 , where N is the number of vehicle colors in the dataset, and vehicle colors are predicted through a fully connected structure with two fully connected (fc) layers;

[0025] S23, in vector E f Find the index i corresponding to the highest value in the table and output the vehicle color result i.

[0026] Furthermore, step S3 is as follows:

[0027] S31, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_2 to obtain the vector V∈R D×1 Used for feature map Fp Information compression;

[0028] S32. Input vector V into the BN layer conv11_1_bn to obtain vector V. b ∈R D×1 Vector V b It is the result of normalizing vector V from the batch direction;

[0029] S33. Input vector V into the IN layer conv11_1_in to obtain vector V. i ∈R D×1 Vector V i It is the result of normalizing vector V from the channel direction;

[0030] S34, Transform vector V b With vector V i The concatenation yields vector V. c ∈R 2D×1 ;

[0031] S35, Vector V c The vector Q1 is obtained by passing through convolutional layer conv11_1, ReLU layer conv11_1_relu, convolutional layer conv11_2, and ReLU layer conv11_2_relu. Therefore, vector Q1 is the concatenation of vector V after batch normalization and channel normalization, thus avoiding overfitting and loss of style features.

[0032] Furthermore, step S4 is as follows:

[0033] S41, Transfer feature map F p The input is compressed into the feature detail compression layer horizontal_pooling_1, and then normalized into the BN layer conv12_1_bn to obtain the feature map J. s ∈R C×H×1 ;

[0034] S42, Transfer feature map J s The input is fed into a convolutional layer conv12_1, a ReLU layer conv12_1_relu, a convolutional layer conv12_2, and a ReLU layer conv12_2_relu to obtain the vector Q2.

[0035] Furthermore, the feature detail compression layer is used to compress detailed features to facilitate subsequent recognition, and its working process is as follows:

[0036] Assume the input to the feature detail compression layer is a feature map M∈R C×W×H The output is a feature map Y∈R C×H×1 ,

[0037] For a feature map M, it is divided into a set A of number H along the height direction. * ={A m |1≤m≤H}, where A m ∈R C×W×1 Refers to set A * The feature map of the m-th position in the middle;

[0038] For set A * A certain feature map A z Compress the data to obtain the result k. z The compression formula is as follows:

[0039]

[0040] Where A z,x,y Let set A * Feature map A at the z-th position z The x-th row vector with y channels in the middle;

[0041] For set A * Each feature map in the array is calculated using formula (1) and aggregated into a column vector T∈R. H×1 The value at the t-th position in column vector T represents set A. * The result of calculating the feature map at position t using formula (1).

[0042] Furthermore, the cosine distance is used to calculate the similarity between two vectors, and the calculation process is as follows:

[0043] Suppose we calculate the cosine distance between vectors J and B, obtaining the value l. The calculation expression is as follows:

[0044] l = 1 - J·B T .

[0045] In the formula, the symbol "·" indicates that the elements in the vector are multiplied one by one, and the superscript "T" indicates the vector transpose operation.

[0046] The present invention has the following advantages and effects compared with the prior art:

[0047] (1) The present invention can find 5 vehicles similar to the image of the covered license plate in the image library, and can find the image of the uncovered vehicle that matches the image in the 5 images.

[0048] (2) Compared with traditional methods, the present invention improves the neural network and feature vector normalization to enhance the expressive power of feature vectors for car face images.

[0049] (3) Compared with the traditional method, the present invention adds a color prediction module as an aid to the subsequent recognition. The color of the car face is classified and stored in folders of different colors, which makes the recognition of the car face more targeted. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0051] Figure 1 This is a flowchart of a vehicle face recognition method for obscured license plates disclosed in this invention.

[0052] Figure 2 This is a schematic diagram of a vehicle face recognition method for obscured license plates disclosed in this invention.

[0053] Figure 3 This is a bar chart comparing the accuracy of this method with other methods when using the VehicleID dataset, under the premise of Example 1;

[0054] Figure 4 This is a bar chart comparing the accuracy of this method with other methods when using the VeRi dataset, under the premise of Example 2;

[0055] Figure 5 This is a structural diagram of the backbone network in this invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1

[0058] In this Example 1, the VehicleId dataset is used. Each image in the dataset has an ID label corresponding to a real-world identity. The training set contains 13,164 vehicles and 113,346 images. The test set contains 2,400 vehicles and 19,777 images.

[0059] The method includes the following steps:

[0060] Step S1: Obtain the number of vehicle color types N in the dataset. In this dataset, N=9. The number of color types is used to set the dimension in the fc_2 layer of the color prediction module. Also, randomly select 1000 images from the test set according to the vehicle ID label as the test image library.

[0061] Step S2, Experimental Environment: This embodiment only requires general hardware configuration and a Graphics Processing Unit (GPU) to accelerate computation. Model building, training, and testing of training results are all completed using the PyTorch deep learning framework and FastReid tool, employing the Compute Unified Device Architecture (CUDA) to enable the GPU to solve complex computational problems. The specific runtime environment configuration required for this embodiment is shown in Table 1.

[0062] Table 1. Experimental Operating Environment Configuration Table for this Embodiment

[0063]

[0064] Step S3: Construct a vehicle face recognition method for vehicles with obscured license plates, as shown in the structure diagram below. Figure 2 As shown, the specific steps for constructing the structure are as follows:

[0065] Step S31: Construct the backbone network, such as... Figure 5As shown, the backbone network structure, from the input layer to the output layer, is connected as follows: convolutional layer conv1_1, ReLU layer conv1_1_relu, convolutional layer conv1_2, ReLU layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, ReLU layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, ReLU layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, ReLU layer conv3_... 1_relu, convolutional layer conv3_2, Relu layer conv3_2_relu, convolutional layer conv3_3, Relu layer conv3_3_relu, pooling layer max_pooling3, convolutional layer conv4_1, Relu layer conv4_1_relu, convolutional layer conv4_2, Relu layer conv4_2_relu, convolutional layer conv4_3, Relu layer conv4_3_relu, pooling layer max_pooling4, convolutional layer conv5_1, Relu layer conv5_1_ ReLU, convolutional layer conv5_2, ReLU layer conv5_2_relu, convolutional layer conv5_3, ReLU layer conv5_3_relu, pooling layer max_pooling5, convolutional layer fc6, ReLU layer fc6_relu, convolutional layer fc7, ReLU layer fc7_relu, convolutional layer conv6_1, ReLU layer conv6_1_relu, convolutional layer conv6_2, ReLU layer conv6_2_relu, convolutional layer conv7_1, ReLU layer conv7_1_relu Convolutional layer conv7_2, ReLU layer conv7_2_relu, convolutional layer conv8_1, ReLU layer conv8_1_relu, convolutional layer conv8_2, ReLU layer conv8_2_relu, convolutional layer conv9_1, ReLU layer conv9_1_relu, convolutional layer conv9_2, ReLU layer conv9_2_relu, convolutional layer conv10_1, ReLU layer conv10_1_relu, convolutional layer conv10_2, ReLU layer conv10_2_relu;

[0066] Step S32: Construct a color prediction module. The color prediction module structure is connected sequentially from the input layer to the output layer as follows: pooling layer global_pooling_1, fc layer fc_1, fc layer fc_2;

[0067] Step S33: Construct the global perception module. The global perception module structure, from the input layer to the output layer, is connected as follows: pooling layer global_pooling_2, multi-dimensional normalization layer conv1_1_in_bn, convolutional layer conv11_1, ReLU layer conv11_1_relu, convolutional layer conv11_2, and ReLU layer conv11_2_relu; among which, the multi-dimensional normalization layer conv1_1_in_bn is composed of BN layer conv11_1_bn and IN layer conv11_1_in.

[0068] Step S34: Construct the detail-aware module. The structure of the detail-aware module from the input layer to the output layer is as follows: feature detail compression layer horizontal_pooling_1, BN layer conv12_1_bn, convolutional layer conv12_1, ReLU layer conv12_1_relu, convolutional layer conv12_2, and ReLU layer conv12_2_relu.

[0069] Step S4: Divide the training set images into an image database and a query database based on vehicle IDs. The vehicle IDs in the query database are unique, and the image database can contain multiple images with the same vehicle ID. First, train the processed training set images using this method. During training, the optimizer used is Adam, with a learning rate of 0.00035, a learning rate momentum of 0.0005, a batch size of 64, and 60 epochs. The learning rate decreases by a factor of 0.1 at epochs 30 and 50.

[0070] Step S5: Input a specific car face image P from the dataset into the backbone network to obtain the feature map F. p ∈R C×W×H Where C represents the number of feature map channels, W and H are the width and height of the feature map, and R represents the real number field;

[0071] Step S6: Transfer feature map F p The input is fed into the color prediction module to predict the vehicle color, and the predicted vehicle color result is i, where 1≤i≤N;

[0072] The specific process is as follows:

[0073] S61, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_1 to obtain the feature map E. a ∈R C×1 Used for feature map F p Information compression;

[0074] S62, Transfer feature map Ea The input is fed into the fully connected layers fc_1 and fc_2, and the vector E is obtained through the softmax function. f ∈R N ×1 The vehicle color is predicted through a fully connected structure with two FC layers.

[0075] S63, in vector E f Find the index i corresponding to the highest value in the table, and output the vehicle color result i.

[0076] Step S7: Transfer feature map F p The input is fed into the global perception module for prediction, resulting in a vector. Where D is the dimension of the column vector, and vector Q1 contains global information of the car face image; in Example 1, D = 2048;

[0077] The specific process is as follows:

[0078] S71, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_2 to obtain the vector V∈R D×1 ;

[0079] S72. Input vector V into the BN layer conv11_1_bn to obtain vector V. b ∈R D×1 ;

[0080] S73. Input vector V into the IN layer conv11_1_in to obtain vector V. i ∈R D×1 ;

[0081] S74, Transform vector V b With vector V i The concatenation yields vector V. c ∈R 2D×1 ;

[0082] S75, Vector V c The vector Q1 is obtained by passing a convolutional layer conv11_1, a ReLU layer conv11_1_relu, a convolutional layer conv11_2, and a ReLU layer conv11_2_relu.

[0083] Step S8: Transfer feature map F p The input is fed into the detail-aware module for prediction, resulting in a vector. Vector Q2 contains detailed information about the car's front image;

[0084] The specific process is as follows:

[0085] S41, Transfer feature map F p The input is compressed into the feature detail compression layer horizontal_pooling_1, and then normalized into the BN layer conv12_1_bn to obtain the feature map J. s ∈R C×H×1 ;

[0086] S42, Transfer feature map J s The input is fed into a convolutional layer conv12_1, a ReLU layer conv12_1_relu, a convolutional layer conv12_2, and a ReLU layer conv12_2_relu to obtain the vector Q2.

[0087] The working process of the feature detail compression layer in the detail-aware module is as follows:

[0088] Assume the input to the feature detail compression layer is a feature map M∈R C×W×H The output is a feature map Y∈R C×H×1 :

[0089] For a feature map M, it is divided into a set A of number H along the height direction. * ={A m |1≤m≤H}, where A m ∈R C×W×1 Refers to set A * The feature map of the m-th position in the middle;

[0090] For set A * A certain feature map A z Compress the data to obtain the result k. z The compression formula is as follows:

[0091]

[0092] Where A z,x,y Let set A * Feature map A at the z-th position z The x-th row vector with y channels in the middle;

[0093] For set A * Each feature map in the array is calculated using formula (1) and aggregated into a column vector T∈R. H×1 The value at the t-th position in column vector T represents set A. * The result of calculating the feature map at position t using formula (1).

[0094] Step S9: Concatenate vectors Q1 and Q2 to obtain vector Q. r ∈R D×1 Vector Q rIt contains both global and detailed information about the vehicle's front end;

[0095] Step S10: Based on the predicted vehicle color result i from step S6, convert vector Q... r Stored in a folder named i;

[0096] Step S11: Perform steps S5 to S10 sequentially on all car face images in the test set image library to obtain the vector set Q. i ={Q i,k |1≤k≤I n}, where I n Q represents the number of vectors in folder i. i,k This refers to the vector corresponding to the k-th image in folder i;

[0097] Step S12: Input the image of the vehicle face to be tested and execute step S5 sequentially to obtain the feature map F. q ;

[0098] Step S13: Input feature map F q Execute step S6 to obtain the color prediction result i. q The color prediction result i is used in step S11;

[0099] Step S14: Input feature map F q By sequentially performing steps S7, S8, and S9, the concatenated vector Q is obtained. q ∈R D ×1 Vector Q q It contains global and detailed information about the vehicle's front image to be tested;

[0100] Step S15: Transfer vector Q q With folder i q The cosine distance is calculated for all vectors in the set, resulting in the set. I nq For folder i q The number of vectors in the middle. It refers to folder i q The vector corresponding to the g-th image in the image, and the color prediction and classification processing make the recognition of car face images more targeted;

[0101] The calculation process for the cosine distance is as follows:

[0102] Suppose we calculate the cosine distance between vectors J and B, obtaining the value l. The calculation expression is as follows:

[0103] l = 1 - J·B T .

[0104] In the formula, the symbol "·" indicates that the elements in the vector are multiplied one by one, and the superscript "T" indicates the vector transpose operation.

[0105] Step S16: Set Sort the images by distance values ​​from largest to smallest, and select the images corresponding to the top 5 vectors with the highest distance values ​​as the prediction results.

[0106] Step S17: Execute steps S12, S13, S14, and S15 sequentially for the image library in the test set. The specific process is as follows: Figure 1 As shown, we obtain all the prediction results for the image library in the test set and calculate the accuracy.

[0107] The experimental comparison results of Example 1 are as follows: Figure 3 As shown, Example 1 was compared with other methods on a test image library of 1000 images. Images containing the corresponding license plates in 5 images were marked as accurate. This method achieved an accuracy of 88.1%, which is 70.21% higher than the BOW-CN method, 62.47% higher than the LOMO method, 27.61% higher than the FACT method, 27.38% higher than the NuFACT method, 17.81% higher than VAML, and 4.73% higher than QD-DLF. This demonstrates the effectiveness of the method and its ability to successfully identify the corresponding unobstructed license plates.

[0108] Example 2

[0109] In this embodiment 2, the publicly available VeRi dataset is used, which contains over 50,000 images of 776 vehicles, including 37,778 images in the training set, 11,579 images in the image library, and 1,678 images for testing. Each image in the dataset is labeled with an ID tag corresponding to its real-world identity.

[0110] The method includes the following steps:

[0111] Step S1: Obtain the number of vehicle color types N in the dataset. In this dataset, N=9. The number of color types is used to set the dimension in the fc_2 layer of the color prediction module.

[0112] Step S2: Experimental Environment. This embodiment requires only general hardware configuration and a Graphics Processing Unit (GPU) to accelerate computation. Model building, training, and testing of training results are all completed using the PyTorch deep learning framework and Fastreid tool, employing the Compute Unified Device Architecture (CUDA) to enable the GPU to solve complex computational problems. Specifically, the hardware and software configuration used in this embodiment is as follows: GPU version: GeForce RTX 2080Ti; CPU version: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz; Operating system version: CentOS 8.3.2011; Python version: 3.6.13; CUDA version: 11.2.

[0113] Step S3: Construct a vehicle face recognition method for vehicles with obscured license plates, as shown in the structure diagram below. Figure 2 As shown, the specific steps for constructing the structure are as follows:

[0114] Step S31: Construct the backbone network, such as... Figure 5As shown, the backbone network structure, from the input layer to the output layer, is connected as follows: convolutional layer conv1_1, ReLU layer conv1_1_relu, convolutional layer conv1_2, ReLU layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, ReLU layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, ReLU layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, ReLU layer conv3_... 1_relu, convolutional layer conv3_2, Relu layer conv3_2_relu, convolutional layer conv3_3, Relu layer conv3_3_relu, pooling layer max_pooling3, convolutional layer conv4_1, Relu layer conv4_1_relu, convolutional layer conv4_2, Relu layer conv4_2_relu, convolutional layer conv4_3, Relu layer conv4_3_relu, pooling layer max_pooling4, convolutional layer conv5_1, Relu layer conv5_1_ ReLU, convolutional layer conv5_2, ReLU layer conv5_2_relu, convolutional layer conv5_3, ReLU layer conv5_3_relu, pooling layer max_pooling5, convolutional layer fc6, ReLU layer fc6_relu, convolutional layer fc7, ReLU layer fc7_relu, convolutional layer conv6_1, ReLU layer conv6_1_relu, convolutional layer conv6_2, ReLU layer conv6_2_relu, convolutional layer conv7_1, ReLU layer conv7_1_relu Convolutional layer conv7_2, ReLU layer conv7_2_relu, convolutional layer conv8_1, ReLU layer conv8_1_relu, convolutional layer conv8_2, ReLU layer conv8_2_relu, convolutional layer conv9_1, ReLU layer conv9_1_relu, convolutional layer conv9_2, ReLU layer conv9_2_relu, convolutional layer conv10_1, ReLU layer conv10_1_relu, convolutional layer conv10_2, ReLU layer conv10_2_relu;

[0115] Step S32: Construct a color prediction module. The color prediction module structure is connected sequentially from the input layer to the output layer as follows: pooling layer global_pooling_1, fc layer fc_1, fc layer fc_2;

[0116] Step S33: Construct the global perception module. The global perception module structure, from the input layer to the output layer, is connected as follows: pooling layer global_pooling_2, multi-dimensional normalization layer conv1_1_in_bn, convolutional layer conv11_1, ReLU layer conv11_1_relu, convolutional layer conv11_2, and ReLU layer conv11_2_relu; among which, the multi-dimensional normalization layer conv1_1_in_bn is composed of BN layer conv11_1_bn and IN layer conv11_1_in.

[0117] Step S34: Construct the detail-aware module. The structure of the detail-aware module from the input layer to the output layer is as follows: feature detail compression layer horizontal_pooling_1, BN layer conv12_1_bn, convolutional layer conv12_1, ReLU layer conv12_1_relu, convolutional layer conv12_2, and ReLU layer conv12_2_relu.

[0118] Step S4: Divide the training set images into an image database and a query database based on vehicle IDs. The vehicle IDs in the query database are unique, and the image database can contain multiple images with the same vehicle ID. First, train the processed training set images using this method. During training, the optimizer used is SGD, with a learning rate of 0.001, a learning rate momentum of 0.0005, a batch size of 64, and 60 epochs. The learning rate decreases by a factor of 0.1 at epochs 30 and 50.

[0119] Step S5: Input a specific car face image P from the dataset into the backbone network to obtain the feature map F. p ∈R C×W×H Where C represents the number of feature map channels, W and H are the width and height of the feature map, and R represents the real number field;

[0120] Step S6: Transfer feature map F p The input is fed into the color prediction module to predict the vehicle color, and the predicted vehicle color result is i, where 1≤i≤N;

[0121] The specific process is as follows:

[0122] S21, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_1 to obtain the feature map E. a ∈R C×1 Used for feature map F p Information compression;

[0123] S22, Transfer feature map Ea The input is fed into the fully connected layers fc_1 and fc_2, and the vector E is obtained through the softmax function. f ∈R N ×1 The vehicle color is predicted through a fully connected structure with two FC layers.

[0124] S23, in vector E f Find the index i corresponding to the highest value in the table and output the vehicle color result i.

[0125] Step S7: Transfer feature map F p The input is fed into the global perception module for prediction, resulting in a vector. Where D is the dimension of the column vector, and vector Q1 contains global information of the car face image; in Example 2, D = 2048;

[0126] The specific process is as follows:

[0127] S71, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_2 to obtain the vector V∈R D×1 ;

[0128] S72. Input vector V into the BN layer conv11_1_bn to obtain vector V. b ∈R D×1 ;

[0129] S73. Input vector V into the IN layer conv11_1_in to obtain vector V. i ∈R D×1 ;

[0130] S74, Transform vector V b With vector V i The concatenation yields vector V. c ∈R 2D×1 ;

[0131] S75, Vector V c The vector Q1 is obtained by passing a convolutional layer conv11_1, a ReLU layer conv11_1_relu, a convolutional layer conv11_2, and a ReLU layer conv11_2_relu.

[0132] Step S8: Transfer feature map F p The input is fed into the detail-aware module for prediction, resulting in a vector. Vector Q2 contains detailed information about the car's front image;

[0133] The specific process is as follows:

[0134] S41, Transfer feature map F p The input is compressed into the feature detail compression layer horizontal_pooling_1, and then normalized into the BN layer conv12_1_bn to obtain the feature map J. s ∈R C×H×1 ;

[0135] S42, Transfer feature map J s The input is fed into a convolutional layer conv12_1, a ReLU layer conv12_1_relu, a convolutional layer conv12_2, and a ReLU layer conv12_2_relu to obtain vector Q2.

[0136] The working process of the feature detail compression layer in the detail-aware module is as follows:

[0137] Assume the input to the feature detail compression layer is a feature map M∈R C×W×H The output is a feature map Y∈R C×H×1 :

[0138] For a feature map M, it is divided into a set A of number H along the height direction. * ={A m |1≤m≤H}, where A m ∈R C×W×1 Refers to set A * The feature map of the m-th position in the middle;

[0139] For set A * A certain feature map A z Compress the data to obtain the result k. z The compression formula is as follows:

[0140]

[0141] Where A z,x,y Let set A * Feature map A at the z-th position z The x-th row vector with y channels in the middle;

[0142] For set A * Each feature map in the array is calculated using formula (1) and aggregated into a column vector T∈R. H×1 The value at the t-th position in column vector T represents set A. * The result of calculating the feature map at position t using formula (1).

[0143] Step S9: Concatenate vectors Q1 and Q2 to obtain vector Q. r ∈R D×1 Vector Q rIt contains both global and detailed information about the vehicle's front end;

[0144] Step S10: Based on the predicted vehicle color result i from step S6, convert vector Q... r Stored in a folder named i;

[0145] Step S11: Perform steps S5 to S10 sequentially on all car face images in the image library to obtain the vector set Q. i ={Q i,k |1≤k≤I n}, where I n Q represents the number of vectors in folder i. i,k This refers to the vector corresponding to the k-th image in folder i;

[0146] Step S12: Input the image of the vehicle face to be tested and execute step S5 sequentially to obtain the feature map F. q ;

[0147] Step S13: Input feature map F q Execute step S6 to obtain the color prediction result i. q The color prediction result i is used in step S11;

[0148] Step S14: Input feature map F q By sequentially performing steps S7, S8, and S9, the concatenated vector Q is obtained. q ∈R D ×1 Vector Q q It contains global and detailed information about the vehicle's front image to be tested;

[0149] Step S15: Transfer vector Q q With folder i q The cosine distance is calculated for all vectors in the set, resulting in the set. I nq For folder i q The number of vectors in the middle. It refers to folder i q The vector corresponding to the g-th image in the image, and the color prediction and classification processing make the recognition of car face images more targeted;

[0150] The calculation process for the cosine distance is as follows:

[0151] Suppose we calculate the cosine distance between vectors J and B, obtaining the value l. The calculation expression is as follows:

[0152] l = 1 - J·B T .

[0153] In the formula, the symbol "·" indicates that the elements in the vector are multiplied one by one, and the superscript "T" indicates the vector transpose operation.

[0154] Step S16: Set Sort the images by distance values ​​from largest to smallest, and select the images corresponding to the top 5 vectors with the highest distance values ​​as the prediction results.

[0155] Step S17: Execute steps S12, S13, S14, and S15 sequentially for the image library in the test set. The specific process is as follows: Figure 1 As shown, we obtain all the prediction results for the image library in the test set and calculate the accuracy.

[0156] The experimental comparison results of Example 2 are as follows: Figure 4 As shown, Example 2 was compared with other methods on a test image library of 1678 images. Images containing the corresponding license plate were marked as accurate. This method achieved an accuracy of 97.3%, which is 43.61% higher than the BOW-CN method, 50.82% higher than the LOMO method, 24.42% higher than the FACT method, 5.88% higher than the NuFACT method, 6.48% higher than VAML, and 2.84% higher than QD-DLF. This demonstrates the effectiveness of the method and its ability to successfully identify the corresponding unobstructed license plate.

[0157] In summary, the above embodiments disclose a vehicle face recognition method for obscured license plates. This invention obtains color prediction information and image vector information based on a color prediction module, a global perception module, and a detail perception module. Then, based on the color prediction information, it locates the corresponding folder and sequentially calculates the cosine distance between the vector information and the images in the folder. The vectors are then sorted from largest to smallest, and the top five are selected as the recognition results. This addresses the technical problem of recognizing obscured license plates in existing methods. Compared to traditional methods, this invention adds an auxiliary process of vehicle color classification, performs recognition through the global perception module and the detail perception module, and performs multiple normalization calculations from batch and channel perspectives in the global perception module. This improves search capability and the model's representation of obscured vehicle content, successfully identifying vehicles with unobscured license plates. The high recognition rate of this method is demonstrated by the experimental results of Embodiments 1 and 2.

[0158] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for vehicle face recognition for vehicles with obscured license plates, characterized in that, The vehicle face recognition method includes the following steps: S1. Input a specific car face image P from the dataset into the backbone network to obtain the feature map F. p ∈R C×W×H Where C represents the number of feature map channels, W and H are the width and height of the feature map, and R represents the real number field; S2, transfer feature map F p The input is fed into the color prediction module to predict the vehicle color, and the predicted vehicle color result is i. S3, transfer feature map F p The input is fed into the global perception module for prediction, resulting in a vector. Where D is the dimension of the column vector; S4. Transfer feature map F p The input is fed into the detail-aware module for prediction, resulting in a vector. The detail-aware module structure is connected sequentially from the input layer to the output layer as follows: feature detail compression layer horizontal_pooling_1, BN layer conv12_1_bn, convolutional layer conv12_1, ReLU layer conv12_1_relu, convolutional layer conv12_2, and ReLU layer conv12_2_relu; The working process of the feature detail compression layer is as follows: Assume the input to the feature detail compression layer is a feature map M∈R C×W×H The output is a feature map Y∈R C×H×1 , For a feature map M, it is divided into a set A of number H along the height direction. * ={A m |1≤m≤H}, where A m ∈R C×W×1 Refers to set A * The feature map of the m-th position in the middle; For set A * A certain feature map A z Compress the data to obtain the result k. z The compression formula is as follows: Where A z,x,y Let set A * Feature map A at position z z The x-th row vector with y channels in the middle; For set A * Each feature map in the array is calculated using formula (1) and aggregated into a column vector T∈R. H×1 The value at the t-th position in column vector T represents set A. * The result of calculating the feature map at position t using formula (1); S5. Concatenate vectors Q1 and Q2 to obtain vector Q. r ∈R D×1 ; S6. Based on the predicted vehicle color result i from step S2, convert vector Q... r Stored in a folder named i; S7. Repeat steps S1 to S6 for all car face images in the image library of the dataset to obtain the vector set Q. i ={Q i,k |1≤k≤I n }, where I n Q represents the number of vectors in folder i. i,k This refers to the vector corresponding to the k-th image in folder i; S8. Input the image of the vehicle face to be tested and execute step S1 sequentially to obtain the feature map F. q ; S9, Input Feature Map F q After performing step S2, the color prediction result is obtained as i. q ; S10, Input Feature Map F q By sequentially performing steps S3, S4, and S5, the concatenated vector Q is obtained. q ∈R D×1 ; S11, change vector Q q With folder i q The cosine distance is calculated for all vectors in the set, resulting in the set. I nq For folder i q The number of vectors in the middle. It refers to folder i q The vector corresponding to the g-th image in the dataset; S12, Set Sort the images by distance values ​​from largest to smallest, and select the images corresponding to the top 5 vectors with the highest distance values ​​as the prediction results.

2. The vehicle face recognition method for obscured license plates according to claim 1, characterized in that, The backbone network structure, from the input layer to the output layer, is connected sequentially as follows: convolutional layer conv1_1, ReLU layer conv1_1_relu, convolutional layer conv1_2, ReLU layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, ReLU layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, ReLU layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, ReLU layer conv3_1 _relu, convolutional layer conv3_2, Relu layer conv3_2_relu, convolutional layer conv3_3, Relu layer conv3_3_relu, pooling layer max_pooling3, convolutional layer conv4_1, Relu layer conv4_1_relu, convolutional layer conv4_2, Relu layer conv4_2_relu, convolutional layer conv4_3, Relu layer conv4_3_relu, pooling layer max_pooling4, convolutional layer conv5_1, Relu layer conv5_1_ ReLU, convolutional layer conv5_2, ReLU layer conv5_2_relu, convolutional layer conv5_3, ReLU layer conv5_3_relu, pooling layer max_pooling5, convolutional layer fc6, ReLU layer fc6_relu, convolutional layer fc7, ReLU layer fc7_relu, convolutional layer conv6_1, ReLU layer conv6_1_relu, convolutional layer conv6_2, ReLU layer conv6_2_relu, convolutional layer conv7_1, ReLU layer conv7_1_relu Convolutional layer conv7_2, ReLU layer conv7_2_relu, convolutional layer conv8_1, ReLU layer conv8_1_relu, convolutional layer conv8_2, ReLU layer conv8_2_relu, convolutional layer conv9_1, ReLU layer conv9_1_relu, convolutional layer conv9_2, ReLU layer conv9_2_relu, convolutional layer conv10_1, ReLU layer conv10_1_relu, convolutional layer conv10_2, ReLU layer conv10_2_relu.

3. The vehicle face recognition method for obscured license plates according to claim 1, characterized in that, The color prediction module structure, from the input layer to the output layer, is connected in the following order: pooling layer global_pooling_1, fc layer fc_1, and fc layer fc_2.

4. The vehicle face recognition method for obscured license plates according to claim 1, characterized in that, The global perception module structure, from the input layer to the output layer, is connected as follows: pooling layer global_pooling_2, multi-dimensional normalization layer conv1_1_in_bn, convolutional layer conv11_1, ReLU layer conv11_1_relu, convolutional layer conv11_2, and ReLU layer conv11_2_relu; wherein, the multi-dimensional normalization layer conv1_1_in_bn is composed of BN layer conv11_1_bn and IN layer conv11_1_in.

5. A vehicle face recognition method for obscured license plates according to claim 3, characterized in that, The process of step S2 is as follows: S21, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_1 to obtain the feature map E. a ∈R C×1 ; S22, Transfer feature map E a The input is fed into the fully connected layers fc_1 and fc_2, and the vector E is obtained through the softmax function. f ∈R N×1 , where N is the number of vehicle colors in the dataset; S23, in vector E f Find the index i corresponding to the highest value in the table and output the vehicle color result i.

6. A vehicle face recognition method for obscured license plates according to claim 4, characterized in that, The process of step S3 is as follows: S31, Transfer feature map F p The input is fed into the global average pooling layer global_pooling_2 to obtain the vector V∈R D×1 ; S32. Input vector V into the BN layer conv11_1_bn to obtain vector V. b ∈R D×1 ; S33. Input vector V into the IN layer conv11_1_in to obtain vector V. i ∈R D×1 ; S34, Transform vector V b With vector V i The concatenation yields vector V. c ∈R 2D×1 ; S35, Vector V c The vector Q1 is obtained by passing a convolutional layer conv11_1, a ReLU layer conv11_1_relu, a convolutional layer conv11_2, and a ReLU layer conv11_2_relu.

7. A vehicle face recognition method for obscured license plates according to claim 1, characterized in that, The process of step S4 is as follows: S41, Transfer feature map F p The input is fed into the feature detail compression layer horizontal_pooling_1 and the batch normalization layer conv12_1_bn to obtain the feature map J. s ∈R C×H×1 ; S42, Transfer feature map J s The input is fed into a convolutional layer conv12_1, a ReLU layer conv12_1_relu, a convolutional layer conv12_2, and a ReLU layer conv12_2_relu to obtain the vector Q2.

8. A vehicle face recognition method for obscured license plates according to claim 1, characterized in that, The calculation process for the cosine distance is as follows: Suppose we calculate the cosine distance between vectors J and B, obtaining the value l. The calculation expression is as follows: l=1-J·B T In the formula, the symbol "·" indicates that the elements in the vector are multiplied one by one, and the superscript "T" indicates the vector transpose operation.

Images