A vehicle re-identification method fusing MobileNet-V2 and svm

Abstract

A vehicle re-identification method fusing MobileNet-V2 and svm relates to the technical field of vehicle re-identification, based on a deep learning network of MobileNet-V2, a multi-branch network is built and multi-branch network training is carried out, a vehicle re-identification model is obtained based on the svm method in the training process, and a retrieval result of a target vehicle is displayed based on the vehicle re-identification model, the identification method guarantees that the network structure is simple, the vehicle re-identification model obtained has better robustness, effectively improves the accuracy and performance, and achieves the balance of effect and performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle re-identification, and particularly relates to a vehicle re-identification method fusing MobileNet-V2 and svm. BACKGROUND

[0002] Vehicle re-identification aims to find the same target vehicle from images across cameras and scenes, and the difficulty lies in that the difference between different vehicle images with similar appearances is small, and the images of the same vehicle under different angles or light conditions also have great differences, which makes the vehicle re-identification more difficult.

[0003] In recent years, with the continuous development of artificial intelligence technology, especially the gradual deepening of deep learning technology, vehicle re-identification technology based on deep learning method has become a research hotspot, such as using deep learning network to extract features, and then calculating the distance between the features for re-identification. The existing technology proposes a vehicle type identification method based on an improved MobileNet network, a vehicle model identification method based on YOLOv5 and MobileNetV2, and a mobilenet-v1 and mobilenet-v2 for the mobilenet algorithm.

[0004] Most of the existing vehicle re-identification technologies use a deep learning model to extract relevant features. However, due to various objective conditions such as occlusion, light, angle and other factors, the robustness of the extracted features is poor. In order to improve the re-identification effect, it is necessary to replace a more complex network structure, which increases the network calculation amount. Therefore, the existing vehicle re-identification technology is difficult to balance the effect and performance. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application proposes a vehicle re-identification method fusing MobileNet-V2 and svm, and the specific scheme is as follows:

[0006] A vehicle re-identification method fusing MobileNet-V2 and svm, based on a deep learning network of MobileNet-V2, a multi-branch network is built and multi-branch network training is performed, a vehicle re-identification model is obtained based on svm method in the training process, and a retrieval result of a target vehicle is displayed based on the vehicle re-identification model.

[0007] Further, the specific steps of the re-identification method include the following:

[0008] Step one is to preprocess the training data: the sample data is preprocessed, and then the sample data preprocessed is calibrated;

[0009] Step two is a multi-branch network construction: a multi-branch network is constructed based on MobileNet-V2 as a basic network, and the multi-branch network includes a global feature branch network and a vehicle component branch network;

[0010] Step three is a multi-branch network training: first, the training image and sample label are sent into the vehicle component branch network to be trained, and when the loss function reaches convergence, a vehicle component detection model is obtained, then the global feature branch network is added to adjust the vehicle component detection model, and when the loss function reaches convergence, a vehicle re-identification model is obtained, that is, one calculation training process is completed, N calculation training processes during model training correspond to N vehicle re-identification models, and then the final vehicle re-identification model is obtained based on the N vehicle re-identification models through svm weight screening;

[0011] Step four is to show the recognition effect: based on the final vehicle re-identification model, a calling code is constructed, the calling code includes reading an image, then performing image preprocessing, then loading the final vehicle re-identification model for feature extraction, and storing the features of each image in file A after repeating the above operations, and then constructing a matching code, the matching code includes reading an image, then performing image preprocessing, then loading the final vehicle re-identification model for feature extraction, importing file A, performing feature similarity calculation, taking the top 100 images with high similarity, and writing the obtained 100 images to a large image for result display.

[0012] Further, in step one, the image preprocessing includes image scaling, image format conversion, and image rotation.

[0013] The image scaling is to scale the obtained snapshot image by using a bilinear interpolation algorithm to simulate snapshot images in different actual scenes;

[0014] The image format conversion is to convert images of different formats into a unified format based on a conversion formula;

[0015] The image rotation is to rotate the image based on the Cartesian coordinate system with the center of the image as the center.

[0016] Further, the data labeling is set as region box labeling, which can include car face region box labeling, car headlight region box labeling, car wheel box labeling, and vehicle rearview mirror labeling.

[0017] Further, when each region box is labeled, the box should completely include the entire corresponding vehicle part, and when the area of the vehicle part is less than one third of the entire wheel area, no labeling is performed.

[0018] Further, in the step three, the obtained vehicle part detection model is verified on the test set, the test set sets the vehicle sub-region position frame in different scenes as the answer, matches the detection frame predicted by the vehicle part detection model with the sub-region position frame, and judges whether the current vehicle part detection model reaches the expected effect according to the accuracy of different coincidence thresholds.

[0019] Further, in the step three, the obtained vehicle part detection model is verified on the test set, the test set sets the vehicle sub-region position frame in different scenes as the answer, matches the detection frame predicted by the vehicle part detection model with the sub-region position frame, and judges whether the current vehicle part detection model reaches the expected effect according to the accuracy of different coincidence thresholds.

[0020] Further, in the step three, the obtained vehicle part detection model is verified on the test set, the test set sets the vehicle sub-region position frame in different scenes as the answer, matches the detection frame predicted by the vehicle part detection model with the sub-region position frame, and judges whether the current vehicle part detection model reaches the expected effect according to the accuracy of different coincidence thresholds.

[0021] Compared with the prior art, the beneficial effects of the present application are as follows:

[0022] (1) Compared with the prior art, the vehicle re-identification model is mostly directly trained on the vehicle region, based on the above analysis, the vehicle re-identification method provided by the present application adopts the structure based on MobileNet-V2 network, and fuses svm processing, so that the vehicle re-identification model obtained on the basis of ensuring the simple network structure has better robustness, effectively improves the accuracy and performance, and achieves the balance between effect and performance. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The bilinear interpolation graph in the present application;

[0024] Figure 2 The formula graph obtained by linear interpolation in the x direction in image scaling;

[0025] Figure 3 The formula graph obtained by linear interpolation in the y direction in image scaling;

[0026] Figure 4A formulaic diagram of the result f(x,y) in image scaling;

[0027] Figure 5 A Cartesian coordinate system diagram of the image during rotation;

[0028] Figure 6 A diagram showing the transformation relationship between the midpoint (x', y') of the matrix and the Cartesian coordinate system (x, y) during image rotation;

[0029] Figure 7 A diagram showing the transformation relationship between the Cartesian coordinate system (x,y) and the midpoint (x',y') of a matrix during image rotation;

[0030] Figure 8 A schematic diagram showing the calibration of each sub-region on the vehicle;

[0031] Figure 9 This is a schematic diagram of the original MobileNet-V2 network architecture;

[0032] Figure 10 This is a diagram of the overall network structure of the present invention;

[0033] Figure 11 This is a flowchart of the vehicle re-identification model based on the network structure diagram of the present invention. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0035] like Figure 1 As shown, a vehicle re-identification method integrating MobileNet-V2 and SVM is used to find the same target vehicle from images across cameras and scenes. The challenge lies in the small differences between images of different vehicles with similar appearances, and the significant differences between images of the same vehicle under different viewing angles or lighting conditions. Therefore, this invention builds a multi-branch network based on the MobileNet-V2 deep learning network and trains it. During the training process, a vehicle re-identification model is obtained based on the SVM method, and the retrieval results for the target vehicle are displayed based on the vehicle re-identification model.

[0036] To obtain the vehicle re-identification model described above, the specific steps of the re-identification method include the following: Step 1 is training data preprocessing; Step 2 is multi-branch network construction; Step 3 is multi-branch network training; and Step 4 is recognition result demonstration. Each step is explained in detail below.

[0037] The training data preprocessing specifically comprises image preprocessing of sample data, and then data calibration of the sample data subjected to the image preprocessing. The setting of step one can effectively enhance the richness and diversity of the sample data used in step three, so that the training data is closer to various actual scenes and is more robust.

[0038] It should be noted that in step one, the image preprocessing includes image scaling, image format conversion and image rotation. The three processing methods are aimed at vehicle real image, and the three preprocessing operations can be performed or selected, depending on actual needs, and the order of the three preprocessing operations is also determined according to actual needs, which will be specifically described as follows:

[0039] In actual scenes, the position of the vehicle in the image captured by the camera is not fixed, and the size of the vehicle in the captured image is not fixed. In order to enable the model to detect vehicles in different scenes, the image scaling is performed by using a bilinear interpolation algorithm to scale the obtained captured image, so as to simulate the captured image in different actual scenes, so that the sample data used for training is more rich and closer to the actual scene image.

[0040] Bilinear interpolation is also called bilinear interpolation. In mathematics, bilinear interpolation is a linear interpolation extension of the interpolation function with two variables. The core idea is to perform linear interpolation in two directions respectively.

[0041] As shown in Figure 1 , Q11, Q12, Q21, Q22 are known data points, and P is the point to be solved. If the value of the unknown function f at point P=(x,y) is to be obtained, the values of red points Q11=(x1,y1), Q12=(x1,y2), Q21=(x2,y1) and Q22=(x2,y2) are known. First, linear interpolation is performed in the x direction to obtain the algorithm as shown in Figure 2 . Secondly, linear interpolation is performed in the y direction to obtain the algorithm as shown in Figure 3 . In this way, the desired result f(x,y) is obtained, as shown in the algorithm as shown in Figure 4 . The image scaling is realized based on the algorithm as shown in Figure 4 .

[0042] In actual use, the format of the image sent into the vehicle attribute recognition is various, such as nv12, rgb, yuv, etc. In order to effectively avoid the difference in vehicle attribute recognition results caused by different formats of images, the image format conversion operation is added during image preprocessing, so as to increase the robustness of the sample. The image format conversion is based on a conversion formula to convert different formats of images into a unified format. For example, the RGB to YUV formula is as follows:

[0043] Y = 0.299R + 0.587G + 0.114B

[0044] U = -0.1687R - 0.3313G + 0.5B + ​​128

[0045] V = 0.5R - 0.4187G - 0.0813B + 128

[0046] The formula for converting YUV to RGB is as follows:

[0047] R = Y + 1.402(V - 128)

[0048] G=Y-0.34414(U-128)-0.71414(V-128)

[0049] B = Y + 1.772(U - 128)

[0050] As for the conversion formulas between other formats, this invention will not elaborate on them one by one.

[0051] Because the camera is mounted at different angles, the vehicles appear in the images from various angles. In order to better identify the attributes of vehicles at different angles, this invention sets up an image rotation function for the sample data, performing rotation operations to make the sample data more diverse.

[0052] Image rotation is based on a Cartesian coordinate system and involves rotating the image around its centroid. Specifically, it involves rotating the image around its center, such as... Figure 5 As shown, the coordinate system in a matrix is ​​usually along the AD and AB directions, while the traditional Cartesian coordinate system is established at the center of the matrix. Let the image be represented as an M×N matrix. For point A, the coordinates in the two coordinate systems are (0, 0) and (-N / 2, M / 2), respectively. The transformation relationship from point (x', y') in the matrix to (x, y) in the Cartesian coordinate system is as follows: Figure 6 As shown. The inverse transformation formula for converting the Cartesian coordinate system (x,y) to the midpoint (x',y') of a matrix is ​​as follows: Figure 7 As shown. In this way, the aforementioned image rotation function can be achieved.

[0053] As for the data calibration in step one, it is set as region box calibration. Region box calibration is to obtain the correct answers for the detection of each sub-region corresponding to the vehicle parts in the specimen label. All correct answers are summarized to obtain the test set, which serves as a reference for the prediction results in the training process of the multi-branch network in the subsequent step three.

[0054] The region frame labeling includes, but is not limited to, vehicle face region frame labeling, vehicle headlamp region frame labeling, vehicle wheel frame labeling and vehicle rearview mirror labeling. In each region frame labeling, the frame needs to completely include the entire corresponding vehicle part, and when the area of the vehicle part is less than one third of the entire wheel area, no labeling is performed. For example, the wheel frame needs to completely include the entire wheel, and when the area of the wheel is less than one third of the entire wheel area, no labeling is performed. Figure 8 The schematic diagram is labeled for each sub-region.

[0055] The multi-branch network is built based on MobileNet-V2 as a basic network, and the multi-branch network includes a global feature branch network and a vehicle component branch network. Due to the setting of step two, the vehicle component branch network is built, which not only detects each sub-region, but also plays a supervisory role on the global feature branch network, effectively improving the effect of recognizing global features.

[0056] In the application, the improved MobileNet-V2 is used as a basic network to build a multi-branch network structure for vehicle re-identification analysis. In order to ensure detection accuracy while effectively improving operation speed, the MobileNet-V2 is channel pruned and structure combined in the patent, so as to have faster inference speed. Figure 9 The network structure diagram of the original MobileNet-V2 is shown. First, the image input resolution is changed, and the resolution image of 384*192*3 is used as input in the application. At the same time, in order to improve the network running speed and reduce the calculation amount, the channel number in the structure is optimized, and a set of channel combination modes matching the effect and performance is obtained through experiments. At the same time, part of the bottleneck in the original network structure is deleted. After the feature layer is 96*48 size, one branch connection region frame branch is used for region frame detection, and the other branch is a global feature branch.

[0057] In summary, the network structure of MobileNet-V2 is obtained through channel pruning and structure combination, and the overall network structure diagram of the application is shown in Figure 10

[0058] ​The multi-branch network structure is constructed for vehicle re-identification, and therefore, a step-by-step training strategy is adopted in the training process to improve the accuracy of vehicle re-identification. The multi-branch network training is specifically as follows. First, the training image and sample label are sent to the vehicle component branch network to be built to perform training, wherein the training image refers to a vehicle picture, and the sample label refers to a labeled answer of each vehicle picture, which refers to a vehicle sub-region position box and an ID number corresponding to each vehicle picture. Through the sample label, such as coordinate position and other information of each vehicle picture, the network can be fed back in the training process that the vehicle picture needs to be trained. In the training process, when the loss function reaches convergence, a vehicle component detection model is obtained.

[0059] The obtained vehicle component detection model is optimized, and the effect verification is performed on a test set. The test set labels vehicle sub-region position boxes in different scenes as answers. The detection boxes predicted by the vehicle component detection model are matched with the sub-region position boxes, and the accuracy according to different coincidence thresholds is used to determine whether the current vehicle component detection model achieves the expected effect.

[0060] Since the vehicle component detection model has been trained, the detection of each sub-region box plays a role in alignment to a certain extent, and also enhances the response between regions, so the stable output of the vehicle component detection model is beneficial to the extraction of global features. Then, a global feature branch network is added to adjust the vehicle component detection model. When the loss function reaches convergence, a vehicle re-identification model is obtained, that is, one calculation training process is completed. N calculation training processes in the model training correspond to N vehicle re-identification models. Then, the final vehicle re-identification model is obtained based on the N vehicle re-identification models.

[0061] The obtained vehicle re-identification model is optimized, and the effect verification is performed on a test set. The test set labels M ID vehicles, each ID has multiple vehicle images of the same vehicle taken by a camera in different scenes, and then has N random vehicle images. The N random vehicle images and all vehicle images corresponding to each ID are extracted by the obtained vehicle re-identification model, and the features are stored in a file A. Then, the features of each vehicle image labeled under the M IDs are extracted one by one, and the extracted features are compared with the file A. According to the order from high to low of the similarity scores, the top 100 are taken, and finally it is checked whether the vehicle images under each ID are searched from other remaining vehicle images and ranked in the front. The accuracy of Top1, Top5 and Top10 is counted to verify the effect of the vehicle re-identification model.

[0062] The foregoing accuracy and expected effect are not limited by the present application, and are reasonably set in actual operation.

[0063] During model training, each training calculation process saves a model, and when the network structure converges, N vehicle re-identification models are obtained. Since the number of obtained vehicle re-identification models is large, the present application finds, through experiments, that the eigenvalues of different models differ with the number of training times. Through continuous experiments, it is found that better results can be obtained if a series of post-processing is performed on the features of each layer of each model. Therefore, as shown in Figure 11 the present application sets up svm weight screening, which screens each layer feature of the N vehicle re-identification models to obtain the best feature weight of each layer, and then rewrites the final feature weight of each layer obtained into the vehicle re-identification model, so as to obtain the final vehicle re-identification final model.

[0064] For example, if 100 vehicle re-identification models are obtained during the entire training process, each vehicle re-identification model contains 80 layers. Taking the 15th layer as an example, the features of the 15th layer of the 100 models are extracted, and a set of final features is obtained by using the svm method. This set of features is the final feature of the 15th layer. Other layers are similar. Then, the new features of the 80 layers obtained are written into a new vehicle re-identification model according to the order, and this model is the final vehicle re-identification model.

[0065] The global feature of the present application is a 1024-bit floating-point eigenvalue, and 64-bit floating-point eigenvalues are extracted for each component area. The component areas of the present application include but are not limited to vehicle rearview mirror area, vehicle lamp area, vehicle face area, vehicle wheel area, and N vehicle component areas.

[0066] After obtaining a stable vehicle re-identification model, the recognition effect display of step four can be performed. The recognition effect display is specifically based on the final vehicle re-identification model, building a calling code, the calling code includes reading an image, then performing image preprocessing, then loading the final vehicle re-identification model for feature extraction, and storing the features of each image in file A after repeating the above operation. Then, a matching code is built, which includes reading an image, then performing image preprocessing, then loading the final vehicle re-identification model for feature extraction, importing file A, performing feature similarity calculation, taking the top 100 images with high similarity, and writing the obtained 100 images into a large image for result display.

[0067] In summary, the core of the present application is to balance the effects of each vehicle re-identification model by using the svm algorithm on multiple vehicle re-identification models, so that the final vehicle re-identification model has better robustness. Therefore, the vehicle re-identification method of the present application improves the search accuracy by 1 to 2 percentage points based on the vehicle re-identification model.

[0068] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-described embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A vehicle re-identification method fusing MobileNet-V2 and svm, characterized in that, The deep learning network based on MobileNet-V2 is used to build a multi-branch network and train the multi-branch network, and a vehicle re-identification model is obtained based on an svm method in the training process, and a retrieval result of a target vehicle is displayed based on the vehicle re-identification model; The specific steps of the re-identification method include the following: Step one is training data preprocessing: the sample data is preprocessed, and then the sample data after image preprocessing is calibrated; Step two is multi-branch network construction: a multi-branch network is built based on MobileNet-V2, and the multi-branch network includes a global feature branch network and a vehicle component branch network; Step three is multi-branch network training: first, the training image and sample label are sent into the built vehicle component branch network for training, and when the loss function reaches convergence, a vehicle component detection model is obtained, then the global feature branch network is added to adjust the vehicle component detection model, and when the loss function reaches convergence, a vehicle re-identification model is obtained, that is, one calculation training process is completed, N calculation training processes during model training correspond to N vehicle re-identification models, and then the final vehicle re-identification model is obtained based on svm weight screening of the N vehicle re-identification models; Step four is to display the identification effect: based on the final vehicle re-identification model, a calling code is built, the calling code includes reading an image, then performing image preprocessing, then loading the final vehicle re-identification model for feature extraction, storing the features of each image in file A by repeating the above operation, then building a matching code, the matching code includes reading an image, then performing image preprocessing, then loading the final vehicle re-identification model for feature extraction, importing file A, calculating the feature similarity, taking the top 100 images with high similarity, and writing the obtained 100 images to a large image for result display.

2. The vehicle re-identification method of claim 1, wherein, In step one, image preprocessing includes image scaling, image format conversion, and image rotation; Image scaling uses a bilinear interpolation algorithm to scale the obtained snapshot image to simulate snapshot images in different actual scenes; Image format conversion converts images of different formats into a unified format based on a conversion formula; Image rotation is based on the Cartesian coordinate system and rotates around the center of gravity of the image. 3.The vehicle re-identification method of claim 2, wherein, The data calibration is set as regional box calibration, including vehicle face regional box calibration, vehicle headlight regional box calibration, vehicle wheel box calibration, and vehicle rearview mirror calibration.

4. The vehicle re-identification method of claim 3, wherein, When each regional box is calibrated, the box should completely include the entire corresponding vehicle part, and when the area of the vehicle part is less than one-third of the entire wheel area, no calibration is performed. 5.The vehicle re-identification method of claim 1, wherein, In step three, the vehicle component detection model obtained is verified on a test set, the test set calibrates the position box of each sub-region of the vehicle in different scenes as the answer, the detection box predicted by the vehicle component detection model is matched with the sub-region position box, and whether the current vehicle component detection model meets the expected effect is judged according to the accuracy of different coincidence thresholds.

6. The vehicle re-identification method of claim 5, wherein, In the third step, the obtained vehicle re-identification model is verified for effect on the test set, the test set calibrates M ID vehicles, each ID has multiple vehicle images of the same vehicle taken by different cameras in different scenes, then there are N random vehicle images, the features of the N random vehicle images and all vehicle images corresponding to each ID are extracted by the obtained vehicle re-identification model and stored in a file A, then the features of one vehicle image under each of the calibrated M IDs are extracted one by one, the extracted features are compared with the file A, and the order is sorted according to the similarity score from high to low, the top 100 is taken, and finally it is checked whether the remaining vehicle images under each ID are searched and sorted in front, and the accuracy of Top1, Top5 and Top10 is counted to verify the effect of the vehicle re-identification model.

7. The vehicle re-identification method of claim 6, wherein, In the third step, the svm weight screening is to screen each layer feature of N vehicle re-identification models to obtain the best feature weight of each layer, and then the obtained final feature weight of each layer is written back to the vehicle re-identification model, that is, the final vehicle re-identification final model is obtained.

Images