A car body scratch detection method based on improved Faster RCNN
By improving the Faster RCNN network, using MobileNet V3 and feature pyramid network to extract features, and combining the RoI Align method, the problem of low accuracy in body scratch detection is solved, and efficient detection on mobile devices is achieved.
Patent Information
- Application Number
- CN202210679662.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-06-15
AI Technical Summary
The existing body scratch detection methods are not very accurate, and the models trained by traditional object detection algorithms are huge in size and difficult to reason and use on mobile devices.
Using the improved Faster RCNN network, the feature extraction layer uses the MobileNet V3 network integrated with the feature pyramid network, combined with the RoI Align method, cancels the quantization operation of the candidate box, and uses a bilinear interpolation algorithm to improve detection accuracy.
It improves the accuracy and usability of body scratch detection, making the detection results more accurate and suitable for inference on mobile devices.
Smart Images

Figure CN115131623B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision target detection, and more specifically to a vehicle body scratch detection method based on improved Faster RCNN. Background Art
[0002] In recent years, the problem of damage assessment of damaged vehicles has attracted the attention of a large number of domestic and foreign companies. For example, Snapsheet, an American company, has built a virtual insurance claim service point, which integrates remote information services, image detection and other technologies to assist companies in efficiently handling insurance claims. Spex, an American company, has built a digital survey and reporting platform, which consists of two parts: Spex control end and Spex investigation end. The control end is mainly responsible for business division and online cooperation between administrators and investigators. The investigation end mainly provides investigators with various necessary services and collects and processes images and other data collected by investigators, and finally generates a claim plan. ClaimDi, a Thai company, has developed a mobile claims software based on LBS technology and the Uber model, which enables rapid compensation for small cases. When users use the software to report a case, they can use positioning technology to dispatch the nearest surveyor to conduct an investigation. Although the above methods have improved efficiency compared to traditional methods, relevant personnel still need to be dispatched to the scene to assess the damage. In China, taking "Dingsunbao" as an example, users upload panoramic views of the vehicle and photos of damaged parts to the cloud server, and obtain the claims results after evaluation by traditional target detection algorithms. Common target detection algorithms include SSD, Faster RCNN, etc. Although traditional damage assessment methods can meet the detection of most damaged parts of vehicles, some scratches on the body have unclear damage features, which makes traditional algorithms unable to accurately identify and locate them. In addition, the model trained using traditional target detection algorithms is large in size and difficult to use for inference on mobile devices.
[0003] Therefore, how to solve the problem that the existing detection methods have low accuracy and poor usability in detecting scratches on vehicle bodies is an urgent problem that technicians in this field need to solve. Summary of the invention
[0004] In view of this, the present invention provides a vehicle body scratch detection method based on improved Faster RCNN, which is used to at least partially solve the technical problems existing in the background technology.
[0005] In order to achieve the above object, the present invention adopts the following technical solution:
[0006] A vehicle body scratch detection method based on improved Faster RCNN includes the following steps:
[0007] S1, obtain the image to be detected and the vehicle body scratch dataset with labels;
[0008] S2. Constructing an improved Faster RCNN network model, wherein the improved Faster RCNN network model specifically includes: the feature extraction layer of the Faster RCNN network adopts an M-FPN network, and the M-FPN network is a MobileNet V3 network fused with a feature pyramid network;
[0009] S3, using the labeled car body scratch dataset as training samples to train the improved Faster RCNN network model;
[0010] S4. Input the image to be detected into the trained improved Faster RCNN network model to obtain the vehicle body scratch detection result.
[0011] Preferably, the specific steps of integrating the MobileNet V3 network with the feature pyramid network in step S2 include:
[0012] The MobileNet V3 network is layered according to the dimension size of the output feature map of the MobileNet V3 network;
[0013] For each layer, the feature matrix of the corresponding convolution layer output is increased or decreased in dimension through a convolution kernel of 1*1*256 and a step size of 1, so that the dimension of the feature matrix output by each layer is the same;
[0014] An upsampling operation is performed on the feature matrix of the previous layer with the same dimension to obtain a feature matrix with the same spatial size as the feature matrix of the next layer and feature fusion is performed;
[0015] Perform 3*3*256 convolution operations on the feature matrix after feature fusion to obtain the new feature matrix corresponding to each layer of the MobileNet V3 network output feature map;
[0016] The feature matrix with the highest dimension after the 3*3*256 convolution operation is passed through a 1*1*256 convolution layer with a stride of 2 and downsampled to obtain an additional feature matrix;
[0017] The new feature matrix and the additional feature matrix are output as feature maps of corresponding scales to achieve the fusion of the MobileNet V3 network and the feature pyramid network.
[0018] Preferably, the training and improving of the Faster RCNN network model in step S3 specifically includes the following steps:
[0019] S31. Use the ImageNet dataset to pre-train the improved Faster RCNN network model, obtain the pre-training weights, and use the pre-training weights to initialize the RPN network of the improved Faster RCNN network model. On this basis, use the labeled body scratch dataset as a training sample to separately train the RPN network layer parameters of the improved Faster RCNN network model.
[0020] S32, fix the convolutional layer and fully connected layer parameters unique to the RPN network, and use the target proposal box generated by the RPN network to train the Fast RCNN network parameters;
[0021] S33. Use the pre-convolutional network layer parameters trained by Fast RCNN to fine-tune the convolutional layer and fully connected layer parameters unique to the RPN network;
[0022] S34, keep the parameters of the front convolutional network layer fixed, and fine-tune the parameters of the fully connected layer of the Fast RCNN network;
[0023] The S35, RPN network and Fast RCNN network share the parameters of the front convolutional network layer to form a unified network.
[0024] Preferably, step S4 specifically includes:
[0025] S41, input the image to be detected into the M-FPN network to obtain the corresponding feature matrix;
[0026] S42, using the RPN network to generate a candidate box with scratches on the vehicle body;
[0027] S43. Input the candidate box and the feature matrix into the trained Fast RCNN network to fine-tune the candidate box, detect the category of the body scratches in the candidate box, and obtain the final detection result.
[0028] Preferably, the above method also includes: replacing the RoI Pooling layer of the improved Faster RCNN network with RoI Align, wherein the RoI Align cancels the quantization operation on the candidate box boundary and the boundary coordinates of each unit, uses a bilinear interpolation algorithm to calculate the boundary coordinates of each unit, and then performs a maximum pooling operation, and finally inputs the obtained feature matrix of the same size into the improved Fast RCNN network.
[0029] The quantization in the present invention can be understood as a rounding operation on data, for example, 1.78 becomes 1 after the quantization operation.
[0030] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a vehicle body scratch detection method based on improved Faster RCNN, which has the following beneficial effects:
[0031] 1. Based on the performance of Faster RCNN, in order to further improve the usability of Faster RCNN in the actual application scenarios of car body scratch detection, the present invention introduces the MobileNet-V3 network suitable for mobile terminal reasoning to replace the VGG and ZF networks commonly used in the feature extraction layer of Faster RCNN, and proposes an M-FPN network structure, that is, the present invention uses the MobileNet V3 network integrated with the feature pyramid network FPN of the multi-scale target detection algorithm to replace the VGG and ZF networks commonly used in the feature extraction layer of Faster RCNN, so as to better extract the feature information of some smaller scratches and the detailed information about the scratch area, thereby improving the accuracy and usability of the existing detection method.
[0032] 2. The present invention introduces the RoI Align method, cancels the two quantization operations on the target candidate box in the Faster RCNN algorithm, and uses the bilinear interpolation method to improve the accuracy of target positioning in the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0034] Figure 1 A flow chart of the method provided by the present invention;
[0035] Figure 2 The M-FPN network framework diagram provided by the embodiment of the present invention;
[0036] Figure 3 The overall framework diagram of the improved Faster RCNN algorithm provided in the embodiment of the present invention. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] like Figure 1 As shown, the present invention discloses a vehicle body scratch detection method based on improved Faster RCNN, comprising the following steps:
[0039] S1, obtain the image to be detected and the vehicle body scratch dataset with labels;
[0040] In a specific embodiment, the labeled vehicle body scratch dataset can be constructed by expanding the dataset with only scratch type labels provided by the internship enterprise using data enhancement technology.
[0041] S2. Constructing an improved Faster RCNN network model, wherein the improved Faster RCNN network model specifically includes: the feature extraction layer of the Faster RCNN network adopts an M-FPN network, and the M-FPN network is a MobileNet V3 network fused with a feature pyramid network;
[0042] The specific steps of integrating the MobileNet V3 network with the feature pyramid network include:
[0043] The MobileNet V3 network is layered according to the dimension size of the output feature map of the MobileNet V3 network;
[0044] For each layer, the feature matrix of the convolution layer output is reduced in dimension through a convolution kernel of 1*1*256 and a step size of 1;
[0045] Perform upsampling operation on the feature matrix after dimension reduction in each layer to obtain the feature matrix with the same dimension as the previous layer and perform feature fusion;
[0046] Perform 3*3*256 convolution operations on the feature matrix after feature fusion to obtain the new feature matrix corresponding to the feature dimension of each layer of the MobileNet V3 network;
[0047] The feature matrix with the highest dimension after the 3*3*256 convolution operation is downsampled to 1*1*256 with a step size of 2 to obtain an additional feature matrix;
[0048] The new feature matrix and the attached feature matrix are output as feature maps of corresponding scales to achieve the fusion of the MobileNet V3 network and the feature pyramid network.
[0049] S3, using the labeled car body scratch dataset as training samples to train the improved Faster RCNN network model;
[0050] The specific steps include:
[0051] S31. Use the ImageNet dataset to pre-train the improved Faster RCNN network model, obtain the pre-training weights, and use the pre-training weights to initialize the RPN network of the improved Faster RCNN network model. On this basis, use the labeled body scratch dataset as a training sample to separately train the RPN network layer parameters of the improved Faster RCNN network model.
[0052] S32, fix the convolutional layer and fully connected layer parameters unique to the RPN network, and use the target proposal box generated by the RPN network to train the Fast RCNN network parameters;
[0053] S33. Use the pre-convolutional network layer parameters trained by Fast RCNN to fine-tune the convolutional layer and fully connected layer parameters unique to the RPN network;
[0054] S34, keep the parameters of the front convolutional network layer fixed, and fine-tune the parameters of the fully connected layer of the Fast RCNN network;
[0055] The S35, RPN network and Fast RCNN network share the parameters of the front convolutional network layer to form a unified network.
[0056] Because the car body scratch data is less than ImageNet data, the present invention first uses the ImageNet data set to train the network to obtain the pre-trained weights of the network, and then uses the scratch data set for training to further adjust the obtained weights. Both data sets need to participate in the training of the entire network, where the pre-convolutional network layer parameters are the weights of the network layer. The purpose of training the network is to adjust the weights of each layer of the network so that the network can correctly detect the category and location of the car body scratches. Using two training sets during the training process improves the accuracy of network prediction.
[0057] S4. Input the image to be detected into the trained improved Faster RCNN network model to obtain the vehicle body scratch detection result. Specifically including:
[0058] S41, input the image to be detected into the M-FPN network to obtain the corresponding feature matrix;
[0059] S42, using the RPN network to generate a candidate box with scratches on the vehicle body;
[0060] S43. Input the candidate box and the feature matrix into the trained Fast RCNN network to fine-tune the candidate box, detect the category of the body scratches in the candidate box, and obtain the final detection result.
[0061] The following is a combination of specific embodiments and Figure 2 , Figure 3, the technical solution of the present invention is further described:
[0062] The traditional Faster RCNN network model uses classic VGG Net and ZF Net to extract features from the original image. The amount of parameter calculation is very huge, and the resulting network model is too large to be suitable for forward propagation on mobile devices. The sizes of scratches on the car body vary, and the Faster RCNN algorithm only uses the feature map obtained by the last layer of convolution of the feature extraction network. This causes the feature information input to the RPN network to lack some small-sized scratch feature information and detailed information about the scratch area, thereby reducing the accuracy of the model's positioning and detection of various types of scratches on the car body. In order to be able to extract more feature information of various types of scratches in the feature extraction layer and reduce the amount of calculation of the network model; the present invention uses MobileNet V3 as the feature extraction layer of the network. MobileNet V3 is a lightweight network structure that is optimized based on the inverted residual structure proposed by the MobileNet V2 network. The feature matrix is dimensionally upgraded through a 1*1 convolution layer, and the dimensionally upgraded feature matrix enters the depthwise separable convolution (Depthwise Separable Convolution). Convolution is referred to as dw convolution. After the convolution layer, the SE module is introduced to perform pooling on the predicted feature layer. After two fully connected layers, the weight value of each layer of the feature matrix is obtained, and each feature layer is multiplied by the feature matrix before pooling. Finally, the 1*1 convolution layer without activation function is used for dimensionality reduction to obtain the same feature matrix as the input matrix. Finally, the input matrix is added to the feature matrix output by the inverted residual block. This not only effectively reduces the number of network parameters and calculations through dw convolution, but also solves the problem that most of the convolution kernel parameters of the dw convolution part will be 0 during the training process to a certain extent, ensuring that the effective features of the feature matrix are not lost.
[0063] On this basis, the present invention integrates the Feature Pyramid Network (FPN) with the MobileNet V3 network, that is, proposes an M-FPN network model.
[0064] Low-dimensional feature maps lose less detail information in the original image (good for positioning) but have weaker semantic features (good for classification), while high-dimensional feature maps lose more detail information but have stronger semantic features. Therefore, the MobileNetV3 network structure is divided into three categories as shown in Table 1 to obtain feature maps of different depths. Taking the input image size of 640*640*3 as an example, the entire MobileNet V3 network is divided into three categories (layers) of convolution blocks according to the size of the output feature map. The feature matrix output by each category (layer) of convolution layer is combined with the FPN network to obtain four types of feature maps.
[0065] Specifically, Figure 2 As shown in the figure, taking the input image size of 640*640*3 as an example, through the above three types of convolution blocks, three feature matrices of 80*80*40, 40*40*112, and 20*20*960 are obtained respectively, and then the feature matrices with dimensions of 256 are obtained by dimensionality increase or dimensionality reduction through the convolution kernel of 1*1*256 and stride 1, and then the feature matrix of the previous layer is upsampled (Upsample) to obtain a feature matrix with the same dimension (shape) as the feature matrix of the next layer and a feature fusion operation is performed (indicated by "+" in the figure). Finally, a new feature matrix (P1, P2, P3) corresponding to each layer is obtained through 3*3*256 convolution. The aliasing effect of upsampling can be reduced through 3*3*256 convolution, and the P3 feature map is obtained through a convolution layer of 1*1*256 and stride 2. The feature matrix P4 of 10*10*256 is obtained, and finally 4 feature maps of different scales are obtained, realizing the combination of MobileNet V3 network and FPN network.
[0066]
[0067] Table 1 MobileNet V3 network classification table
[0068] The above feature map is input into the RPN network. First, a 3*3 sliding window with a step size of 1 is used to obtain the feature vector. At the same time, for each 3*3 sliding window on the feature map, the mapping relationship between the center point of the sliding window and the corresponding center point on the original image is calculated, and k rectangular boxes on the pixel point are calculated. Rectangular boxes with sizes of {322, 642, 1282, 2562, 5122} are used to correspond to aspect ratios of {1:2, 1:1, 2:1}, and rectangular boxes are generated on the corresponding feature matrix in turn. Then, the probability of the target being the foreground or background and the regression parameters of the target suggestion box are obtained through two fully connected layers. The predicted regression parameters include the coordinates of the center point of the suggestion box and the width and height. Therefore, each feature matrix will obtain 2k probability scores and 4k suggestion box regression parameters. After ignoring the rectangular boxes that cross the boundary, the first n candidate boxes are selected according to the probability score of the target being the foreground corresponding to each region candidate box. 2000 candidate boxes are selected during training and 1000 candidate boxes are selected during testing. Calculate the IoU value (Intersection over Union) between each candidate box and the real target box, and select the rectangular boxes with IoU values greater than the specified threshold as candidate boxes. The detection process of the improved Faster RCNN network model input into the image to be detected is consistent with the above process.
[0069] like Figure 3As shown in the figure, for the RoI Pooling layer used in the traditional Faster RCNN algorithm, the present invention introduces RoIAlign, uses bilinear interpolation to calculate the boundary coordinates of each unit, and then performs the maximum pooling operation, and finally inputs the obtained feature matrix of the same size into the Fast RCNN network. The feature matrix obtained by the pooling layer predicts the target category in each candidate frame through the fully connected layer and softmax to obtain the category probability, and at the same time obtains the regression parameters of the candidate frame through another fully connected layer, and further adjusts the candidate frame to improve the accuracy of the candidate frame.
[0070] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0071] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A vehicle body scratch detection method based on improved Faster RCNN, characterized in that: The following steps are involved: S1, obtain the image to be detected and the vehicle body scratch dataset with labels; S2. Construct an improved Faster RCNN network model, wherein the improved Faster RCNN network model specifically includes: the feature extraction layer of the Faster RCNN network adopts an M-FPN network, and the M-FPN network is a MobileNet V3 network integrated with a feature pyramid network; wherein the specific steps of integrating the MobileNet V3 network with the feature pyramid network include: The MobileNet V3 network is layered according to the dimension size of the MobileNet V3 network output feature map; For each layer, the feature matrix of the convolution layer output corresponding to each layer is increased or decreased in dimension through a convolution kernel of 1*1*256 and a step size of 1, so that the dimension of the feature matrix output by each layer is the same; An upsampling operation is performed on the feature matrix of the previous layer with the same dimension to obtain a feature matrix with the same spatial size as the feature matrix of the next layer and feature fusion is performed; Perform 3*3*256 convolution operations on the feature matrix after feature fusion to obtain the new feature matrix corresponding to each layer of the MobileNet V3 network output feature map; The feature matrix with the highest dimension after the 3*3*256 convolution operation is passed through a 1*1*256 convolution layer with a stride of 2 and downsampled to obtain an additional feature matrix; The new feature matrix and the additional feature matrix are output as feature maps of corresponding scales to achieve the fusion of the MobileNet V3 network and the feature pyramid network. S3. Using the labeled car body scratch dataset as training samples to train and improve the Faster RCNN network model; training and improving the Faster RCNN network model specifically includes the following steps: S31. Use the ImageNet dataset to pre-train the improved Faster RCNN network model, obtain the pre-training weights, and use the pre-training weights to initialize the RPN network of the improved Faster RCNN network model. On this basis, use the labeled body scratch dataset as a training sample to separately train the RPN network layer parameters of the improved Faster RCNN network model. S32, fix the convolutional layer and fully connected layer parameters unique to the RPN network, and use the target proposal box generated by the RPN network to train the Fast RCNN network parameters; S33. Use the pre-convolutional network layer parameters trained by Fast RCNN to fine-tune the convolutional layer and fully connected layer parameters unique to the RPN network; S34, keep the parameters of the front convolutional network layer fixed, and fine-tune the parameters of the fully connected layer of the Fast RCNN network; S35, RPN network and Fast RCNN network share the parameters of the front convolutional network layer to form a unified network; S4. Input the image to be detected into the trained improved Faster RCNN network model to obtain the vehicle body scratch detection result.
2. The vehicle body scratch detection method based on improved Faster RCNN according to claim 1, characterized in that: Step S4 specifically includes: S41, input the image to be detected into the M-FPN network to obtain the corresponding feature matrix; S42, using the RPN network to generate a candidate box with scratches on the vehicle body; S43. Input the candidate box and the feature matrix into the trained Fast RCNN network to fine-tune the candidate box, detect the category of the body scratches in the candidate box, and obtain the final detection result.
3. The vehicle body scratch detection method based on improved Faster RCNN according to claim 1, characterized in that: It also includes replacing the RoIPooling layer of the improved Faster RCNN network with RoIAlign, wherein RoI Align cancels the quantization operation on the candidate box boundary and the boundary coordinates of each unit, uses a bilinear interpolation algorithm to calculate the boundary coordinates of each unit, and then performs a maximum pooling operation, and finally inputs the obtained feature matrix of the same size into the improved Fast RCNN network.
