Vehicle damage detection method and device, terminal, and storage medium

Abstract

The application provides a vehicle damage detection method and device, a terminal and a storage medium, and belongs to the technical field of vehicle detection. The vehicle damage detection method comprises the following steps: obtaining a target vehicle damage image to be detected; inputting the target vehicle damage image into a preset vehicle damage detection model to obtain a damage detection result of the target vehicle damage image; wherein the vehicle damage detection model is used to generate a classification feature map of the target vehicle damage image and determine a co-occurrence causal diagram of the target vehicle damage image, and output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal diagram. The application can effectively improve the accuracy of the vehicle damage detection result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of vehicle detection, and more particularly to a vehicle damage detection method and device, a terminal, and a storage medium. BACKGROUND

[0002] With the development of social economy, the quality of life of residents has been significantly improved, and the number of motor vehicles owned by residents is rising. However, road traffic accidents have become a prominent social problem. Whenever a vehicle collision occurs, the insurance company will conduct vehicle damage detection. At present, the insurance company generally uses manual recognition to detect vehicle damage. The vehicle owner or business personnel takes pictures of the vehicle damage, and the loss assessment personnel of the insurance company manually recognizes and determines the damage type and damage area of the damaged part of the vehicle in the image. The manual recognition method for vehicle damage detection has a large workload, low efficiency, and is not timely. Moreover, the vehicle damage detection result is not accurate enough due to personal factors of the loss assessment personnel. SUMMARY

[0003] The application aims to provide a vehicle damage detection method and device, a terminal, and a storage medium to solve the problem of inaccurate vehicle damage detection result in the prior art.

[0004] In a first aspect, the application provides a vehicle damage detection method, which includes:

[0005] obtaining a target vehicle damage image to be detected;

[0006] inputting the target vehicle damage image into a preset vehicle damage detection model to obtain a damage detection result of the target vehicle damage image;

[0007] The vehicle damage detection model is used to generate a classification feature map of the target vehicle damage image and determine a co-occurrence causal graph of the target vehicle damage image, and output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal graph.

[0008] In a possible implementation, the vehicle damage detection model includes a backbone network, a feature pyramid network, a region proposal network, a region of interest pooling unit, a co-occurrence causal unit, and a fully connected layer network.

[0009] The backbone network is used to extract feature information of the target vehicle damage image.

[0010] The feature pyramid network is used to fuse the feature information to obtain a multi-scale feature map.

[0011] The region proposal network is used to intercept a vehicle damage region in the feature map to generate a vehicle damage proposal box.

[0012] The region of interest pooling unit is configured to perform region pooling on the vehicle damage proposal box to obtain a base feature map of the same scale;

[0013] The co-occurrence causality unit is configured to determine a prior factor vector corresponding to the base feature map according to a predetermined co-occurrence causality graph and the base feature map;

[0014] The fully connected network is configured to classify the base feature map to obtain a classification feature map, perform feature fusion on the classification feature map and the prior factor vector to obtain an enhanced classification feature map, and detect the target vehicle damage image according to the enhanced classification feature map to output a damage detection result.

[0015] In a possible implementation, the co-occurrence causality graph includes probabilities of simultaneous occurrence of the same damaged type and probabilities of simultaneous occurrence of different damaged types on the same vehicle damage image; and the method for determining the co-occurrence causality graph includes: obtaining a historical damage image dataset of a target vehicle, wherein each vehicle damage image in the historical damage image dataset contains corresponding damage annotation information, and the damage annotation information includes a damaged region, a damaged component name, and a damaged type.

[0016] The damaged region, the damaged component name, and the damaged type corresponding to each vehicle damage image are counted to obtain the probabilities of simultaneous occurrence of the same damaged type and the probabilities of simultaneous occurrence of different damaged types on the same vehicle damage image.

[0017] In a possible implementation, the co-occurrence causality graph includes a corresponding relationship between a true value of the target vehicle damage image and an influence vector when the same damaged type occurs simultaneously, and a corresponding relationship between the true value of the target vehicle damage image and the influence vector when different damaged types occur simultaneously; and the method for determining the prior factor vector corresponding to the base feature map according to the predetermined co-occurrence causality graph and the base feature map includes:

[0018] The true value of the target vehicle damage image is determined based on the damage annotation information;

[0019] The prior factor vector corresponding to the base feature map is obtained based on the true value of the target vehicle damage image and the co-occurrence causality graph;

[0020] The method for determining the prior factor vector includes:

[0021] p=α×p1+β×p2(3);

[0022] Wherein, p is a prior factor vector, a is a first constant, β is a second constant, p1 is an influence vector determined based on the correspondence when the same damaged type appears simultaneously, and p2 is an influence vector determined based on the correspondence when different damaged types appear simultaneously.

[0023] In a possible implementation, the damage detection result of the target vehicle damage image is output based on the classification feature map and the co-occurrence causal graph, including:

[0024] The prior factor vector and the classification feature map are fused to obtain an enhanced classification feature map;

[0025] The fusion method is:

[0026] f p =(E+p)×f (4);

[0027] Wherein, f p is the enhanced classification feature map, E is a unit vector matrix, the dimension is consistent with the column vector dimension of the prior factor vector, p is the prior factor vector, and f is the classification feature map;

[0028] The target vehicle damage image is detected based on the enhanced classification feature map to obtain the damage detection result of the target vehicle damage image.

[0029] In a possible implementation, before the damage image dataset of the target vehicle history is acquired, the method further includes:

[0030] For each damaged type, whether the damaged type is a rare type is determined based on the number of damage areas corresponding to the damaged type and the number of vehicle damage images;

[0031] If the damaged type is a rare type, the dataset corresponding to the damaged type is expanded, and the expanded dataset is added to the damage image dataset;

[0032] If the damaged type is not a rare type, a preset number of vehicle damage images are selected as a basic dataset, the basic dataset is expanded, and the expanded dataset is added to the damage image dataset.

[0033] In a possible implementation, whether the damaged type is a rare type is determined based on the number of damage areas corresponding to the damaged type and the number of vehicle damage images, including:

[0034] For each damaged type, the number of damage areas corresponding to the damaged type is determined as a first number corresponding to the damaged type, and the number of vehicle damage images corresponding to the damaged type is determined as a second number corresponding to the damaged type;

[0035] if at least one of the first number and the second number is less than a preset threshold, determining that the damage type is a rare type;

[0036] if neither of the first number and the second number is less than the preset threshold, determining that the damage type is not a rare type.

[0037] In a second aspect, the embodiment of the present application provides a vehicle damage detection device, the device comprising:

[0038] an image acquisition module configured to acquire a target vehicle damage image to be detected;

[0039] an output module configured to input the target vehicle damage image into a preset vehicle damage detection model to obtain a damage detection result of the target vehicle damage image;

[0040] The vehicle damage detection model is configured to generate a classification feature map of the target vehicle damage image and determine a co-occurrence causal graph of the target vehicle damage image, and output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal graph.

[0041] In a third aspect, the embodiment of the present application provides a terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the vehicle damage detection method described above when executing the computer program.

[0042] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program implements the steps of the vehicle damage detection method described above when executed by a processor.

[0043] The vehicle damage detection method and device, the terminal, and the storage medium provided by the embodiment of the present application have the following beneficial effects:

[0044] In the embodiment of the present application, considering that the vehicle damage is detected by a damage assessor in the prior art, this method not only has a large workload and low efficiency, but also is prone to cause the vehicle damage detection result to be inaccurate due to the personal factors of the damage assessor. The vehicle damage detection model is constructed to realize the vehicle damage detection, which not only reduces the workload, but also improves the efficiency and detection accuracy. The classification feature map generated by the vehicle damage detection model is enhanced to fuse more vehicle damage features, and the co-occurrence causal graph is generated by analyzing the co-occurrence relationship between the feature information, the co-occurrence causal graph provides more prior knowledge for the vehicle damage detection model, the prior knowledge is fused with the classification feature map, and the target vehicle damage image is detected based on the fusion result, which further improves the accuracy of the vehicle damage detection. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0046] Figure 1 The flowchart of the vehicle damage detection method provided by the embodiments of the present application is shown in the figure.

[0047] Figure 2 The co-occurrence causal diagram of the vehicle damage detection model provided by the embodiments of the present application is shown in the figure.

[0048] Figure 3 The instantiation analysis diagram of the co-occurrence causal diagram provided by the embodiments of the present application is shown in the figure.

[0049] Figure 4 The model architecture diagram of the vehicle damage detection model provided by the embodiments of the present application is shown in the figure.

[0050] Figure 5 The architecture diagram of the prior factor vector determination method provided by the embodiments of the present application is shown in the figure.

[0051] Figure 6 The number distribution diagram of each damaged type before the damage image dataset is de-tailed by the embodiments of the present application.

[0052] Figure 7 The number distribution diagram of each damaged type after the damage image dataset is de-tailed by the embodiments of the present application.

[0053] Figure 8 The structure diagram of the vehicle damage detection device provided by the embodiments of the present application is shown in the figure.

[0054] Figure 9 The schematic diagram of the vehicle damage detection terminal provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0055] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary details.

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will be described by specific embodiments combined with the drawings.

[0057] Please refer to Figure 1 , Figure 1 The flowchart of the vehicle damage detection method provided in the embodiments of the present application is shown in the figure, and the method comprises:

[0058] S101: Obtain a target vehicle damage image to be detected.

[0059] In the embodiments of the present application, by detecting the vehicle damage, the accident liability party and the accident claim party can be quickly determined by the vehicle insurance, and the damage personnel can execute the corresponding claim scheme according to the vehicle damage detection result. The target vehicle damage image is the damage image of the target vehicle that needs to be detected at present. The target vehicle damage image can be multiple vehicle damage images taken from multiple directions and preprocessed. In general, the damage personnel will collect the vehicle damage image from the accident scene after the traffic accident case occurs. In order to collect the vehicle damage image from the accident scene in all directions, the shooting terminal can be adjusted from different shooting angles and different shooting distances, and the multiple vehicle damage images taken from multiple directions can be preprocessed, such as luminosity adjustment, pixel adjustment, etc., so that the target vehicle damage image obtained is more in line with the actual collection requirements.

[0060] Among them, the damage detection needs to detect the damaged area, the damaged type and the damaged component name. The damaged type can be scratches, scratches, mild deformation, medium deformation, severe deformation, glass crack

[0061] (Glass_crack), glass breakage, tearing, missing, dislocation, etc. The damaged component name can be bumper, hood, door, A-pillar, leaf plate, mesh, rearview mirror, headlight, fog lamp, tail lamp, etc.

[0062] S102: Input the target vehicle damage image into a preset vehicle damage detection model to obtain a damage detection result of the target vehicle damage image.

[0063] Among them, the vehicle damage detection model is used to generate a classification feature map of the target vehicle damage image and determine a co-occurrence causal diagram of the target vehicle damage image, and output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal diagram.

[0064] In the embodiments of the present application, the co-occurrence causal diagram is used to describe the probability of the same damaged type appearing at the same time and the probability of different damaged types appearing at the same time. Referring toFigure 2 The co-occurrence causal graph is shown. It can be seen from the co-occurrence causal graph that the probability of the simultaneous occurrence of misalignment and scratches is the highest in the combination of the two types of damage, and the scratches basically occur with the scratches, and the probability of misalignment and scratches also greatly increases when the slight deformation, moderate deformation and severe deformation occur, which is consistent with the vehicle damage at the vehicle accident scene. The image analysis of the simultaneous occurrence of slight deformation, misalignment and scratches is shown in Figure 3 It can be seen from the instantiation analysis graph of the co-occurrence causal graph that when misalignment occurs, the probability of scratches is 41%; when scratches occur, the probability of misalignment is 17%; when slight deformation occurs, the probability of misalignment is 29%; and when slight deformation occurs, the probability of scratches is 34%.

[0065] The co-occurrence causal graph in the present application is extended on the basis of the co-occurrence of different damage types in the original, and the co-occurrence causal relationship between the original different damage types is retained, and the co-occurrence causal relationship between the same damage types is further extended. Therefore, introducing the co-occurrence causal graph as prior knowledge into the vehicle damage detection model can improve the detection effect to a certain extent.

[0066] In the embodiment of the present application, considering that in the prior art, the damage detection of the vehicle is performed by the damage determination personnel, this method not only has a large workload and low efficiency, but also is prone to cause the vehicle damage detection result to be not accurate enough due to the personal factors of the damage determination personnel. The vehicle damage detection model is constructed in the present application to realize the damage detection of the vehicle, which can not only reduce the workload, but also improve the efficiency and detection accuracy. The classification feature map generated by the vehicle damage detection model is subjected to feature enhancement, more vehicle damage features can be fused, and the co-occurrence causal graph is generated by analyzing the co-occurrence relationship between the feature information. The co-occurrence causal graph provides more prior knowledge for the vehicle damage detection model. The prior knowledge is fused with the classification feature map, and the target vehicle damage image is detected based on the fusion result, which further improves the accuracy of the vehicle damage detection.

[0067] In a possible implementation manner, the vehicle damage detection model comprises: a backbone network, a feature pyramid network, a region proposal network, a region of interest pooling unit, a co-occurrence causal unit and a fully connected layer network.

[0068] The backbone network is used to extract feature information of the target vehicle damage image.

[0069] The feature pyramid network is used to fuse the feature information to obtain a multi-scale feature map.

[0070] The region proposal network is used to intercept a vehicle damage region in the feature map to generate a vehicle damage proposal box.

[0071] The region of interest pooling unit is configured to perform region pooling on the vehicle damage proposal frame to obtain a basic feature map of the same scale.

[0072] The co-occurrence causality unit is configured to determine a prior factor vector corresponding to the basic feature map according to a predetermined co-occurrence causality graph and the basic feature map.

[0073] The fully connected network is configured to classify the basic feature map to obtain a classification feature map. Feature fusion is performed on the classification feature map and the prior factor vector to obtain an enhanced classification feature map. The target vehicle damage image is detected according to the enhanced classification feature map to output a damage detection result.

[0074] In the embodiments of the present application, the backbone network can be a network for extracting image feature information, such as VGG (Visual Geometry Group, super-resolution test sequence), ResNet (Residual Network), ResNet50 (Residual Network 50), DenseNet (Dense Convolutional Network), etc.

[0075] In the embodiments of the present application, the vehicle damage detection model adopts a model architecture of the Mask R-CNN image recognition method, as shown in FIG. 1. Figure 4A model architecture diagram of the vehicle damage detection model is shown, wherein a residual network (Residual Network 50, ResNet50) and a feature pyramid network (Feature Pyramid Network, FPN) are backbone networks. After a target vehicle damage image is input, the convolutional layers in the backbone network are used as an image feature extractor to extract feature information of the image at different levels, and the compressed feature image is used for feature fusion in the feature pyramid network to obtain a multi-scale feature map. Then, a region proposal network (Region Proposal Network, RPN) obtains a region possibly containing damage according to the feature map, generates a vehicle damage proposal box, the vehicle damage proposal box intercepts the vehicle damage region in the feature map, and adjusts the feature map to a uniform size through a region of interest alignment unit (Region Of Interest Align, ROI Align) to obtain a basic feature map. The basic feature map is used to determine a prior factor vector corresponding to the basic feature map according to a pre-determined co-occurrence causal graph. The co-occurrence post-processing module (Co-occurrence Post-Processing Module, CPPM) is used to obtain the prior factor vector. Finally, the basic feature map is input into a fully connected network for classification processing to obtain a classification feature map. The target vehicle damage image is detected according to the classification feature map and the co-occurrence causal graph, and a damage detection result is output. The vehicle damage detection model has the characteristics of integrating damage detection and instance segmentation, and has high recognition rate and high accuracy.

[0076] In a possible implementation, the co-occurrence causal graph includes probabilities of the same damage type appearing simultaneously and probabilities of different damage types appearing simultaneously on the same vehicle damage image. A method for determining the co-occurrence causal graph is as follows: obtaining a damage image dataset of a target vehicle history, wherein each vehicle damage image in the damage image dataset of the history contains corresponding damage annotation information, and the damage annotation information includes a damaged area, a damaged component name, and a damage type.

[0077] The damaged area, the damaged component name, and the damage type corresponding to each vehicle damage image are counted to obtain the probabilities of the same damage type appearing simultaneously and the probabilities of different damage types appearing simultaneously on the same vehicle damage image.

[0078] In the embodiments of the present application, the damage image of the vehicle can be obtained by shooting from the accident scene in life, or can be obtained through the Internet, news reports and other ways. According to the obtained damage image, a damage image data set is established. In the present application, 10 types of vehicle damage type images are selected as experimental objects, and the construction method of COCO2017 data set is referred to, and a damage image data set is constructed, wherein the sample images of the training set and the test set are 25901 and 8634 respectively, the quantity ratio is 3:1, a total of 69654 vehicle damage target numbers are contained, and the data set is not disclosed at present. The vehicle damage detection model constructed in the present application is trained and tested by using NVIDIA3090 professional acceleration card. The operating system used is Ubuntu16.04.6LTS, and CUDA10.0 is used to accelerate the training. The computer language used is Python3.8, and the network development framework is Pytorch. In the training stage, the batchsize is set to 2, the stochastic gradient descent (SGD) algorithm is used, the initial learning rate is set to 0.001, the learning rate is reduced to 10% of the original after every 9 epochs, and the maximum iteration training is 30 epochs. By using GPUs, this method can be easily extended to distributed systems.

[0079] The vehicle damage image is imported in the Labelme software tool, and the damage is labeled. Because the shape of the damage is variable, the simple label such as circle and rectangle is easy to contain more normal background, which is not conducive to obtaining the characteristics of the vehicle damage, therefore, the damaged area is labeled by using polygon, and the damaged component name and damage type are marked, and the damage labeling information is obtained.

[0080] In the embodiments of the present application, considering that there is a close causal relationship between many damaged types in actual vehicle damage detection, which is the same as the multiple damages of the vehicle in the vehicle accident, for example, the missing of the vehicle component is often accompanied by the appearance of vehicle scratches and scratches, and for example, the probability of glass damage is greatly improved in the case of severe deformation. Therefore, the co-occurrence causal graph is introduced in the present application, and the correlation between the damaged types is defined by deeply mining the internal relationship in the vehicle damage image data set, and the co-occurrence causal probability between the damaged types is expressed by the conditional probability.

[0081] In the embodiments of the present application, the probability determination method of the same damaged type appearing at the same time on the same vehicle damage image is as follows:

[0082]

[0083] Wherein, P(C y |C x ) is the probability of the occurrence of damaged type y when the damaged type x occurs in the same vehicle damage image, P(C y ,C xP(C x ) is the probability of occurrence of damaged type x.

[0084] The probability determination method of different damaged types appearing simultaneously on the same vehicle damage image is:

[0085]

[0086] P(C x2 |C x1 ) is the probability of occurrence of the same damaged type, C x1 and C x2 are different damaged areas of the same damaged type in a vehicle damage image, N(C x2 , C x1 ) is the number of images in which multiple damaged areas of the same damaged type appear in a vehicle damage image, and N(C x ) is the number of images in which damaged type x appears.

[0087] In the embodiments of the present application, the present application considers that there is also a causal relationship between damaged components, so the present application can also calculate a co-occurrence causal graph of damaged component names, which includes the probability of the same damaged component appearing simultaneously and the probability of different damaged components appearing simultaneously on the same vehicle damage image.

[0088] In a possible implementation, the co-occurrence causal graph includes a correspondence relationship between the true value of the target vehicle damage image and the influence vector when the same damaged type appears simultaneously, and a correspondence relationship between the true value of the target vehicle damage image and the influence vector when different damaged types appear simultaneously. The prior factor vector corresponding to the basic feature graph is determined according to the predetermined co-occurrence causal graph and the basic feature graph, and includes:

[0089] The true value of the target vehicle damage image is determined based on the damage annotation information.

[0090] The prior factor vector corresponding to the basic feature graph is obtained based on the true value of the target vehicle damage image and the co-occurrence causal graph.

[0091] The determination method of the prior factor vector is:

[0092] p = a x p1 + b x p2 (3).

[0093] Wherein, p is the prior factor vector, a is the first constant, b is the second constant, p1 is the influence vector when the same damaged type appears simultaneously based on the correspondence relationship, and p2 is the influence vector when different damaged types appear simultaneously based on the correspondence relationship.

[0094] In this embodiment of the application, the true value of the target vehicle damage image is the damage type marked on a vehicle damage image. It can be one of the above damage types, or multiple damage types that appear simultaneously on a target vehicle damage image. Therefore, the true value of the target vehicle damage image is not fixed.

[0095] In the embodiments of this application, see Figure 5 The schematic diagram of the prior factor vector determination method shows that the co-occurrence causal relationship between damage types is updated by mapping the current damage type through constants α and β, resulting in the prior factor vector p. The first constant α and the second constant β can be experimentally measured or adaptively learned. Currently, the prior factor vector obtained when α = 0.1 and β = 0.2 is more accurate. The first constant α regulates the influence of the simultaneous occurrence of the same damage type on the damage type in the target vehicle damage image, while the second constant β regulates the influence of the simultaneous occurrence of different damage types on the damage type in the target vehicle damage image. p1 and p2 are obtained by mapping the ground truth of the target vehicle damage image from the co-occurrence causal graph. Based on p1 and p2, the prior factor vector can be determined. Through the regulation of the first constant α and the second constant β, the prior factor vector p becomes more generalizable and better represents the relationship between the simultaneous occurrence of the same damage type and different damage types and the damage type in the target vehicle damage image.

[0096] In one possible implementation, the damage detection results of the target vehicle damage image are output based on the classification feature map and the co-occurrence causal map, including:

[0097] The prior factor vector and the classification feature map are fused to obtain the enhanced classification feature map.

[0098] The fusion method is as follows:

[0099] f p = (E+p)×f(4).

[0100] Where f p To enhance the classification feature map, E is a unit vector matrix with the same dimension as the column vector dimension of the prior factor vector, p is the prior factor vector, and f is the classification feature map.

[0101] Damage detection results for target vehicle images are obtained by detecting damage based on enhanced classification feature maps.

[0102] In this embodiment of the application, the classification feature map f is enhanced. p The loss is calculated by comparing the ground truth y of the target vehicle damage image with the actual loss value. Finally, backpropagation is used to update the weight parameters of the network. The loss value is calculated as follows:

[0103]

[0104] wherein L oss (f p , y) is a loss value, f p is an enhanced classification feature map.

[0105] The present application uses the evaluation index commonly used in the current target detection model, bbox_mAP 50 and segm_mAP 50 to evaluate the overall model, mAP 50 The precision index is defined as:

[0106]

[0107] wherein C is the number of damaged types of vehicle damage, AR n 50 represents the recall rate when the IoU of the damaged type n is equal to 0.5, AP n 50 (AR n 50 ) is the AR n 50 corresponding to the damaged type n when the recall rate of the damaged type n is AR n 50 precision. bbox_mAP 50 is used to quantitatively evaluate the training effect of the model on the regression positioning frame, segm_mAP 50 can evaluate the training effect of the model on the pixel-level mask matching, and the combination of the two reflects the performance of the trained model at the same time.

[0108] In a possible implementation, before obtaining the damage image dataset of the target vehicle history, the method further includes:

[0109] For each damaged type, it is determined whether the damaged type is a rare type based on the number of damage regions corresponding to the damaged type and the number of vehicle damage images.

[0110] If the damaged type is a rare type, the dataset corresponding to the damaged type is expanded, and the expanded dataset is added to the damage image dataset.

[0111] If the damaged type is not a rare type, a preset number of vehicle damage images are selected as a basic dataset, the basic dataset is expanded, and the expanded dataset is added to the damage image dataset.

[0112] In the embodiments of the present application, the implementer can set the preset number according to actual needs.

[0113] The application considers that the obtained vehicle damage image data is generally in a long-tailed distribution, and therefore uses a rare type extraction and integration method to alleviate the long-tailed distribution problem. If a certain damaged type is judged to be a rare type, the data set corresponding to the damaged type needs to be expanded to increase the number of vehicle damage images corresponding to the damaged type. If a certain damaged type is not a rare type, the number of vehicle damage images corresponding to the damaged type may need to be reduced to make the number of vehicle damage images corresponding to each damaged type in the overall vehicle damage image data set more balanced. Reducing the number of vehicle damage images corresponding to the damaged type can reduce the workload and improve efficiency.

[0114] By comparing Figure 6 the number distribution diagram of each damaged type before the damage image data set is de-tailed, and Figure 7 the number distribution diagram of each damaged type after the damage image data set is de-tailed, it can be seen that after de-tailing, the number of damaged types such as misalignment, glass breakage, glass crack and scratch increases significantly, and the number distribution of each damaged type in the data set is more balanced.

[0115] In a possible implementation, whether the damaged type is a rare type is determined based on the number of damage areas corresponding to the damaged type and the number of vehicle damage images, and includes:

[0116] For each damaged type, the number of damage areas corresponding to the damaged type is determined as a first number corresponding to the damaged type, and the number of vehicle damage images corresponding to the damaged type is determined as a second number corresponding to the damaged type.

[0117] If at least one of the first number and the second number is less than a preset threshold, the damaged type is determined to be a rare type.

[0118] If neither the first number nor the second number is less than the preset threshold, the damaged type is determined to be not a rare type.

[0119] In the embodiments of the application, the preset threshold can be set by the implementer according to actual needs.

[0120] In the embodiments of the application, the case of being determined to be a rare type includes: the number of damage areas is less than the preset threshold, in which case the damaged type is determined to be a rare type regardless of whether the number of vehicle damage images is less than the preset threshold. The number of vehicle damage images is less than the preset threshold, in which case the damaged type is determined to be a rare type regardless of whether the number of damage areas is less than the preset threshold. The number of damage areas and the number of vehicle damage images are both less than the preset threshold, in which case the damaged type is determined to be a rare type. In the embodiments of the application, the case of being determined to be not a rare type includes: neither the number of damage areas nor the number of vehicle damage images is less than the preset threshold.

[0121] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0122] The following is the device embodiment of the present application. For details not described in detail, please refer to the corresponding method embodiments described above.

[0123] Figure 8 The structure diagram of the vehicle damage detection device provided by the embodiments of the present application is shown. For the convenience of description, only the part related to the embodiments of the present application is shown, and the details are as follows:

[0124] As shown in Figure 8 The vehicle damage detection device 80 includes an image acquisition module 81 and an output module 82.

[0125] The image acquisition module 81 is configured to acquire a target vehicle damage image to be detected.

[0126] The output module 82 is configured to input the target vehicle damage image into a preset vehicle damage detection model to obtain a damage detection result of the target vehicle damage image.

[0127] The vehicle damage detection model is configured to generate a classification feature map of the target vehicle damage image and determine a co-occurrence causal graph of the target vehicle damage image, and output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal graph.

[0128] In one possible implementation, the vehicle damage detection model in the output module 82 includes a backbone network, a feature pyramid network, a region proposal network, a region of interest pooling unit, a co-occurrence causal unit, and a fully connected layer network.

[0129] The backbone network is configured to extract feature information of the target vehicle damage image.

[0130] The feature pyramid network is configured to fuse the feature information to obtain a multi-scale feature map.

[0131] The region proposal network is configured to intercept a vehicle damage region in the feature map to generate a vehicle damage proposal box.

[0132] The region of interest pooling unit is configured to perform region pooling on the vehicle damage proposal box to obtain a basic feature map of the same scale.

[0133] The co-occurrence causal unit is configured to determine a prior factor vector corresponding to the basic feature map according to a predetermined co-occurrence causal graph and the basic feature map.

[0134] The full connection network is used for classifying the basic feature map to obtain a classification feature map. The classification feature map and the prior factor vector are fused to obtain an enhanced classification feature map. The target vehicle damage image is detected according to the enhanced classification feature map, and a damage detection result is output.

[0135] In a possible implementation, the co-occurrence causal graph includes probabilities of the same damage type appearing simultaneously and probabilities of different damage types appearing simultaneously on the same vehicle damage image. The output module 82 is specifically configured to determine the co-occurrence causal graph. The method for determining the co-occurrence causal graph is as follows: obtaining a damage image data set of the target vehicle history, wherein each historical vehicle damage image in the historical damage image data set contains corresponding damage annotation information, and the damage annotation information includes a damaged area, a damaged component name, and a damage type.

[0136] The damaged area, the damaged component name, and the damage type corresponding to each vehicle damage image are counted to obtain probabilities of the same damage type appearing simultaneously and probabilities of different damage types appearing simultaneously on the same vehicle damage image.

[0137] In a possible implementation, the co-occurrence causal graph includes: a corresponding relationship between a true value of the target vehicle damage image and an influence vector when the same damage type appears simultaneously, and a corresponding relationship between the true value of the target vehicle damage image and the influence vector when different damage types appear simultaneously. The output module 82 is specifically configured to determine the prior factor vector corresponding to the basic feature map according to the predetermined co-occurrence causal graph and the basic feature map.

[0138] Determining the prior factor vector corresponding to the basic feature map according to the predetermined co-occurrence causal graph and the basic feature map includes:

[0139] Determining the true value of the target vehicle damage image based on the damage annotation information.

[0140] Obtaining the prior factor vector corresponding to the basic feature map based on the true value of the target vehicle damage image and the co-occurrence causal graph.

[0141] The method for determining the prior factor vector is as follows:

[0142] p = a x p1 + b x p2 (3).

[0143] Wherein, p is the prior factor vector, a is the first constant, b is the second constant, p1 is the influence vector when the same damage type appears simultaneously based on the corresponding relationship, and p2 is the influence vector when different damage types appear simultaneously based on the corresponding relationship.

[0144] In a possible implementation, the output module 82 is specifically configured to output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal graph.

[0145] Output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal diagram, including:

[0146] Fuse the prior factor vector and the classification feature map to obtain an enhanced classification feature map.

[0147] The fusion method is:

[0148] f p =(E+p)xf(4).

[0149] Where f p is the enhanced classification feature map, E is a unit vector matrix, the dimension is consistent with the column vector dimension of the prior factor vector, p is the prior factor vector, and f is the classification feature map.

[0150] Detect the target vehicle damage image based on the enhanced classification feature map to obtain the damage detection result of the target vehicle damage image.

[0151] In a possible implementation, the output module 82 is specifically configured to obtain the damage image data set of the target vehicle history before.

[0152] Before obtaining the damage image data set of the target vehicle history, the method further includes:

[0153] For each damaged type, determine whether the damaged type is a rare type based on the number of damage areas corresponding to the damaged type and the number of vehicle damage images.

[0154] If the damaged type is a rare type, expand the data set corresponding to the damaged type, and add the expanded data set to the damage image data set.

[0155] If the damaged type is not a rare type, select a preset number of vehicle damage images as a basic data set, expand the basic data set, and add the expanded data set to the damage image data set.

[0156] In a possible implementation, the output module 82 is specifically configured to determine whether the damaged type is a rare type based on the number of damage areas corresponding to the damaged type and the number of vehicle damage images.

[0157] Determine whether the damaged type is a rare type based on the number of damage areas corresponding to the damaged type and the number of vehicle damage images, including:

[0158] For each damaged type, determine the number of damage areas corresponding to the damaged type as a first number corresponding to the damaged type, and determine the number of vehicle damage images corresponding to the damaged type as a second number corresponding to the damaged type.

[0159] If at least one of the first quantity and the second quantity is less than a preset threshold, the damaged type is determined to be a rare type.

[0160] If the first quantity and the second quantity are both not less than the preset threshold, then the damaged type is determined not to be a rare type.

[0161] Figure 9 This is a schematic diagram of the vehicle damage detection terminal provided in an embodiment of this application. Figure 9 As shown, the terminal 900 in this embodiment includes a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, it implements the steps described in the various vehicle damage detection method embodiments above, for example... Figure 1 Steps 101 to 102 are shown. Alternatively, when processor 901 executes computer program 903, it implements the functions of each module in the above-described device embodiments, for example... Figure 8 The functions of modules 81 to 82 are shown.

[0162] For example, computer program 903 can be divided into one or more modules, one or more modules are stored in memory 902 and executed by processor 901 to complete this application. The one or more modules can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 903 in terminal 900. For example, computer program 903 can be divided into... Figure 8 Modules 81 to 82 are shown.

[0163] Terminal 900 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal 900 may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will understand that... Figure 9 This is merely an example of terminal 900 and does not constitute a limitation on terminal 900. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal may also include input / output devices, network access devices, buses, etc.

[0164] The processor 901 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0165] The memory 902 can be an internal storage unit of the terminal 900, for example, a hard disk or a memory of the terminal 900. The memory 902 can also be an external storage device of the terminal 900, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal 900. Further, the memory 902 can include both the internal storage unit and the external storage device of the terminal 900. The memory 902 is used to store computer programs and other programs and data required by the terminal. The memory 902 can also be used to temporarily store data that has been output or will be output.

[0166] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0167] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0168] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0169] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented by other ways. For example, the apparatus / terminal embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0170] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0171] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0172] The integrated module, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiments can also be implemented by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each vehicle damage detection method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0173] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting vehicle damage, characterized in that, The damage detection method includes: Acquire images of damage to the target vehicle to be detected; The target vehicle damage image is input into a preset vehicle damage detection model to obtain the damage detection result of the target vehicle damage image. The vehicle damage detection model is used to generate a classification feature map of the target vehicle damage image and determine the co-occurrence causal map of the target vehicle damage image, and output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal map; the co-occurrence causal map includes the probability of the same damage type occurring simultaneously and the probability of different damage types occurring simultaneously on the same vehicle damage image.

2. The vehicle damage detection method as described in claim 1, characterized in that, The vehicle damage detection model includes: a backbone network, a feature pyramid network, a region proposal network, a region of interest pooling unit, a co-occurrence causal unit, and a fully connected layer network; The backbone network is used to extract feature information from the target vehicle damage image. The feature pyramid network is used to fuse the feature information to obtain a multi-scale feature map; The region proposal network is used to extract vehicle damage regions from the feature map and generate vehicle damage proposal boxes. The region of interest pooling unit is used to perform region pooling on the vehicle damage proposal box to obtain a basic feature map of the same scale. The co-occurrence causal unit is used to determine the prior factor vector corresponding to the basic feature map based on the pre-determined co-occurrence causal map and the basic feature map; The fully connected layer network is used to classify the basic feature map to obtain a classification feature map; the classification feature map and the prior factor vector are fused to obtain an enhanced classification feature map; the target vehicle damage image is detected based on the enhanced classification feature map, and the damage detection result is output.

3. The vehicle damage detection method as described in claim 1, characterized in that, The method for determining the co-occurrence causal graph is as follows: obtain a historical damage image dataset of the target vehicle, wherein each vehicle damage image in the historical damage image dataset contains corresponding damage annotation information, the damage annotation information including: damaged area, damaged component name and damage type; By statistically analyzing the damaged area, damaged component name, and damage type corresponding to each vehicle damage image, the probability of the same damage type appearing simultaneously and the probability of different damage types appearing simultaneously on the same vehicle damage image are obtained.

4. The vehicle damage detection method as described in claim 3, characterized in that, The co-occurrence causal graph includes: the correspondence between the ground truth of the target vehicle damage image and the influence vector when the same damage type occurs simultaneously, and the correspondence between the ground truth of the target vehicle damage image and the influence vector when different damage types occur simultaneously; determining the prior factor vector corresponding to the basic feature map based on the predetermined co-occurrence causal graph and the basic feature map includes: The ground truth of the target vehicle damage image is determined based on the damage annotation information; Based on the ground truth of the target vehicle damage image and the co-occurrence causal graph, the prior factor vector corresponding to the basic feature map is obtained. The method for determining the prior factor vector is as follows: （3）； in, For the prior factor vector, It is the first constant. It is the second constant. It is the influence vector when the same damage type occurs simultaneously, determined based on the correspondence. It is the influence vector when different damage types occur simultaneously, determined based on the correspondence.

5. The vehicle damage detection method as described in claim 4, characterized in that, Based on the classification feature map and co-occurrence causal map, the damage detection results of the target vehicle damage image are output, including: The prior factor vector and the classification feature map are fused to obtain an enhanced classification feature map; The fusion method is as follows: （4）； in To enhance the classification feature map, It is a unit vector matrix, with the same dimensions as the column vectors of the prior factor vectors. For the prior factor vector, For classification feature maps; The damage detection results of the target vehicle image are obtained by detecting the damage based on the enhanced classification feature map.

6. The vehicle damage detection method as described in claim 3, characterized in that, Before acquiring the historical damage image dataset of the target vehicle, the following steps are also included: For each type of damage, determine whether the damage type is a rare type based on the number of damaged areas corresponding to that type and the number of vehicle damage images; If the damage type is rare, the dataset corresponding to the damage type is expanded, and the expanded dataset is added to the damaged image dataset. If the damage type is not rare, a preset number of vehicle damage images are selected as the basic dataset, the basic dataset is expanded, and the expanded dataset is added to the damage image dataset.

7. The vehicle damage detection method as described in claim 6, characterized in that, The determination of whether a damage type is rare based on the number of damaged areas corresponding to that damage type and the number of vehicle damage images includes: For each type of damage, the number of damaged areas corresponding to that type of damage is determined as the first number corresponding to that type of damage, and the number of vehicle damage images corresponding to that type of damage is determined as the second number corresponding to that type of damage. If at least one of the first quantity and the second quantity is less than a preset threshold, then the damage type is determined to be a rare type; If neither the first quantity nor the second quantity is less than a preset threshold, then the damaged type is determined not to be a rare type.

8. A vehicle damage detection device, characterized in that, The device includes: The image acquisition module is used to acquire images of damage to the target vehicle to be detected. The output module is used to input the target vehicle damage image into a preset vehicle damage detection model to obtain the damage detection result of the target vehicle damage image. The vehicle damage detection model is used to generate a classification feature map of the target vehicle damage image and determine the co-occurrence causal map of the target vehicle damage image, and output the damage detection result of the target vehicle damage image based on the classification feature map and the co-occurrence causal map; the co-occurrence causal map includes the probability of the same damage type occurring simultaneously and the probability of different damage types occurring simultaneously on the same vehicle damage image.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7 above.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images