A Multimodal Vehicle Damage Detection Method and System Based on Damage Activation Maps

Abstract

This application provides a multimodal vehicle damage detection method and system based on damage activation maps. The method includes: acquiring an image of a vehicle to be detected and pre-determined damage category information; generating a damage activation map based on the damage category information and the image of the vehicle to be detected; generating a vehicle feature map based on the image of the vehicle to be detected using a backbone network model; performing image fusion based on the vehicle feature map and the damage activation map to determine an enhanced feature map; and performing vehicle damage detection on the enhanced feature map to determine the detection result of the vehicle damage. Compared with existing vehicle damage detection based on single-modal detection frameworks, this application uses a multimodal damage activation mechanism to generate a high-purity damage activation map and combines it with a dual-branch fusion module for image fusion to obtain an effective enhanced feature map, which can improve the quality of the enhanced feature map; and performing vehicle damage detection using the enhanced feature map can improve the accuracy and efficiency of vehicle damage detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a multimodal vehicle damage detection method and system based on damage activation maps. Background Technology

[0002] With the surge in car ownership and frequent traffic accidents, the insurance industry's demand for automated and precise vehicle damage detection is becoming increasingly urgent. Traditional manual damage assessment suffers from low efficiency, high subjectivity, and high costs, necessitating intelligent solutions. Against this backdrop, automated vehicle damage detection has become a key technology for intelligent damage assessment, capable of automatically identifying vehicle defects and accurately locating damaged areas. In practical use, users simply need to take photos of the damaged vehicle using a camera or smartphone; these images are fed into the automated vehicle damage detection system, which automatically identifies and determines the defects, then transmits the damage information to the insurance claims system to generate an assessment report instantly.

[0003] Despite the significant practical need, research on automated vehicle damage detection remains relatively scarce. Recent work has primarily relied on single-modal visual detection (such as Mask R-CNN), but due to the complexity of vehicle damage types, large scale differences, and numerous background interferences, it is prone to missed detections and false detections (such as misidentifying reflections as dents). Summary of the Invention

[0004] This application aims to provide a multimodal vehicle damage detection method and system based on damage activation maps, which can improve the accuracy of vehicle damage detection.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a multimodal vehicle damage detection method based on damage activation maps, the method comprising:

[0007] Acquire images of the vehicle to be inspected and pre-determined damage category information;

[0008] Based on the damage category information and the image of the vehicle to be detected, a damage activation map is generated; and based on the image of the vehicle to be detected, a vehicle feature map is generated through a backbone network model.

[0009] Based on the vehicle feature map and the damage activation map, image fusion is performed to determine the enhanced feature map;

[0010] Vehicle damage detection is performed on the enhanced feature map to determine the detection results of vehicle damage.

[0011] In the above scheme, generating a damage activation map based on the damage category information and the image of the vehicle to be detected includes:

[0012] Based on the damage category information, determine the text information;

[0013] Based on the text information and the preset redundant prompt text information, encoding processing is performed to determine multiple types of damage information vectors and redundant prompt text vectors;

[0014] Based on the image of the vehicle to be detected, the multi-type damage information vectors, and the redundant prompt text vectors, a damage activation map and a mapping map are generated.

[0015] The damage activation map is determined based on the damage activation map and the mapping map.

[0016] In the above scheme, generating a damage activation map and a mapping map based on the image of the vehicle to be detected, the multi-type damage information vectors, and the redundant prompt text vectors includes:

[0017] Based on the image of the vehicle to be detected, normalization processing and self-attention mechanism processing are performed to obtain an image feature map; wherein, the self-attention mechanism is a value-to-value self-attention mechanism.

[0018] The damage activation map is generated based on the multi-type damage information vectors, the image feature map, and the redundant prompt text vectors;

[0019] The mapping map is generated based on the multi-type damage information vectors and the image feature map.

[0020] In the above scheme, the step of performing image fusion based on the vehicle feature map and the damage activation map to determine the enhanced feature map includes:

[0021] The first feature map is obtained by element-wise multiplication of the vehicle feature map and the damage activation map;

[0022] The vehicle feature map is convolved to obtain a second feature map;

[0023] Based on preset learning parameters, the first feature map and the second feature map are fused to obtain the enhanced feature map.

[0024] In the above scheme, the step of performing convolution processing on the vehicle feature map to obtain the second feature map includes:

[0025] The vehicle feature map is subjected to three convolution processes in sequence to obtain a first initial feature map, a second initial feature map, and a third initial feature map.

[0026] Based on the first initial feature map and the second initial feature map, a weight map is determined;

[0027] The second feature map is obtained by multiplying the weight map and the third initial feature map by a dot product.

[0028] In the above scheme, determining the weight map based on the first initial feature map and the second initial feature map includes:

[0029] The first and second initial feature maps are subjected to feature deformation to obtain the fourth and fifth initial feature maps;

[0030] Multiply the fourth initial feature map and the fifth initial feature map to obtain the sixth initial feature map;

[0031] The sixth initial feature map is subjected to average pooling to obtain the weight map.

[0032] In the above scheme, the step of performing vehicle damage detection on the enhanced feature map and determining the detection result of vehicle damage includes:

[0033] Vehicle damage detection is performed on the enhanced feature map using a preset detection model to obtain vehicle damage classification results, location information, and damage segmentation results. The preset detection model includes four detection heads: a classification head, a bounding box regression head, a masking head, and a location quality head.

[0034] The detection result of the vehicle damage is determined based on the classification result of the vehicle damage, the location information, and the damage segmentation result.

[0035] Secondly, embodiments of this application provide a multimodal vehicle damage detection system based on damage activation maps. The system includes: an acquisition module, a generation module, and a detection module, wherein...

[0036] The acquisition module is used to acquire images of the vehicle to be detected and pre-determined damage category information;

[0037] The generation module is used to generate a damage activation map based on the damage category information and the image of the vehicle to be detected; and to generate a vehicle feature map based on the image of the vehicle to be detected through a backbone network model.

[0038] The detection module is used to perform image fusion based on the vehicle feature map and the damage activation map to determine an enhanced feature map; and to perform vehicle damage detection on the enhanced feature map to determine the detection result of vehicle damage.

[0039] Thirdly, embodiments of this application provide a multimodal vehicle damage detection device based on damage activation maps, comprising: a processor and a memory; wherein,

[0040] The memory is used to store computer programs;

[0041] The processor is configured to call and run the computer program from the memory to perform the method as described in the first aspect.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions for causing a processor to perform the method described in the first aspect.

[0043] This application provides a multimodal vehicle damage detection method and system based on damage activation maps. The method includes: acquiring an image of a vehicle to be detected and pre-determined damage category information; generating a damage activation map based on the damage category information and the image of the vehicle to be detected; generating a vehicle feature map based on the image of the vehicle to be detected using a backbone network model; performing image fusion based on the vehicle feature map and the damage activation map to determine an enhanced feature map; and performing vehicle damage detection on the enhanced feature map to determine the detection result of vehicle damage. In the above scheme, acquiring the image of the vehicle to be detected and pre-determined damage category information, using a multimodal damage activation mechanism to generate a high-purity damage activation map through a semantic subtraction prompting strategy, generating a vehicle feature map based on the image of the vehicle to be detected using a backbone network model, performing image fusion by combining the vehicle feature map and the damage activation map with a dual-branch fusion module to obtain an effective enhanced feature map, which can improve the quality of the enhanced feature map; and performing vehicle damage detection using the enhanced feature map can improve the accuracy and efficiency of vehicle damage detection. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0045] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0046] Figure 1 A schematic diagram of an automated insurance claims system provided in this application embodiment;

[0047] Figure 2 This is an optional flowchart illustrating a multimodal vehicle damage detection method based on damage activation maps provided in an embodiment of this application.

[0048] Figure 3 A comparative schematic diagram of two vehicle damage detection processes provided in the embodiments of this application;

[0049] Figure 4 A schematic diagram illustrating the generation of a damage activation map in a multimodal vehicle damage detection method based on damage activation maps, provided in an embodiment of this application.

[0050] Figure 5 A schematic diagram of a damage activation map fusion module for a multimodal vehicle damage detection method based on damage activation maps provided in this application embodiment;

[0051] Figure 6 This application provides a schematic diagram of the structure of a multimodal vehicle damage detection system based on damage activation maps, as shown in the embodiments of this application.

[0052] Figure 7 This is a schematic diagram of the structure of a multimodal vehicle damage detection device based on damage activation maps, provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0056] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0057] By integrating automated vehicle damage detection systems into automated insurance claims systems, insurance companies can reduce their reliance on manual inspections, minimize workflow delays caused by human intervention in traditional assessment methods, and drive cost-effectiveness. Figure 1 This paper describes the entire process of an automated insurance claims system. Despite significant practical demand, research on automated vehicle damage detection remains relatively scarce. Recent work has primarily relied on single-modal visual detection (such as Mask R-CNN), but this is limited by factors such as the complexity of vehicle damage types, large scale differences, and numerous background interferences, leading to frequent missed detections and false detections (e.g., misclassifying reflections as dents). The automated vehicle damage detection system proposed in this application is mainly based on a detection network. Unlike existing single-modal damage detection systems, this network integrates prior knowledge from a visual language model. This application's system novelly enhances the automated vehicle damage detection system with prior knowledge embedded in a visual language pre-trained model.

[0058] Based on this, embodiments of this application provide a multimodal vehicle damage detection method based on damage activation maps. Figure 2 This is an optional flowchart illustrating a multimodal vehicle damage detection method based on damage activation maps provided in an embodiment of this application, which will be combined with... Figure 2 The steps shown are explained.

[0059] S101. Obtain the image of the vehicle to be inspected and the pre-determined damage category information.

[0060] In some embodiments of this application, the damage category information is predetermined, and the image of the vehicle to be detected can be acquired by a variety of different acquisition devices.

[0061] In some embodiments of this application, the image of the vehicle to be detected generally includes images of the vehicle from multiple angles and perspectives. For example, it may include a front view, a left view, a right view, and a top view of the vehicle, etc.

[0062] It should be noted that the image of the vehicle to be detected includes all information about the exterior of the vehicle.

[0063] In some embodiments of this application, the multimodal vehicle damage detection method based on damage activation maps can be used for automated non-contact vehicle surface damage detection and quantitative assessment, and is applicable to scenarios such as automobile insurance damage assessment and repair quality inspection.

[0064] In some embodiments of this application, the multimodal vehicle damage detection method based on damage activation maps is applicable to multimodal vehicle damage detection systems based on damage activation maps.

[0065] In some embodiments of this application, the multimodal vehicle damage detection system framework based on damage activation maps combines MaskR-CNN with a visual language pre-trained model for vehicle damage detection.

[0066] For example, this application defines the damage category as [C]={ dent, scratch, crack, glassshatter, tire flat, lamp broken} and uses the prompt template "a damaged car with class", where 'class' represents each damage category, resulting in a total of 6 texts.

[0067] S102. Based on the damage category information and the image of the vehicle to be detected, generate a damage activation map; and based on the image of the vehicle to be detected, generate a vehicle feature map through a backbone network model.

[0068] In some embodiments of this application, text information is determined based on damage category information. Encoding processing is performed on the text information and preset redundant prompt text information to determine multi-class damage information vectors and redundant prompt text vectors. Damage activation maps and mapping maps are generated based on the vehicle image to be detected, the multi-class damage information vectors, and the redundant prompt text vectors. A vehicle feature map is generated based on the vehicle image to be detected using a backbone network model.

[0069] In some embodiments of this application, based on damage category information and the image of the vehicle to be detected, a vehicle damage activation map, i.e., the vehicle damage region, is generated using a visual language pre-trained model and the proposed semantic subtraction operation; and a vehicle feature map is generated using a backbone network model based on the image of the vehicle to be detected.

[0070] It should be noted that the vehicle damage activation map is the same as the damage activation map.

[0071] For example, the six acquired text inputs are processed by CLIP's text encoder to generate multiple text embeddings. These embeddings are then averaged to obtain a vector F containing multiple types of damage information. t Next, this application designs a redundant prompt text, "a perfect car," and obtains its corresponding embedding F. r Through F t and F r By employing simple subtraction operations, this application can reduce noise activation occurring on the vehicle body, thereby allowing for better focus on defective areas of the vehicle.

[0072] It should be noted that F t For multi-class damage information vectors; F r This is a redundant prompt text vector.

[0073] S103. Based on the vehicle feature map and the damage activation map, perform image fusion to determine the enhanced feature map.

[0074] In some embodiments of this application, the vehicle feature map and the damage activation map are element-wise multiplied to obtain a first feature map; the vehicle feature map is convolved to obtain a second feature map; and the first feature map and the second feature map are fused based on preset learning parameters to obtain an enhanced feature map.

[0075] It should be noted that the preset learning parameters are set in advance and are used to perform weighted fusion of the first feature map and the second feature map.

[0076] S104. Perform vehicle damage detection on the enhanced feature map and determine the detection results of vehicle damage.

[0077] In some embodiments of this application, a preset detection model is used to detect vehicle damage on the enhanced feature map, obtaining bounding box regression loss value, mask loss value, cross-entropy loss value, and mean squared error loss value. The preset detection model includes four detection heads: a classification head, a bounding box regression head, a mask head, and a localization quality head. Based on the bounding box regression loss value, mask loss value, cross-entropy loss value, and mean squared error loss value, a classification loss value is determined. Based on the classification loss value, the classification result, localization information, and damage segmentation result of the vehicle damage are determined. And based on the classification result, localization information, and damage segmentation result of the vehicle damage, the detection result of the vehicle damage is determined.

[0078] Unlike the single-modal models of existing technical solutions, this application provides a multimodal vehicle damage detection scheme based on damage activation maps: it is based on Mask-RCNN and a multimodal feature extractor, and for the first time adopts a damage activation map method based on a semantic subtraction prompting strategy. Furthermore, it enhances feature realization performance by fusing damage activation maps and MAE image representations through a region-activated attention mechanism.

[0079] For example, the model in this application is built on the Mask R-CNN architecture, which basically includes the following key components: (1) a deep convolutional backbone network responsible for extracting feature maps from the input image, generally using the ResNet series model; (2) a neck module, usually a feature pyramid network, used to generate multi-scale feature pyramids to enhance the detection of objects of various sizes; (3) next is the region candidate extraction network (RPN), which can efficiently generate bounding boxes of candidate objects; (4) next is the region alignment layer (ROI Align), which can accurately extract fixed-size feature maps corresponding to each proposed region of interest; (5) finally, the detection head performs classification, bounding box regression and pixel-level mask prediction for each region of interest, as follows. Figure 3As shown in (a), the input image is processed by the backbone network, neck module, RPN and ROI Align, which can accurately extract a fixed-size feature map corresponding to each proposed region of interest; then classification, bounding box regression and pixel-level mask prediction are performed on each region of interest through classification head, bounding box regression head and masking head.

[0080] A multimodal vehicle surface damage detection framework is proposed. This application utilizes a visual language pre-trained model to design a novel detection network tailored for vehicle damage detection. The method first uses the original visual language pre-trained CLIP model to effectively generate damage activation maps that highlight vehicle damage areas. Then, a damage activation map fusion mechanism is proposed to effectively integrate the damage activation maps into the detection framework. Finally, an IoU output head is added to the detection head to improve the quality of defect mask generation. Overall, an effective integration framework is constructed based on the visual language pre-trained model, achieving effective training with minimal trainable parameters and without requiring a large-scale damaged vehicle image dataset. Furthermore, MAE is used as the backbone network. Prior knowledge from the visual language pre-training further enhances the backbone functionality. Specifically, to fully utilize the visual language pre-trained model, a novel set of cues is designed to generate CLIP-based damage activation maps. Then, the damage activation maps are fused with the Mask R-CNN backbone feature maps through a designed damage activation map fusion module to generate rich feature representations. These features are then processed sequentially by the neck module, RPN, and ROI Align; finally, bounding boxes and instance mask predictions are obtained through three independent detection heads. The unique activation mechanism of this application effectively learns contextual information during training, significantly enhancing the model's performance in feature extraction and improving its generalization ability in downstream tasks. The overall framework of the model is as follows: Figure 3 As shown in (b), the model mainly includes MAE, a lesion activation map fusion module, a lesion activation map generation module, a neck module, RPN, and ROI Align. Unlike Mask R-CNN, the backbone network in this application uses the MAE model and utilizes a visual language model to generate lesion activation maps. Subsequently, the features are enhanced by fusing the lesion activation maps, and the output includes lesion classification, localization, and segmentation.

[0081] Understandably, the process involves acquiring an image of the vehicle to be detected and pre-determined damage category information, using a multimodal damage activation mechanism to generate a high-purity damage activation map through a semantic subtraction prompting strategy, generating a vehicle feature map based on the vehicle image, and then fusing the vehicle feature map and the damage activation map using a dual-branch fusion module to obtain an effective enhanced feature map, thereby improving the quality of the enhanced feature map. Using the enhanced feature map for vehicle damage detection can improve the accuracy and efficiency of vehicle damage detection.

[0082] In some embodiments of this application, the generation of the damage activation map based on damage category information and the image of the vehicle to be detected in S102 can be implemented through S201-S204, as follows:

[0083] S201. Determine text information based on damage category information.

[0084] In some embodiments of this application, text information corresponding to the damage category information can be determined based on the damage category information; generally, one type of damage category information corresponds to one type of text information.

[0085] S202. Based on the text information and the preset redundant prompt text information, perform encoding processing to determine the multi-type damage information vector and the redundant prompt text vector.

[0086] In some embodiments of this application, based on text information, a CLIP text encoder is used to process the text information to obtain multiple text embeddings corresponding to multiple warm text information. The average of these multiple embeddings is then taken to obtain a multi-class damage information vector. Based on a preset redundant prompt text, encoding processing is performed to obtain a redundant prompt text vector corresponding to the preset redundant prompt text.

[0087] S203. Based on the image of the vehicle to be detected, multi-type damage information vectors, and redundant prompt text vectors, generate a damage activation map and a mapping map.

[0088] In some embodiments of this application, a normalization process and a self-attention mechanism are performed on the image of the vehicle to be detected to obtain an image feature map; wherein, the self-attention mechanism is a value-to-value self-attention mechanism; a damage activation map is generated based on multi-class damage information vectors, the image feature map, and redundant prompt text vectors; and a mapping map is generated based on multi-class damage information vectors and the image feature map.

[0089] S204. Determine the damage activation map based on the damage activation map and the mapping map.

[0090] For example, a damage activation map can be generated using a damage activation map generation module. Using the CLIP model to generate the damage activation map, this application introduces a novel cueing strategy based on semantic subtraction to enhance the activation of damage.

[0091] To address the missing activation problem, this application proposes a value-to-value (vv) self-attention mechanism instead of the traditional attention mechanism using query keys (qk), and innovatively utilizes vv attention features to obtain an activation map based on the difference.

[0092] The vv-attention mechanism ensures that attention is focused on features with the same semantics, rather than scattered across irrelevant background elements. The new vv-attention mechanism can be represented as follows: (1)

[0093] Furthermore, V represents the features obtained by linear transformation of the input image patch x; the FFN layer in the Vision Transformer (ViT) architecture may introduce external noise into the activation map. By omitting the FFN layer, the forward process of CLIP's ViT can be formulated as:

[0094] (2)

[0095] Where x represents the input image patch, Norm is the layer normalization, and F attn This represents the vv attention applied to the normalized input x. In obtaining the text embedding F... t F r and image feature map F i (Excluding class markers) the damage activation graph M can be obtained using the following formula. a :

[0096] (3)

[0097] Among them, M a The damage activation diagram; Image feature map; It is a vector of damage information of multiple types; This is a redundant prompt text vector.

[0098] Through the above operations, the method of this application can largely alleviate the problem of noise activation by eliminating redundant features. However, some damaged areas are inevitably overlooked. To address this issue, this application finds that extending the focus to damaged vehicles in certain specific scenarios can effectively mitigate this oversight. Therefore, this application addresses the issue of F... i and F t Element-wise multiplication is performed. Then, the maximum value is taken along the channel dimension to form a highlighted mapping M. s , means as follows:

[0099] (4)

[0100] Among them, Ms For mapping graphs; It is a vector of damage information of multiple types; This is an image feature map.

[0101] Finally, damage activation map M d For M a and M s The summation, the overall simplified process is as follows: Figure 4 As shown, a text encoder encodes multiple types of text prompts and redundant text prompts to obtain the multi-type text embedding F. t and redundant text embedding Vehicle images are encoded using an image encoder based on the vv attention mechanism (i.e., vv attention mechanism) to obtain image embeddings. Based on multi-class text embedding F t Redundant text embedding and image embedding Damage activation map calculation is performed to obtain the damage activation map.

[0102] It should be noted that multi-class text embedding F t This refers to multi-class damage information vectors; redundant text embedding. This refers to redundant prompt text vectors; image embedding. It refers to image feature maps. Multiple text prompts are simply text information; redundant text prompts are pre-set redundant prompt text information.

[0103] Understandably, a multimodal damage activation mechanism is used to generate a high-purity damage heatmap, i.e. a damage activation map, through a semantic subtraction prompting strategy. This facilitates subsequent damage activation map fusion and improves the accuracy of vehicle damage detection.

[0104] In some embodiments of this application, S103 can be implemented by S301-S303, as follows:

[0105] S301. Multiply the vehicle feature map and the damage activation map element by element to obtain the first feature map.

[0106] S302. Perform convolution processing on the vehicle feature map to obtain the second feature map.

[0107] In some embodiments of this application, the vehicle feature map is subjected to three convolution processes in sequence to obtain a first initial feature map, a second initial feature map, and a third initial feature map; a weight map is determined based on the first initial feature map and the second initial feature map; and a dot product is performed between the weight map and the third initial feature map to obtain a second feature map.

[0108] In some embodiments of this application, the first initial feature map and the second initial feature map are subjected to feature deformation to obtain a fourth initial feature map and a fifth initial feature map; the fourth initial feature map and the fifth initial feature map are multiplied to obtain a sixth initial feature map; and the sixth initial feature map is subjected to average pooling to obtain a weight map.

[0109] S303. Based on preset learning parameters, perform image fusion on the first feature map and the second feature map to obtain an enhanced feature map.

[0110] For example, image fusion can be performed through the damage activation map fusion module to obtain enhanced feature maps. After obtaining the damage activation map, the key is how to integrate it into the detection model. The damage activation map fusion module mainly consists of two branches: (1) The first branch is the vehicle feature map obtained by the backbone network MAE. The latent representation is obtained through a 1 × 1 convolutional layer; subsequently, this feature representation is compared with the damage activation map M. d Element-wise multiplication is performed to produce a feature representation F. d Thus, the feature representation embeds semantic information about vehicle defects; (2) To enhance the feature representation, this module introduces a second branch. Unlike traditional self-attention which enhances the feature representation by aggregating semantically similar features in the image, this branch specifically highlights the features of the candidate damaged region to ensure accurate localization. Specifically, the vehicle feature map obtained by the underlying network MAE The query Q, key K, and value V are obtained by performing three convolutions respectively. Q and K are then reshaped to obtain values ​​of size V. and The characteristics of these, when multiplied, yield the following... The features are processed by an activation function and average pooling is performed on the last dimension to obtain a size of . Weighted graph C f Finally, C f Performing a dot product with V yields the second feature representation F. c This module differs from traditional self-attention mechanisms. Traditional self-attention, after multiplying Q and K to obtain a feature map, does not perform pooling to obtain an average map; instead, it performs a weighted sum of global features to obtain new features. The method proposed in this application focuses more on obtaining the activation of the region, generating a feature map of the activation loss region. To balance the output F of the two branches... d and F c A preset learning parameter α is introduced. The flowchart of the fusion module is as follows: Figure 5 As shown, the feature map is processed by a convolutional layer and a QKV damage attention module, and then fused with the damage activation map to finally output the enhanced feature map F. o The formula is as follows:

[0111] (5)

[0112] in, To enhance the feature map; This is the first feature map; This is the second feature map; These are the preset learning parameters.

[0113] It should be noted that query Q is the first initial feature map; key K is the second initial feature map; and value V is the third initial feature map. The feature is the fourth initial feature map; The feature is the fifth initial feature map; The feature is the sixth initial feature map; It is a preset learning parameter that can be learned; Enhanced feature map F o That is to say Figure 5 Enhanced features in.

[0114] Understandably, the use of a multimodal damage activation mechanism to generate a high-purity damage heatmap through a semantic subtraction prompting strategy, combined with a dual-branch fusion module, effectively enhances the feature map and improves the accuracy and speed of vehicle damage detection.

[0115] In some embodiments of this application, S104 can be implemented by S401-S402, as follows:

[0116] S401. Using a preset detection model, vehicle damage detection is performed on the enhanced feature map to obtain the classification results, localization information, and damage segmentation results of the vehicle damage. The preset detection model includes four detection heads: a classification head, a bounding box regression head, a masking head, and a localization quality head.

[0117] S402. Based on the classification results, location information, and damage segmentation results of vehicle damage, determine the detection results of vehicle damage.

[0118] For example, after obtaining the enhanced feature map, it passes through the RPN and ROI Align networks. To detect the class of instance masks and damaged regions, the model in this application includes four detection heads: a classification head, a bounding box regression head, a masking head, and the IoU head proposed in this model. The IoU head directly predicts a scalar value for each class, representing the IoU between each predicted instance mask and its corresponding ground truth value, and also serves as a confidence index of mask quality. The higher the IoU value, the higher the mask confidence. During actual prediction, the predicted IoU value is complementary to the score output by the classification head, effectively suppressing false positive predictions in normal regions. Specifically, for the IoU head, the predicted mask from the masking branch is concatenated with the features generated by RoI Align to form its input. Then, it passes through four convolutional blocks and two fully connected layers, with the final fully connected layer outputting the number of classes. The model's loss function is constructed as follows:

[0119] (6)

[0120] Where L represents the total loss of the model, L cls To represent the classification loss, and in order to adaptively balance the difficulty of samples, the focus loss function is used for classification loss; L box The bounding box regression loss is represented by the balanced L1 loss function; L mask To represent the mask loss, the cross-entropy loss function is used; L IoU To quantify the difference between the predicted and true IoU scores, a mean squared error loss function is used with a loss weight γ of 0.5. Unlike the CarDD method, this application introduces an IoU loss to enhance detection accuracy.

[0121] Understandably, existing vehicle damage detection methods face several core challenges—false detections due to imaging condition sensitivity (e.g., vehicle reflections being misjudged as dents), significant differences in damage scale (from millimeter-level scratches to large-area dents), and missed detections in overlapping areas of multiple damage types. Traditional single-modal detection frameworks and open-vocabulary models have significant limitations in addressing these issues. This application proposes the first vehicle damage detection framework based on visual language pre-training. It employs a multimodal damage activation mechanism to generate high-purity damage heatmaps through a semantic subtraction prompting strategy. Combined with a dual-branch fusion module, it effectively enhances the feature maps, achieving performance and speed improvements. This application provides a fully automated, non-contact assessment solution for insurance claims systems, achieving lower process time, lower damage quantification errors, and better pixel-level segmentation results, thus promoting the standardization and intelligent evolution of auto insurance claims.

[0122] Based on the damage activation map-based multimodal vehicle damage detection method described in the above embodiments, this application also provides a damage activation map-based multimodal vehicle damage detection system, such as... Figure 6 As shown, Figure 6 This application provides a schematic diagram of the structure of a multimodal vehicle damage detection system based on damage activation maps. The multimodal vehicle damage detection system 6 includes: an acquisition module 601, a generation module 602, and a detection module 603; wherein,

[0123] The acquisition module 601 is used to acquire the image of the vehicle to be detected and the pre-determined damage category information;

[0124] The generation module 602 is used to generate a damage activation map based on the damage category information and the image of the vehicle to be detected; and to generate a vehicle feature map based on the image of the vehicle to be detected through a backbone network model.

[0125] The detection module 603 is used to perform image fusion based on the vehicle feature map and the damage activation map to determine an enhanced feature map; and to perform vehicle damage detection on the enhanced feature map to determine the detection result of vehicle damage.

[0126] In some embodiments of this application, the detection module 603 is further configured to determine text information based on the damage category information; and to perform encoding processing based on the text information and preset redundant prompt text information to determine multiple damage information vectors and redundant prompt text vectors.

[0127] The generation module 602 is further configured to generate a damage activation map and a mapping map based on the vehicle image to be detected, the multi-type damage information vector, and the redundant prompt text vector;

[0128] The detection module 603 is further configured to determine the damage activation map based on the damage activation map and the mapping map.

[0129] In some embodiments of this application, the generation module 602 is further configured to perform normalization processing and self-attention mechanism processing based on the vehicle image to be detected to obtain an image feature map; wherein, the self-attention mechanism is a value-to-value self-attention mechanism; generate the damage activation map based on the multi-class damage information vector, the image feature map and the redundant prompt text vector; and generate the mapping map based on the multi-class damage information vector and the image feature map.

[0130] In some embodiments of this application, the acquisition module 601 is further configured to perform element-wise multiplication on the vehicle feature map and the damage activation map to obtain a first feature map; perform convolution processing on the vehicle feature map to obtain a second feature map; and perform image fusion on the first feature map and the second feature map based on preset learning parameters to obtain the enhanced feature map.

[0131] In some embodiments of this application, the acquisition module 601 is further configured to perform three convolution processes on the vehicle feature map in sequence to obtain a first initial feature map, a second initial feature map, and a third initial feature map;

[0132] The detection module 603 is further configured to determine a weight map based on the first initial feature map and the second initial feature map;

[0133] The acquisition module 601 is further configured to perform a dot product between the weight map and the third initial feature map to obtain the second feature map.

[0134] In some embodiments of this application, the acquisition module 601 is further configured to perform feature deformation on the first initial feature map and the second initial feature map to obtain a fourth initial feature map and a fifth initial feature map; multiply the fourth initial feature map and the fifth initial feature map to obtain a sixth initial feature map; and perform average pooling on the sixth initial feature map to obtain the weight map.

[0135] In some embodiments of this application, the acquisition module 601 is further configured to perform vehicle damage detection on the enhanced feature map using a preset detection model, and obtain the classification result, location information and damage segmentation result of the vehicle damage; the preset detection model includes four detection heads; the four detection heads are a classification head, a bounding box regression head, a masking head and a location quality head;

[0136] The detection module 603 is further configured to determine the detection result of the vehicle damage based on the classification result of the vehicle damage, the positioning information, and the damage segmentation result.

[0137] Based on the damage activation map-based multimodal vehicle damage detection method described in the above embodiments, this application also provides a damage activation map-based multimodal vehicle damage detection device, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of a multimodal vehicle damage detection device based on damage activation maps, provided in an embodiment of this application. The device includes a processor 701 and a memory 702. The memory 702 stores a computer program; the processor 701 retrieves and runs the computer program from the memory to execute the multimodal vehicle damage detection method based on damage activation maps as described in the above embodiment.

[0138] In the embodiments of this application, the processor 701 described above can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0139] This application provides a computer-readable storage medium storing a computer program for implementing the multimodal vehicle damage detection method based on damage activation maps as described in any of the above embodiments when executed by a processor.

[0140] For example, the program instructions corresponding to a multimodal vehicle damage detection method based on damage activation maps in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a multimodal vehicle damage detection method based on damage activation maps in the storage media are read or executed by an electronic device, the multimodal vehicle damage detection method based on damage activation maps as described in any of the above embodiments can be implemented.

[0141] Furthermore, in the embodiments of this application, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0142] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0144] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0146] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0147] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0148] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0149] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0150] The above description is merely an embodiment of this application, but the protection scope of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A multi-modal vehicle damage detection method based on lesion activation map, characterized in that, The method comprises: acquiring a vehicle image to be detected and predetermined damage category information; generating a damage activation map based on the damage category information and the vehicle image to be detected, and generating a vehicle feature map through a backbone network model based on the vehicle image to be detected; performing image fusion based on the vehicle feature map and the damage activation map to determine an enhanced feature map; performing vehicle damage detection on the enhanced feature map to determine a detection result of vehicle damage; the generation of the damage activation map based on the damage category information and the vehicle image to be detected comprises: determining text information based on the damage category information; performing encoding processing based on the text information and preset redundant prompt text information to determine a multi-class damage information vector and a redundant prompt text vector; performing normalization processing and self-attention mechanism processing based on the vehicle image to be detected to obtain an image feature map; wherein the self-attention mechanism processing is a value-to-value self-attention mechanism processing; generating a damage activation map based on the multi-class damage information vector, the image feature map and the redundant prompt text vector; generating a mapping map based on the multi-class damage information vector and the image feature map; determining a damage activation map based on the damage activation map and the mapping map; the image fusion based on the vehicle feature map and the damage activation map to determine the enhanced feature map comprises: performing element multiplication on the vehicle feature map and the damage activation map to obtain a first feature map; performing convolution processing on the vehicle feature map three times in turn to obtain a first initial feature map, a second initial feature map and a third initial feature map; determining a weight map based on the first initial feature map and the second initial feature map; performing dot multiplication on the weight map and the third initial feature map to obtain a second feature map; performing image fusion on the first feature map and the second feature map based on preset learning parameters to obtain the enhanced feature map.

2. The method of claim 1, wherein, the determination of the weight map based on the first initial feature map and the second initial feature map comprises: performing feature morphing on the first initial feature map and the second initial feature map to obtain a fourth initial feature map and a fifth initial feature map; performing multiplication on the fourth initial feature map and the fifth initial feature map to obtain a sixth initial feature map; performing average pooling processing on the sixth initial feature map to obtain the weight map.

3. The method of claim 1, wherein, the vehicle damage detection on the enhanced feature map to determine the detection result of vehicle damage comprises: performing vehicle damage detection on the enhanced feature map through a preset detection model to obtain a classification result, positioning information and damage segmentation result of vehicle damage; wherein the preset detection model comprises four detection heads; the four detection heads are a classification head, a bounding box regression head, a mask head and a positioning quality head; determining the detection result of vehicle damage according to the classification result of vehicle damage, the positioning information and the damage segmentation result.

4. A multi-modal vehicle damage detection system based on lesion activation map, characterized in that, The multi-modal vehicle damage detection system based on a damage activation map comprises an acquisition module, a generation module and a detection module, wherein the acquisition module is configured to acquire a vehicle image to be detected and predetermined damage category information; The generation module is configured to generate a damage activation map based on the damage category information and the to-be-detected vehicle image, and generate a vehicle feature map based on the to-be-detected vehicle image by using a backbone network model; The detection module is configured to perform image fusion based on the vehicle feature map and the damage activation map to determine an enhanced feature map, and perform vehicle damage detection on the enhanced feature map to determine a detection result of vehicle damage. The generation module is further configured to determine text information based on the damage category information, perform encoding processing based on the text information and preset redundant prompt text information to determine a multi-class damage information vector and a redundant prompt text vector, perform normalization processing and self-attention mechanism processing based on the to-be-detected vehicle image to obtain an image feature map, wherein the self-attention mechanism processing is value-to-value self-attention mechanism processing, generate a damage activation map based on the multi-class damage information vector, the image feature map and the redundant prompt text vector, and generate a mapping map based on the multi-class damage information vector and the image feature map. The detection module is further configured to multiply the vehicle feature map and the damage activation map to obtain a first feature map, sequentially perform three convolution processing operations on the vehicle feature map to obtain a first initial feature map, a second initial feature map and a third initial feature map, determine a weight map based on the first initial feature map and the second initial feature map, multiply the weight map and the third initial feature map to obtain a second feature map, and perform image fusion on the first feature map and the second feature map based on preset learning parameters to obtain the enhanced feature map.

5. A multi-modal vehicle damage detection apparatus based on lesion activation map, characterized in that, Comprise: A processor and a memory, wherein The memory is configured to store a computer program; The processor is configured to call and run the computer program from the memory to execute the method of any one of claims 1 to 3.

6. A computer readable storage medium characterized by, Executable instructions are stored, which cause the processor to implement the method of any one of claims 1 to 3 when executed.

Images