A vehicle exterior component damage identification method and system based on multi-target query Transformer

Abstract

The application discloses a vehicle appearance part damage identification method based on a multi-target query Transformer, and belongs to the field of image processing; the method comprises the following steps: acquiring a vehicle damage image; pre-processing the vehicle damage image; extracting features of the pre-processed image; generating positional information encoding according to the feature map; performing encoding processing on the feature map and the positional information encoding through a Transformer model; acquiring a target query; performing parallel decoding processing on the target query and an image vector through the Transformer model; and performing classification processing on an image decoding vector. The application also discloses a vehicle appearance part damage identification system based on a multi-target query Transformer. The application reduces the algorithm development cost, trains a single Transformer model for identifying appearance parts and corresponding damage types, does not need to respectively develop segmentation models for detecting appearance parts and damage types, reduces the model training cost, and simultaneously utilizes the correlation between the parts and the damage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a vehicle appearance component damage identification method and system based on multi-target query Transformer. BACKGROUND

[0002] In the vehicle accident damage assessment scene, on-site staff needs to go to the accident scene to investigate the accident vehicle and determine the vehicle damage components and corresponding damage types for subsequent claims settlement. Or let the on-site user take photos, videos, etc. to upload to the relevant software, use a deep learning model to identify the damaged area, determine the damaged components and damage types, and realize intelligent damage assessment. When using a deep learning model for intelligent damage assessment, all components and damage contours in the damaged area need to be accurately labeled, resulting in high data labeling costs; at the same time, component and damage segmentation models need to be trained separately, lacking the interaction between the two and increasing the algorithm development cost.

[0003] Currently, when using a deep learning model for intelligent damage assessment of vehicle appearance components, the positions of components (such as bumpers, hoods, grilles, headlights, etc.) and damages (such as scratches, dents, cracks, etc.) are detected and identified, and then the relevance between components and damages is matched to give damage results for different components. This method requires labeling the positions and names of components and damages, then training component segmentation models and damage segmentation models, and further matching the segmented components and damages to obtain different components and corresponding damage types.

[0004] For vehicle appearance component damage identification scenarios, the existing technical solutions have the following problems:

[0005] 1. High data labeling cost. The contours of different components and damage contours on the image need to be labeled, which takes a long time;

[0006] 2. High algorithm development cost. Component segmentation models and damage segmentation models need to be trained separately, increasing the model development cost. SUMMARY

[0007] The purpose of the present application is to provide a vehicle appearance component damage identification method and system based on multi-target query Transformer with low data labeling cost.

[0008] To solve the above technical problems, the present application provides a vehicle appearance component damage identification method based on multi-target query Transformer, comprising the following steps:

[0009] Obtain a vehicle damage image;

[0010] Preprocess the vehicle damage image to obtain a preprocessed image;

[0011] extracting features of the preprocessed image to obtain a feature map;

[0012] generating positional information based on the feature map;

[0013] encoding the feature map and the positional information by using a Transformer model to obtain an image vector;

[0014] obtaining a target query;

[0015] parallel decoding the target query and the image vector by using the Transformer model to obtain an image decoder vector;

[0016] classifying the image decoder vector to obtain a prediction result.

[0017] Preferably, the step of extracting features of the preprocessed image to obtain a feature map comprises the following steps:

[0018] inputting the preprocessed image into a backbone network to extract features and obtain the feature map.

[0019] Preferably, the backbone network is a swin Transformer network.

[0020] Preferably, the step of generating positional information based on the feature map comprises the following steps:

[0021] performing convolution compression on the feature map to obtain effective features;

[0022] performing sine-cosine calculation on the effective features to obtain the positional information.

[0023] Preferably, the step of encoding the feature map and the positional information by using a Transformer model to obtain an image vector comprises the following steps:

[0024] inputting the positional information into a Transformer encoder structure to encode and obtain the image vector.

[0025] Preferably, the target query and the image vector image embedding are decoded in parallel by the Transformer model to obtain an image decoding vector image decoder embedding, specifically including the following steps:

[0026] The image vector image embedding and the target query are input into the Transformer decoder structure for parallel decoding to obtain the image decoding vector image decoder embedding.

[0027] Preferably, the image decoding vector image decoder embedding is classified to obtain a prediction result, specifically including the following steps:

[0028] The image decoding vector image decoder embedding is input into a classification model for classification to output the prediction result.

[0029] Preferably, the vehicle damage image is preprocessed, specifically including the following steps:

[0030] The vehicle damage image is adjusted to a preset size and then normalized.

[0031] The application also discloses a vehicle appearance component damage identification system based on a multi-target query Transformer, comprising:

[0032] A first acquisition module is configured to acquire a vehicle damage image.

[0033] A preprocessing module is configured to preprocess the vehicle damage image to obtain a preprocessed image.

[0034] A feature extraction module is configured to extract features of the preprocessed image to obtain a feature map.

[0035] A position information generation module is configured to generate position information positional encoding according to the feature map.

[0036] An encoding module is configured to encode the feature map and the position information positional encoding by a Transformer model to obtain an image vector image embedding.

[0037] A second acquisition module is configured to acquire a target query.

[0038] The decoding module is configured to perform parallel decoding processing on the target query and the image vector image embedding by using a Transformer model to obtain an image decoding vector image decoder embedding.

[0039] The classification module is configured to perform classification processing on the image decoding vector image decoder embedding to obtain a prediction result.

[0040] Compared with the prior art, the present application has the following advantages:

[0041] The present application first photographs the vehicle damage area, only needs to mark the component name and the corresponding damage type appearing on the image, then uses the Transformer model for joint training of the component and the damage, and finally infers the component damage result on the image.

[0042] The present application has the following advantages:

[0043] (1) Low data labeling cost. Only the component name and the corresponding damage type on the current vehicle damage area need to be labeled, without the need to label the contour information of different components and damages, which is very convenient; and the cost of subsequently increasing new samples is also relatively low.

[0044] (2) Reduces the algorithm development cost. A single Transformer model is trained to identify the appearance component and the corresponding damage type, without the need to develop separate segmentation models for detecting the appearance component and the damage type, thereby reducing the model training cost and utilizing the correlation between the component and the damage. BRIEF DESCRIPTION OF DRAWINGS

[0045] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] Figure 1 is a schematic view of a right front side view of vehicle damage;

[0047] Figure 2 is a schematic view of a right rear side view of vehicle damage;

[0048] Figure 3 is a schematic view of a Transformer model structure;

[0049] Figure 4 is a schematic view of a vehicle appearance component damage identification method based on a multi-target query Transformer. DETAILED DESCRIPTION

[0050] In the following description, numerous specific details are set forth to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure the present application.

[0051] The terminology used in this description of one or more embodiments should not be interpreted as limiting the scope of the description of one or more embodiments. The singular forms "a," "an," and "the" used in this description of one or more embodiments should not be interpreted as limiting the scope of the description of one or more embodiments to a single referent unless otherwise indicated by context. It will be understood that the terms "and / or," "comprises," "comprising," "includes," and / or "including," as used herein, refer to elements that are either present or not present in the described embodiments. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including," when used in this description of one or more embodiments, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0052] It should be understood that although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a particular order or hierarchy among the information. The terms are used merely to distinguish one from another. For example, a first entity discussed below could be termed a second entity, and, similarly, a second entity could be termed a first entity without departing from the scope of the present description of one or more embodiments. As used herein, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" taking into account the context in which the term is used.

[0053] The present application is further described in detail below with reference to the attached drawing figures:

[0054] Disclosed is a vehicle exterior part damage identification method based on a multi-target query Transformer, comprising the following steps:

[0055] Obtaining a vehicle damage image;

[0056] Pretreating the vehicle damage image to obtain a pretreated image;

[0057] Extracting features of the pretreated image to obtain a feature map;

[0058] Generating positional encoding according to the feature map;

[0059] Encoding the feature map and the positional encoding by a Transformer model to obtain an image vector image embedding;

[0060] Obtaining a target query;

[0061] Parallel decoding processing is performed on the target query and the image vector image embedding by the Transformer model to obtain an image decoding vector image decoder embedding.

[0062] The image decoding vector image decoder embedding is classified to obtain a prediction result.

[0063] Preferably, features of the preprocessed image are extracted to obtain a feature map, specifically including the following steps:

[0064] The preprocessed image is input into a backbone network for extraction to obtain the feature map.

[0065] Preferably, the backbone network is a swin Transformer network.

[0066] Preferably, the position information positional encoding is generated according to the feature map, specifically including the following steps:

[0067] The feature map is convoluted and compressed to obtain effective features.

[0068] The effective features are calculated by sine and cosine to obtain the position information positional encoding.

[0069] Preferably, the feature map and the position information positional encoding are encoded by the Transformer model to obtain the image vector image embedding, specifically including the following steps:

[0070] The position information positional encoding is input into a Transformer encoder structure for encoding to obtain the image vector image embedding.

[0071] Preferably, parallel decoding processing is performed on the target query and the image vector image embedding by the Transformer model to obtain the image decoding vector image decoder embedding, specifically including the following steps:

[0072] The image vector image embedding and the target query are jointly input into a Transformer decoder structure for parallel decoding to obtain the image decoding vector image decoder embedding.

[0073] Preferably, the image decoding vector image decoder embedding is classified to obtain a prediction result, specifically including the following steps:

[0074] The image decoding vector image decoder embedding is input into a classification model for classification, and a prediction result is output.

[0075] Preferably, the vehicle damage image is preprocessed, specifically including the following steps:

[0076] The vehicle damage image is adjusted to a preset size and then normalized.

[0077] The application also discloses a vehicle appearance component damage identification system based on a multi-target query Transformer, comprising:

[0078] The first acquisition module is configured to acquire a vehicle damage image.

[0079] The preprocessing module is configured to preprocess the vehicle damage image to obtain a preprocessed image.

[0080] The feature extraction module is configured to extract features of the preprocessed image to obtain a feature map.

[0081] The position information generation module is configured to generate position information positional encoding according to the feature map.

[0082] The encoding module is configured to encode the feature map and the position information positional encoding by using a Transformer model to obtain an image vector image embedding.

[0083] The second acquisition module is configured to acquire a target query.

[0084] The decoding module is configured to decode the target query and the image vector image embedding in parallel by using a Transformer model to obtain an image decoding vector image decoder embedding.

[0085] The classification module is configured to classify the image decoding vector image decoder embedding to obtain a prediction result.

[0086] In order to better illustrate the technical effects of the application, the application provides the following specific embodiment to illustrate the above technical process:

[0087] Embodiment 1: A vehicle appearance component damage identification method based on a multi-target query Transformer:

[0088] To solve the problems in the existing detection and identification of vehicle appearance component damage solutions, the application provides an end-to-end vehicle appearance component damage identification method based on multi-object query Transformer, which is used to identify the components and corresponding damage types in the image.

[0089] The identification system mainly includes three parts, namely a data module, a model structure module and an analysis module.

[0090] I. Data module

[0091] The damaged area of the vehicle is photographed, which usually contains multiple identifiable appearance components, such as Figure 1 、 Figure 2 As shown in Figure 1 and Figure 2 give the visual effect under different damage areas of the vehicle, and mark the outline information of the components and damage types, which is convenient and intuitive to understand. Figure 1 The damage situation in is: (1) multiple scratches exist on the hub; (2) the fender has a dent and multiple scratches; (3) the bumper has multiple scratches. Figure 2 The door in has a obvious dent.

[0092] In the process of making training data, first determine the appearance components on the vehicle (such as bumper, mesh grille, engine cover and windshield glass, etc.) and common damage types (such as scratches, dents and tears, etc.), then mark the component name and corresponding damage type in the image, and finally map it into the following form:

[0093] imgs / 1.jpg 1272 720 3 0 14 1 0 2

[0094] imgs / 2.jpg 1600 859 14 2 3 2 4 0 13 2 2 2

[0095] imgs / 3.jpg 960 659 0 0 14 4 3 4 2 0 13 0

[0096] imgs / 4.jpg 960 720 7 3 18 3 6 3 17 3 10 0

[0097] Wherein, each row of data represents a sample, and the format is "image relative position width height component name number damage type number component name number damage type number...".

[0098] II. Model structure module

[0099] With the Transformer model in the field of computer vision, the present application adopts a multi-target query Transformer to realize end-to-end vehicle appearance component identification and damage determination, as shown in Figure 3

[0100] The end-to-end multi-target query Transformer algorithm has the following characteristics:

[0101] (1) The target query in the Transformer is mapped to different appearance components, so that the model can directly learn the features of different appearance components for subsequent identification of damaged components;

[0102] (2) The identification component and the damage type are directly integrated into one model, realizing end-to-end model training;

[0103] The proposed end-to-end multi-target query Transformer model mainly consists of a backbone network, a Transformer encoder-decoder and a classification head. The operation process of the model is as follows:

[0104] (1) The obtained vehicle damage area image is subjected to resize and normalization operations to obtain a normalized image;

[0105] (2) The normalized image is input into the backbone network to extract deep image features and obtain a feature map. The backbone used in this paper is swin Transformer;

[0106] (3) The feature map in step (2) is compressed by 1*1 convolution, and then the position information positional encoding is generated by the sine formula to represent the relative relationship between different pixels, which is simultaneously input into the Transformer encoder-decoder structure;

[0107] (4) The feature obtained in step (3) is input into the Transformer encoder (6 layers) model to obtain image embedding;

[0108] (5) The target query representation is mapped to the query features of different appearance components, which is a set of randomly generated learnable embeddings, and the number is consistent with the number of appearance components, which is then combined with image embedding and input into the Transformer decoder structure to obtain new image decoder embedding;

[0109] ​(6) Different image decoder embeddings are used to determine whether the current component exists in the current vehicle damage image through the component classification branch, and the damage type corresponding to the current component is determined through the damage classification branch. The classification branch is mainly implemented by full connection and softmax.

[0110] III. Analysis module

[0111] The model prediction result is analyzed, the component type not in the current image is removed, the component name with damage in the current image is obtained, and the specific damage type is given, which is used for subsequent claim settlement.

[0112] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiment described above is only illustrative, for example, the division of the module, the module or the unit is only a logical function division, and actual implementation can have another division way, for example, a plurality of units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0113] The unit can be or can not be physically separated, and the components displayed as units can be a physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0114] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0115] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the methods of the present disclosure are performed. It should be noted that the above-described computer readable medium of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above.

[0116] The flow charts and block diagrams in the drawings are illustrations of possible architectural, functional, and operational architectures of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow charts or block diagrams can represent a module, a segment, or a portion of code which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, can be implemented by special purpose hardware-based systems which perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0117] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or replacements within the technical scope disclosed by the present disclosure should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for identifying damage to vehicle exterior components based on multi-objective query Transformer, characterized in that, Includes the following steps: Acquire images of vehicle damage; The vehicle damage image is preprocessed to obtain the preprocessed image; Extract features from the preprocessed image to obtain a feature map; Based on the feature map, positional encoding is generated. The feature map and positional information are encoded using the Transformer model to obtain the image embedding. Retrieve the target query; the target query represents the query features mapped to different appearance components; The target query and image embedding are decoded in parallel using the Transformer model to obtain the image decoder embedding. The image decoder embedding is processed through a component classification branch to determine whether the current component exists in the current vehicle damage image; at the same time, it is processed through a damage classification branch to determine the damage type corresponding to the current component. Label the component names and corresponding damage types in the vehicle damage images.

2. The method for identifying vehicle exterior component damage based on multi-objective query Transformer according to claim 1, characterized in that, Extracting features from the preprocessed image to obtain a feature map involves the following steps: The preprocessed image is input into the backbone network for extraction to obtain feature maps.

3. The method for identifying vehicle exterior component damage based on multi-objective query Transformer according to claim 2, characterized in that: The backbone network is a Swing Transformer network.

4. The method for identifying vehicle exterior component damage based on multi-objective query Transformer according to claim 1, characterized in that, Based on the feature map, positional encoding of positional information is generated, which includes the following steps: Convolutional compression is performed on the feature map to obtain effective features; The effective features are subjected to sine and cosine calculations to obtain positional encoding information.

5. The method for identifying vehicle exterior component damage based on multi-objective query Transformer according to claim 1, characterized in that, The feature map and positional information are encoded using the Transformer model to obtain the image embedding. This process includes the following steps: The feature map and positional information are input into the Transformer encoder structure for encoding to obtain the image vector, image embedding.

6. The method for identifying vehicle exterior component damage based on multi-objective query Transformer according to claim 1, characterized in that, The target query and image embedding are decoded in parallel using the Transformer model to obtain the image decoder embedding. The specific steps include: The image vector (image embedding) and the target query are input together into the Transformer decoder structure for parallel decoding to obtain the image decoder embedding.

7. The method for identifying vehicle exterior component damage based on multi-objective query Transformer according to claim 1, characterized in that, The image decoder embedding is classified to obtain the prediction result. The specific steps include: The image decoder embedding is input into the classification model for classification, and the prediction result is output.

8. The method for identifying damage to vehicle exterior components based on multi-objective query Transformer according to claim 1, characterized in that, Preprocessing vehicle damage images includes the following steps: The vehicle damage image is adjusted to a preset size and then normalized.

9. A vehicle exterior component damage recognition system based on multi-objective query Transformer, used to implement the vehicle exterior component damage recognition method based on multi-objective query Transformer as described in any one of claims 1-8, characterized in that, include: The first acquisition module is used to acquire vehicle damage images; The preprocessing module is used to preprocess vehicle damage images to obtain preprocessed images; The feature extraction module is used to extract features from the preprocessed image to obtain a feature map; The position information generation module is used to generate positional encoding based on the feature map. The encoding module is used to encode the feature map and positional information using the Transformer model to obtain the image embedding. The second acquisition module is used to acquire the target query; The decoding module is used to perform parallel decoding of the target query and the image vector image embedding through the Transformer model to obtain the image decoder embedding. The classification module is used to classify the image decoder embedding and obtain the prediction result.

Images