Vehicle loss assessment method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a vehicle damage assessment method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a plurality of to-be-processed vehicle damage images; inputting the plurality of vehicle damage images into a vehicle damage identification model, wherein the vehicle damage identification model is a multi-modal large model based on Transform; and obtaining a vehicle damage assessment result according to the output of the vehicle damage identification model. According to the method, image recognition is carried out based on a Transform multi-mode large model, multi-image input can be accepted, multi-image information can be interacted, and the information utilization rate is greatly improved; the method directly outputs the final prediction, does not need other post-processing processes, and greatly improves the loss assessment efficiency.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a vehicle damage assessment method, device, computer equipment and storage medium. Background Art

[0002] In order to quickly respond to customers' accident claims, insurance companies need to send professional surveyors to the accident site to conduct on-site surveys and assessments, provide vehicle repair plans and compensation amounts, and take photos of the scene. The photos of the assessments are archived for back-end verification personnel to assess the damage and price. Since manual surveys and assessments are required, insurance companies need to consume a lot of manpower, material resources and financial resources.

[0003] The development of artificial intelligence technology has made it possible to use artificial intelligence technology to handle vehicle accident claims. In the existing vehicle intelligent damage assessment system, it is necessary to determine the damage details of the report pictures uploaded by users to assist professionals in vehicle damage assessment. The traditional detection and segmentation method requires independent processing of each picture, and the information cannot be exchanged with each other and requires multiple cumbersome post-processing processes. The detection efficiency of vehicle damage is low, and the damage assessment efficiency is low. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a vehicle damage assessment method, apparatus, computer equipment and storage medium, aiming to solve the problem in the prior art that it is necessary to determine the damage details of the report pictures uploaded by users to assist professionals in vehicle damage assessment, the traditional detection and segmentation method needs to process each picture independently, the information cannot be exchanged with each other and multiple post-processing processes are required, which is cumbersome, the vehicle damage detection efficiency is low, and the damage assessment efficiency is low.

[0005] The technical solution of the present invention is as follows:

[0006] A first embodiment of the present invention provides a vehicle damage assessment method, the method comprising:

[0007] Acquire several vehicle damage images to be processed;

[0008] Inputting the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multimodal model based on Transformer;

[0009] The vehicle damage assessment result is obtained based on the output of the vehicle damage identification model.

[0010] Another embodiment of the present invention provides a vehicle damage assessment device, the device comprising:

[0011] An image acquisition module, used to acquire a number of vehicle damage images to be processed;

[0012] A data input module, used for inputting the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multi-modal model based on Transformer;

[0013] The result acquisition module is used to obtain the vehicle damage assessment result based on the output of the vehicle damage identification model.

[0014] Another embodiment of the present invention provides a computer device, the computer device comprising at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the above-mentioned vehicle damage assessment method.

[0017] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the steps of the above-mentioned vehicle damage assessment method.

[0018] Beneficial effects: The vehicle damage assessment method of the embodiment of the present invention can obtain a number of vehicle damage images to be processed; input the several vehicle damage images into a vehicle damage recognition model, which is a large multimodal model based on Transformer; and obtain the vehicle damage assessment result according to the output of the vehicle damage recognition model. The present invention performs image recognition based on the large multimodal model of Transformer, and can accept multiple image inputs, and the information of multiple images can interact, which greatly improves the information utilization rate; the method directly outputs the final prediction without the need for other post-processing processes, and the efficiency of damage assessment is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of an application environment of an embodiment of a vehicle damage assessment method of the present invention;

[0021] Figure 2 A schematic diagram of a flow chart of a preferred embodiment of a vehicle damage assessment method of the present invention;

[0022] Figure 3 A schematic diagram of a specific application example of a vehicle damage assessment method of the present invention;

[0023] Figure 4 for Figure 3 A schematic flow chart of a specific embodiment of step S12;

[0024] Figure 5 It is a flow chart of another specific application embodiment of a vehicle damage assessment method of the present invention;

[0025] Figure 6 for Figure 5 A schematic flow chart of a specific embodiment of step S22;

[0026] Figure 7 It is a flow chart of another specific application embodiment of a vehicle damage assessment method of the present invention;

[0027] Figure 8 for Figure 2 A flowchart of a specific embodiment of step S100;

[0028] Fig. 9 A schematic diagram of functional modules of a preferred embodiment of a vehicle damage assessment device of the present invention;

[0029] Fig.10 A structural schematic diagram of a preferred embodiment of a computer device of the present invention;

[0030] Fig.11 Another structural schematic diagram of a preferred embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0033] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. Here, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0034] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robots, natural speech processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0035] The vehicle damage assessment method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client obtains several vehicle damage images to be processed; inputs the several vehicle damage images into the vehicle damage recognition model, and the vehicle damage recognition model is a large multimodal model based on Transformer; obtains the vehicle damage assessment result according to the output of the vehicle damage recognition model. In the present invention, image recognition is performed based on the large multimodal model of Transformer, so that multiple images can be accepted as input, and the information of multiple images can interact, and the information utilization rate is greatly improved; the method directly outputs the final prediction without other post-processing processes, and the damage assessment efficiency is greatly improved. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0036] In view of the above problems, the present invention provides a method for assessing vehicle damage. Figure 2 , Figure 2 FIG. 1 is a flow chart of a preferred embodiment of a vehicle damage assessment method of the present invention. Figure 2As shown, it includes:

[0037] Step S100: Acquire a plurality of vehicle damage images to be processed.

[0038] The vehicle damage assessment method in the embodiment of the present invention is applied to a server that analyzes vehicle damage images. The server can generate vehicle damage analysis results based on the vehicle damage images uploaded by the client.

[0039] The vehicle damage assessment method of the embodiment of the present invention can be used for vehicle insurance claims. The vehicle owner can upload the vehicle damage image through the client. In order to improve the efficiency of vehicle damage assessment, it is necessary to provide vehicle images from different angles. Including but not limited to the close-up image of the damaged parts of the vehicle and the long-range image of the whole vehicle.

[0040] Among them, Figure 3 As shown, in step S100, that is, before obtaining a plurality of vehicle damage images to be processed, the following steps are also included:

[0041] Step S11, pre-build a Transformer-based multimodal large model;

[0042] Step S12: Pre-set the prompts for the multimodal large model.

[0043] The embodiment of the present invention uses a multimodal large model to identify several vehicle damage images, so it is necessary to pre-build a multimodal large model. The modal large model is an artificial intelligence model that can process and understand multiple types of data (such as text, images, videos, audio, etc.). The multimodal large model refers to a model that combines multimodal information such as text, images, videos, audio, etc. for training. The multimodal large model can combine different perception channels and data structures to more comprehensively understand its environment and establish connections between multiple modalities, thereby improving the model's understanding and generalization capabilities; it can learn from a large amount of unlabeled data to improve the model's generalization capabilities.

[0044] Large MultiModal Model (LMM) is a model developed based on Large Language Model (LLM) and has the ability to receive, reason and output multimodal information. Large MultiModal Model (LMM) refers to a deep learning model that can process multiple different modal inputs (such as text, images, audio, etc.).

[0045] The multimodal large model can simultaneously process and analyze data from different modalities, such as text, images, audio, etc.; it can learn the associations between different modes through deep learning networks to achieve cross-modal understanding and generation; it demonstrates excellent performance in understanding and generating text, and also shows great potential in visual perception tasks.

[0046] A multimodal large model LMM usually consists of the following components:

[0047] Modality encoder: responsible for converting inputs of different modalities into a unified feature representation. There are multiple pre-trained encoder options for different modalities, such as image encoders (NFNet-F6, ViT, etc.), video encoders (CFormer, HuBERT, etc.), and 3D point cloud modality encoders (ULIP-2, PointBERT, etc.).

[0048] Large Language Model (LLM) backbone: processes these features and generates output. LLM, as the core agent, has salient properties such as zero-shot generalization, few-shot ICL, Chain of Thoughts (CoT), and instruction following.

[0049] Modality interface: responsible for the alignment and conversion between different modalities. This usually involves input and output projectors, which align features of different modalities with the text feature space and produce features as hints to be input into the LLM backbone.

[0050] The multimodal large model in the embodiment of the present invention is constructed based on Transformer. The network structure of Transformer mainly includes an input part, an encoder part, a decoder part and an output part. The structures of each part are as follows:

[0051] The input part includes the source text embedding layer and its position encoder and the target text embedding layer and its position encoder. Embedding layer: Converts symbols such as text and speech into vectors for representation. Words are not single and unrelated, and the correlation between words needs to be considered. Therefore, the embedding layer needs to consider their distance in space when mapping. The closer the distance, the higher the correlation. Position encoder: Generates a position vector for each position in the input sequence so that the model can understand the position information in the sequence. .

[0052] The encoder part is composed of N encoder layers stacked together, each of which consists of two sub-layer connection structures. The first sub-layer connection structure includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection. The second sub-layer connection structure includes a feed-forward fully connected sub-layer, a normalization layer, and a residual connection.

[0053] The decoder part is composed of N decoder layers stacked together, each of which consists of three sub-layer connection structures. The first sub-layer connection structure: includes a masked multi-head self-attention sub-layer and a normalization layer and a residual connection. The second sub-layer connection structure: includes a multi-head attention sub-layer (encoder to decoder) and a normalization layer and a residual connection. The third sub-layer connection structure: includes a feed-forward fully connected sub-layer and a normalization layer and a residual connection.

[0054] The output part includes the linear layer, the Softmax layer, and the output layer (linear layer + Softmax). Linear layer: converts the vector output by the decoder into the final output dimension. Softmax layer: converts the output of the linear layer into a probability distribution for the final prediction.

[0055] Transformer's mechanisms include self-attention mechanism and multi-head attention mechanism, of which self-attention mechanism is the core part of Transformer, which allows the model to compare each element in the input sequence with other elements when processing the sequence, so as to correctly process each element in different contexts. The multi-head attention mechanism learns different context representations by applying the self-attention mechanism to multiple different query matrices Q, key matrices K, and value matrices V.

[0056] Some models were selected from the multimodal large model for evaluation without fine-tuning. To ensure the efficiency of model reasoning, models with less than 10B parameters were selected for evaluation. The models involved in the evaluation included MiniCPM-V-2.6, InternVL2-8B, and GLM-4v-9B. The evaluation used the average classification accuracy rate to focus only on the model's judgment of the vehicle model, whether it was damaged, the type of damage, and the damaged parts. Finally, InternVL2-8B was selected for subsequent operations. Among them, the InternVL2-8B model is a multimodal large language model (MLLM) with strong multimodal understanding capabilities. The model combines visual and language information and can perform complex cross-modal tasks such as image-text matching and image description generation. By integrating visual features and language information, InternVL2-8B has achieved outstanding performance in the multimodal field. InternVL2-8B can process and understand multiple types of data inputs, such as images and text, and achieves a deep fusion of vision and language. The model demonstrates excellent multimodal understanding capabilities and performs well in various benchmarks, comparable to commercial closed-source models. The InternVL2-8B model is moderately sized and can run on consumer-grade graphics cards with 24G video memory, making it easy to deploy and apply in real-world scenarios.

[0057] The average classification accuracy is an important indicator for evaluating the performance of classification models. It reflects the average percentage of correctly classified samples in all categories. This indicator helps to more comprehensively understand the performance of the model in different categories, thereby judging the stability and generalization ability of the model.

[0058] Specifically, the calculation formula for average classification accuracy is usually to add up the classification accuracy of all categories and then divide it by the total number of categories. Classification accuracy refers to the ratio of the number of correctly classified samples in a certain category to the total number of samples in that category. Therefore, the average classification accuracy is the arithmetic mean of the classification accuracy of all categories.

[0059] In practical applications, a higher average classification accuracy usually means a better performance of the model because it can more accurately identify samples of different categories. However, we also need to combine other evaluation indicators (such as overall classification accuracy, Kappa coefficient, precision, recall, etc.) to comprehensively evaluate the performance of the model in order to obtain a more comprehensive understanding.

[0060] In addition to inputting multiple images, multimodal large models can also give prompts, i.e., prompts, to ensure the effectiveness and accuracy of the model's predicted output. A prompt is a piece of text, a sentence, or a question that is used to guide the AI ​​model to generate text or perform a specific task. It is the starting point for the interaction between the user and the model, and is used to clarify the user's needs and intentions.

[0061] The main function of Prompt is to guide the model to perform text generation or other natural language processing tasks. The model will understand the user's requirements based on the prompt and generate the corresponding text as a response. For example, if you want the model to translate a text into another language, you can use the following prompt: "Please translate the following English text into French." In this prompt, the task (translation) and the language to be translated (English to French) are clearly stated.

[0062] The types of prompts include: Task-oriented: explicitly ask the model to perform specific tasks, such as translation, answering questions, generating articles, etc. Task-oriented prompts usually contain clear instructions. Open-ended: allow the model to generate text freely, for example: "Please write an article about summer." This prompt does not clearly specify a specific task, and the model needs to decide what to generate. Contains contextual information: In multi-round dialogues or complex tasks, prompts can contain contextual information to ensure that the model understands the background and conditions of the problem.

[0063] Among them, Figure 4 As shown, step S12, i.e., setting the prompt of the multimodal large model in advance, includes:

[0064] Step S121, pre-acquiring vehicle damage analysis information input by a user;

[0065] Step S122: setting a vehicle damage result prompt of the input image of the multimodal large model according to the vehicle damage analysis information input by the user, wherein the vehicle damage result prompt includes vehicle model, damage type, damage degree, damaged parts, repair plan and claim amount.

[0066] Obtain vehicle damage analysis information input by the user and provide prompts for the model. The specific prompt setting method follows the CO-STAR principle and is designed in combination with actual needs as follows:

[0067] Please determine the vehicle model and whether it is damaged based on all the pictures uploaded by the user. First, give the vehicle model, then determine whether the vehicle is damaged. If the vehicle is damaged, please further determine the type of damage, degree of damage, damaged parts of the vehicle, and provide a repair plan and claim amount. First, give the analysis process, and then return the results strictly in the following JSON format:

[0068] {

[0069] {Model: Brand 1 plus Model 1 / Brand 2 plus Model 2 / XXX},

[0070] {Is it damaged: YES / NO},

[0071] {Damage type: dent / paint peeling / bend / break / XXX},

[0072] {Damage level: Level 1 / Level 2 / Level 3 / Level 4 / Level 5 / Level 6},

[0073] {Damaged parts: fender / door / headlight / bumper / tire / rim / rearview mirror / taillight / door handle / door glass / license plate / windshield / wiper arm / wiper blade / trunk lid / XXX},

[0074] {Maintenance plan: sheet metal / painting / welding / XXX},

[0075] {Claim amount: 1~2k / 2~3k / 3~4k / XXX}

[0076] }

[0077] Among them, the CO-STAR principle is one of the structured prompt words, which aims to help users construct more complete and effective prompt words when using large language models, thereby improving the relevance and effectiveness of AI-generated content.

[0078] The CO-STAR principle consists of six key elements, each of which is represented by the first letter of the acronym CO-STAR, as follows:

[0079] C-Context (clear background information): Provide sufficient background information to help AI understand the context and environment of the task. The background information can include the location, time, people involved, causes and consequences of the event, etc.

[0080] O-Objective (clear goal setting): clearly state the specific goals or tasks you want the AI ​​to accomplish. Goals should be specific, clear, and measurable.

[0081] S-Style (Style of generated text): Specifies the style of AI-generated content, such as formal, humorous, etc. The choice of style should match the preferences of the target audience and the purpose of the content.

[0082] T-Tone (Tone of Generated Text): Determines the tone of AI-generated content, such as politeness, persuasiveness, motivational, etc. The choice of tone should be coordinated with the purpose of the content and the emotional state of the target audience.

[0083] A-Audience (target audience description): Describe the characteristics of the target audience, such as age, interests, occupation, etc. Understanding the audience can help AI use appropriate language, cite relevant examples, and adjust the complexity of the content.

[0084] R-Response (Type of Response): Specifies the format of the AI ​​response, such as table, paragraph, list, etc., as well as the specific requirements of the response. By clarifying the requirements of the response, it can be ensured that the content generated by AI not only meets the requirements in terms of content, but also meets the expectations in terms of form.

[0085] Among them, JSON (JavaScript Object Notation) is a lightweight data exchange format. It is based on a subset of ECMAScript (js specification developed by the European Computer Association) and uses a text format that is completely independent of the language to store and represent data. The JSON format is easy for people to read and write, and it is also easy for machines to parse and generate.

[0086] Among them, Figure 5 As shown, step S12, i.e., after the prompt of the multimodal large model is set in advance, it also includes:

[0087] Step S21, obtaining a data set of vehicle damage images;

[0088] Step S22: training the multimodal large model based on the data set and the prompt, and obtaining a vehicle damage recognition model when the training is completed.

[0089] Get a dataset of vehicle damage images. Too many images input will cause a serious computational burden. In addition, the data processing capacity of existing multimodal large models is limited, and the more images input, the better the effect. Therefore, in the process of data set collection, obtain existing vehicle damage assessment cases, and only retain about 4 images for each case, and try to ensure that each case contains 2 close-up images of damaged parts and 2 long-range images of the whole vehicle. The corresponding output is a string in JSON format, including information about vehicle model, damage, damage type, damage degree, damaged parts, repair plan, and claim amount.

[0090] The dataset is collected into a large multimodal model, and the prompt is input into the large multimodal model. In multimodal pre-training, Prompt can be used for cross-modal retrieval and generation between text and images. Given a picture, using free-style prompts allows the model to generate a description of the picture; given a picture and a description, using fill-in-the-blank prompts allows the model to generate a complete description of the picture; given a topic and background information, using imperative prompts allows the model to generate comments on the topic. After training is completed, a vehicle damage recognition model is obtained.

[0091] During the training process, the data set and prompts are input into the multimodal large model for training. During the training, multi-task learning can be selected, that is, multiple tasks are designed to jointly train the model, such as text classification, image recognition, sentiment analysis, etc. Multi-task learning helps the model learn the intrinsic connection between different modalities and improve the overall performance.

[0092] Joint training: Input data from multiple modalities into the model at the same time for joint optimization. Make full use of the complementary information between different modalities to improve the accuracy and robustness of the model.

[0093] Optimizer and loss function: Choose an appropriate optimizer (such as Adam, SGD, etc.) and loss function (such as cross entropy loss, mean squared error loss, etc.). Monitor the performance of the model during training and make adjustments as needed.

[0094] Parameter adjustment: Adjust parameters such as learning rate and batch size according to actual conditions. Use a dynamic weight allocation strategy to balance the contribution of different modes.

[0095] Model evaluation: Use accuracy, precision, recall and other evaluation indicators to evaluate model performance. Make necessary model adjustments and optimizations based on the evaluation results.

[0096] Among them, Figure 6 As shown, step S22, i.e. training the multimodal large model based on the data set and the prompt, and obtaining a vehicle damage recognition model when the training is completed, includes:

[0097] Step S221, training the multimodal large model based on the data set and the prompt;

[0098] Step S222: fine-tuning the optimizer parameters and gradient parameters of the multimodal large model based on the LoRA algorithm during the training process;

[0099] Step S223: Obtain the fine-tuned optimizer parameters and gradient parameters, and store the optimizer parameters and gradient parameters in a distributed manner;

[0100] Step S224: until the multimodal large model meets the preset convergence condition, the training is completed and a vehicle damage recognition model is obtained.

[0101] The training process uses LoRA to perform efficient fine-tuning on the collected data, and the optimizer parameters and gradient parameters after fine-tuning are distributed and stored.

[0102] Among them, LoRA (Low-Rank Adaptation) is a parameter-efficient method for fine-tuning large language models (LLMs). The core idea of ​​this method is to update a small part of the parameters of the pre-trained model by introducing a low-rank matrix, thereby achieving adaptation to specific tasks or fields without fine-tuning all parameters of the entire model.

[0103] The LoRA method works by inserting two trainable low-rank matrices (usually matrices A and B) into each layer of the pre-trained model. The dimensions of these low-rank matrices are much smaller than the original weight matrices, so they have a relatively small number of parameters. During training, the LoRA method only updates the parameters of these low-rank matrices, while keeping most of the parameters of the pre-trained model unchanged. In this way, LoRA is able to significantly reduce training time and computing resource requirements without sacrificing too much performance. After training is completed, the LoRA method fuses the parameters of the low-rank matrices back into the original weight matrices through specific mathematical operations (although in practical applications, this step is usually not required because the model containing the low-rank matrix can be used directly for inference).

[0104] Since LoRA only updates a small number of parameters, it greatly reduces the computational resources and time required for fine-tuning. Due to the small number of parameters of LoRA, it is easier to deploy and reason on resource-constrained devices. LoRA can be easily applied to different pre-trained models and tasks without major modifications to the model structure.

[0105] In order to reduce the usage of video memory, the embodiment of the present invention uses ZeRO2 for distributed storage of optimizer parameters and gradient parameters. Zero Redundancy Optimizer Level 2 is a memory optimization technology proposed by Microsoft for deep learning model training, which is part of the ZeRO optimizer technology. Based on ZeRO-1, ZeRO2 further implements gradient partitioning to more effectively reduce memory usage and improve the efficiency of model training.

[0106] In terms of distributed storage, ZeRO2 achieves memory optimization in the following ways:

[0107] Optimizer state partitioning: Like ZeRO-1, ZeRO2 also distributes the optimizer state across devices. This means that each device no longer needs to store the entire optimizer state, but only a portion of it, significantly reducing memory usage.

[0108] Gradient partitioning: ZeRO2 further partitions the gradient storage. In a distributed training environment, each device only needs to process and store the gradient information related to it, rather than the gradient of the entire model. In this way, the memory burden of each device is further reduced, making it possible to train larger models on the same hardware, or train the same model on smaller hardware.

[0109] Through ZeRO2's distributed storage strategy, memory usage during deep learning model training has been greatly optimized. This not only improves training efficiency, but also makes larger-scale model training possible. At the same time, by reducing memory redundancy, ZeRO2 also helps reduce training costs and improve resource utilization.

[0110] Among them, Figure 7 As shown, step S22, i.e. training the multimodal large model based on the data set and the prompt, and obtaining the vehicle damage recognition model after the training is completed, comprises:

[0111] Step S31: performing deep learning model quantization processing on the vehicle damage identification model.

[0112] To speed up the inference process, the model is quantized using INT4, and the quantization method is AWQ (Activation-aware Weight Quantization).

[0113] Among them, INT4 quantization is a deep learning model quantization technology that converts parameters such as weights and activation values ​​in the model from floating point numbers to 4-bit integers. The principle of INT4 quantization is achieved by mapping floating point parameters to the range of 4-bit integers. Specifically, it converts continuous floating point values ​​into discrete integer values, thereby greatly reducing the storage requirements and computational complexity of the model. In INT4 quantization, parameters are usually quantized to integers between -8 and 7 (signed) or 0 and 15 (unsigned).

[0114] Since INT4 quantization converts parameters into 4-bit integers, it can significantly reduce the storage space and computational complexity of the model. This makes the deployment of the model on mobile devices or embedded systems more efficient. Since the quantized model has fewer parameters and lower computational complexity, the inference speed of the model can be increased. The INT4 quantized model consumes less power during inference and is suitable for scenarios with strict requirements on power consumption.

[0115] AWQ (Activation-aware Weight Quantization) is an activation-aware weight quantization method. The core idea of ​​AWQ is that not all weights are equally important, and only a small number of salient weights have a greater impact on the output accuracy of the model. Therefore, if we can only protect the accuracy of these salient weights and perform low-bit quantization on other weights, we can significantly reduce the model memory usage and improve the inference speed while keeping the accuracy almost unchanged.

[0116] The AWQ method selects significant weights based on the distribution of activation values. Specifically, it calculates the activation value of each weight channel and selects the channel with the largest absolute value of activation value as the significant channel. The weights of these significant channels will maintain high precision (such as FP16) during quantization, while the weights of other channels will be quantized with low bits (such as INT4).

[0117] To further reduce the quantization error, the AWQ method introduces a scaling factor. For significant weights, multiplying by a larger scaling factor can reduce their relative quantization error; while for non-significant weights, multiplying by a smaller scaling factor. This can improve quantization efficiency while maintaining model accuracy.

[0118] The AWQ method also implements online dequantization without reordering. This means that the dequantization operation can be performed directly on the quantized weights without reordering the weights. This helps improve quantization performance and reduce inference overhead.

[0119] The AWQ method does not rely on any back-propagation or reconstruction process, so it can well maintain the generalization ability of large language models (LLMs) across different domains and modes. Although the AWQ method introduces a scaling factor to protect significant weights, it avoids the hardware-inefficient mixed-precision implementation in a clever way. Specifically, it uses a channel-by-channel scaling method to automatically search for the optimal scaling factor that minimizes the quantization error under all weights. This method not only improves quantization accuracy but also maintains hardware efficiency. Since the AWQ method can significantly reduce the storage requirements and computational complexity of the model, it can increase the inference speed of the model. The AWQ method outperforms existing work in multiple language modeling, commonsense question answering, and domain-specific benchmarks.

[0120] Step S32: deploying the quantified vehicle damage identification model on the vehicle damage identification platform.

[0121] To further improve the inference speed, use LmDeploy for deployment. LmDeploy provides comprehensive lightweight, deployment, and service solutions for multimodal large models. LMDeploy supports efficient inference of multiple quantization methods and quantized models. Quantization can reduce the model size and increase the inference speed while maintaining good accuracy. You can use the KV Cache quantization, AWQ quantization solution and other technologies provided by LMDeploy for performance optimization. You can also adjust parameters such as model configuration and parallel number according to actual needs to obtain better inference performance.

[0122] Among them, Figure 8 As shown, step S100, i.e. obtaining a plurality of vehicle damage images to be processed, includes:

[0123] Step S101, obtaining at least two close-up images of damaged parts of the vehicle to be processed.

[0124] Step S102: Obtain at least two vehicle perspective images of the vehicle to be processed.

[0125] Step S103: obtaining a plurality of vehicle damage images to be processed based on the damaged component close-up image and the whole vehicle long-range image.

[0126] In order to improve the recognition efficiency of vehicle damage images, users need to upload at least 2 close-up images of damaged parts and at least 2 long-range images of the whole vehicle. Damaged parts include but are not limited to the vehicle's fenders, doors, headlights, bumpers, tires, steel rims, rearview mirrors, taillights, door handles, door glass, license plates, front windshields, wiper arms, wiper blades, luggage compartment lids and other parts. Users upload images to the auto insurance platform, and the platform prompts users to enter at least 2 close-up images of damaged parts and 2 long-range images. If the number of close-up images of damaged parts is less than 2 or the number of long-range images of the whole vehicle is less than 2, the user will be prompted to re-enter.

[0127] Step S200: input the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multimodal model based on Transformer.

[0128] Several vehicle damage images are input into the vehicle damage recognition model, and a convolutional neural network (CNN) is used to process the image data. Appropriate preprocessing is performed on the images, such as scaling, cropping, and normalization.

[0129] Perform modal alignment, which includes: timestamp-based alignment: applicable to time series data, such as audio and image frames in videos. Spatial position-based alignment: applicable to data with clear spatial position information, such as text and objects in images.

[0130] Then, modal fusion is performed, including splicing, weighted averaging and attention mechanism. Splicing refers to directly splicing the feature vectors of different modalities together. Weighted averaging refers to weighted averaging of feature vectors of different modalities. Attention mechanism refers to weighted fusion of features of different modalities through attention mechanism to highlight important information. After fusion, the corresponding vehicle damage recognition result is decoded by decoder.

[0131] Step S300: Obtain vehicle damage assessment results based on the output of the vehicle damage identification model.

[0132] The output of the vehicle damage identification model is a string in JSON format, which includes but is not limited to the vehicle model, damage status, damage type, damage degree, damaged parts, repair plan, and claim amount. The staff can determine the vehicle damage results based on the output of the vehicle damage identification model. The damage status refers to whether the vehicle is damaged.

[0133] Compared with the prior art, the vehicle damage assessment method of the embodiment of the present invention has the following technical advantages:

[0134] The Transformer-based multimodal large model performs image recognition. Thanks to the unique structure of the Transformer, the multimodal large model can accept multiple image inputs and obtain a unified output, thus getting rid of tedious post-processing operations. In addition, the information of each image can be interacted in the attention module so that the information can be fully utilized and integrated.

[0135] It can accept multiple image inputs, and the information of multiple images can interact with each other, which greatly improves the utilization rate of information;

[0136] The final prediction is directly output without any other post-processing, which greatly improves the efficiency of damage assessment.

[0137] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0138] Another embodiment of the present invention provides a vehicle damage assessment device, which corresponds one-to-one to the vehicle damage assessment method of the above embodiment. Fig. 9 As shown, the device 1 comprises:

[0139] An image acquisition module 100 is used to acquire a plurality of vehicle damage images to be processed;

[0140] A data input module 200, used to input the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multi-modal model based on Transformer;

[0141] The result acquisition module 300 is used to obtain the vehicle damage assessment result according to the output of the vehicle damage identification model.

[0142] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0143] In one embodiment, the image acquisition module 100 is specifically used for:

[0144] Obtain at least 2 close-up photos of the damaged parts of the vehicle to be processed;

[0145] Obtain at least 2 vehicle perspective images of the vehicle to be processed;

[0146] Based on the damaged component close-up image and the whole vehicle long-range image, a plurality of vehicle damage images to be processed are obtained.

[0147] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0148] In one embodiment, the device further comprises a model building module, the model building module being configured to:

[0149] Pre-build a large multi-modal model based on Transformer;

[0150] The prompts for the multimodal large model are set in advance.

[0151] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0152] In one embodiment, the device further comprises a model training module, the model training module being configured to:

[0153] Obtain a dataset of vehicle damage images;

[0154] The multimodal large model is trained based on the data set and the prompt, and a vehicle damage recognition model is obtained when the training is completed.

[0155] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0156] In one embodiment, the model building module is further configured to:

[0157] Pre-acquire vehicle damage analysis information input by the user;

[0158] The vehicle damage result prompt of the input image of the multimodal large model is set according to the vehicle damage analysis information input by the user, and the vehicle damage result prompt includes vehicle model, damage type, damage degree, damaged parts, repair plan and claim amount.

[0159] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0160] In one embodiment, the model training module is further used to:

[0161] Training the multimodal large model based on the data set and the prompt;

[0162] During the training process, the optimizer parameters and gradient parameters of the multimodal large model are fine-tuned based on the LoRA algorithm;

[0163] Obtain the fine-tuned optimizer parameters and gradient parameters, and store them in a distributed manner;

[0164] The training is completed until the multimodal large model meets the preset convergence conditions, and a vehicle damage recognition model is obtained.

[0165] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0166] In one embodiment, the vehicle damage assessment device of the embodiment of the present invention is applied to a vehicle damage identification platform, and the device further includes a model deployment module, which is used to:

[0167] Performing deep learning model quantization processing on the vehicle damage identification model;

[0168] The vehicle damage identification model after quantification processing is deployed on the vehicle damage identification platform.

[0169] The specific implementation method is shown in the method embodiment, which will not be described in detail here.

[0170] The present invention provides a vehicle damage assessment device, which performs image recognition based on a large multimodal model of Transformer, and can accept multi-image input. The multi-image information can interact, and the information utilization rate is greatly improved. The method directly outputs the final prediction without the need for other post-processing processes, and the damage assessment efficiency is greatly improved.

[0171] Another embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as shown in Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements the functions or steps on the service side of a vehicle damage assessment method.

[0172] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a vehicle damage assessment method.

[0173] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0174] Acquire several vehicle damage images to be processed;

[0175] Inputting the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multimodal model based on Transformer;

[0176] The vehicle damage assessment result is obtained based on the output of the vehicle damage identification model.

[0177] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0178] Acquire several vehicle damage images to be processed;

[0179] Inputting the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multimodal model based on Transformer;

[0180] The vehicle damage assessment result is obtained based on the output of the vehicle damage identification model.

[0181] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0183] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0185] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

[0186] Conditional language such as "can," "can," "might," or "may," among other things, unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey that particular embodiments can include (while other embodiments do not) particular features, elements, and / or operations. Thus, such conditional language is also generally intended to imply that features, elements, and / or operations are anyway required for one or more embodiments or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0187] What has been described herein in this specification and the accompanying drawings includes examples that can provide vehicle damage assessment methods and devices. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it can be recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure. In addition, or in the alternative, other embodiments of the present disclosure may be apparent from consideration of this specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples set forth in this specification and the accompanying drawings be considered in all respects to be illustrative and not restrictive. Although specific terms are employed herein, they are used in a general and descriptive sense and are not used for limiting purposes.

Claims

1. A vehicle damage assessment method, characterized in that , the method comprises: Acquire several vehicle damage images to be processed; Inputting the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multimodal model based on Transformer; The vehicle damage assessment result is obtained based on the output of the vehicle damage identification model.

2. The vehicle damage assessment method according to claim 1, characterized in that: The step of obtaining a plurality of vehicle damage images to be processed includes: Obtain at least 2 close-up photos of the damaged parts of the vehicle to be processed; Obtain at least 2 vehicle perspective images of the vehicle to be processed; Based on the damaged component close-up image and the whole vehicle long-range image, a plurality of vehicle damage images to be processed are obtained.

3. The vehicle damage assessment method according to claim 1, characterized in that: Before obtaining the plurality of vehicle damage images to be processed, the method further includes: Pre-build a large multi-modal model based on Transformer; The prompts for the multimodal large model are set in advance.

4. The vehicle damage assessment method according to claim 3, characterized in that: After the prompt of the multimodal large model is set in advance, it also includes: Obtain a dataset of vehicle damage images; The multimodal large model is trained based on the data set and the prompt, and a vehicle damage recognition model is obtained when the training is completed.

5. The vehicle damage assessment method according to claim 3, characterized in that: The step of presetting the prompt of the multimodal large model includes: Pre-acquire vehicle damage analysis information input by the user; The vehicle damage result prompt of the input image of the multimodal large model is set according to the vehicle damage analysis information input by the user, and the vehicle damage result prompt includes vehicle model, damage type, damage degree, damaged parts, repair plan and claim amount.

6. The vehicle damage assessment method according to claim 4, characterized in that: The multimodal large model is trained based on the data set and the prompt, and a vehicle damage recognition model is obtained when the training is completed, including: Training the multimodal large model based on the data set and the prompt; During the training process, the optimizer parameters and gradient parameters of the multimodal large model are fine-tuned based on the LoRA algorithm; Obtain the fine-tuned optimizer parameters and gradient parameters, and store them in a distributed manner; The training is completed until the multimodal large model meets the preset convergence conditions, and a vehicle damage recognition model is obtained.

7. The vehicle damage assessment method according to claim 4, characterized in that: Applied to the vehicle damage identification platform, the multimodal large model is trained based on the data set and the prompt, and after the training is completed, the vehicle damage identification model is obtained, including: Performing deep learning model quantization processing on the vehicle damage identification model; The vehicle damage identification model after quantification processing is deployed on the vehicle damage identification platform.

8. A vehicle damage assessment device, characterized in that: The device comprises: An image acquisition module, used to acquire a number of vehicle damage images to be processed; A data input module, used for inputting the plurality of vehicle damage images into a vehicle damage recognition model, wherein the vehicle damage recognition model is a large multi-modal model based on Transformer; The result acquisition module is used to obtain the vehicle damage assessment result based on the output of the vehicle damage identification model.

9. A computer device, characterized in that: The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the vehicle damage assessment method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the steps of the vehicle damage assessment method described in any one of claims 1-7.

Images