Vehicle loss assessment method and system

Abstract

The invention discloses a vehicle loss assessment method and system, and relates to the technical field of automobile information. A specific embodiment of the method comprises the following steps: receiving natural language information describing a vehicle damage condition; based on the natural language information and a preset cue word, calling a large language model for processing, and generating structured data including at least one loss assessment item; based on the structured data, generating operation guidance information corresponding to the loss assessment project and outputting the operation guidance information to the user; and obtaining a media file related to the vehicle damage, and associating the media file with the loss assessment item to generate a loss assessment result. According to the embodiment, intelligent assistance can be performed on the loss assessment process, so that the vehicle loss assessment efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of automotive information technology, and in particular to a method and system for assessing vehicle damage. Background Technology

[0002] With the continuous growth of car ownership, the volume of vehicle insurance claims is increasing rapidly. In the vehicle accident loss assessment (damage assessment) stage, the damage assessor needs to go to the scene in person, rely on personal experience to judge the scope of the loss, determine the repair plan, and manually take a large number of photos. Afterwards, the photos need to be manually linked with the damage assessment items and entered into the system.

[0003] However, traditional manual damage assessment methods have the following drawbacks: When faced with complex vehicle models, novel structures, or hidden damage, inexperienced assessors are prone to misjudgments, leading to unreasonable repair plans or a lack of bargaining power when negotiating with repair shops, thus increasing insurance claims costs. Therefore, the assessor's skill level directly affects the accuracy of the judgment. Furthermore, the lack of standardized operating guidelines for collecting on-site evidence (such as photos and fault codes) frequently results in incomplete photo angles, omissions of key damaged areas, and oversights of electronic diagnostic information, thus prolonging the claims process and increasing operating costs. Additionally, manually correlating dozens or even hundreds of on-site photos with the damage assessment items is time-consuming, labor-intensive, and prone to errors. Therefore, traditional manual damage assessment methods cannot meet the industry's requirements for efficiency, accuracy, and risk control. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a vehicle damage assessment method and system that can intelligently assist the damage assessment process, thereby improving the efficiency of vehicle damage assessment.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for assessing vehicle damage is provided, comprising:

[0006] Receive natural language information describing the vehicle damage;

[0007] Based on natural language information and preset prompts, a large language model is invoked for processing to generate structured data containing at least one damage assessment item.

[0008] Based on structured data, generate operational guidance information corresponding to the damage assessment items and output it to the user;

[0009] Obtain media files related to vehicle damage, associate these media files with damage assessment items, and generate damage assessment results.

[0010] Preferably, based on natural language information and preset prompt words, a large language model is invoked for processing, including:

[0011] Convert natural language information into text information;

[0012] Based on the format defined by the preset prompt words, the text information is combined to form the model input information;

[0013] The model input information is fed into the large language model, where preset prompt words are configured to guide the large language model to output structured data in a predetermined format based on the text information.

[0014] Preferably, the preset prompts include at least one of the following for guiding the large language model: role definition information, task objective description, output format specification, and standardized mapping rules for colloquial expressions in the input text.

[0015] Preferably, based on structured data, operational guidance information corresponding to the loss assessment item is generated, including:

[0016] Based on the damage assessment items in the structured data, determine the corresponding parts information and / or vehicle information;

[0017] Based on parts information and / or vehicle information, retrieve the corresponding operation guidance information from the pre-defined standard operating procedure knowledge base.

[0018] Preferably, the operation guidance information includes recommended shooting instructions for the specified accessories.

[0019] Preferably, the media files are collected based on the operation guide information.

[0020] Preferably, associating media files with damage assessment items includes:

[0021] Categorize media files and filter out images that belong to the vehicle photo category;

[0022] For vehicle photos, the vehicle accessories are identified using a trained multimodal recognition model.

[0023] The identified vehicle parts are matched with the damage assessment items to establish a correlation.

[0024] Preferably, the multimodal recognition model is a model trained based on supervised fine-tuning techniques.

[0025] Preferably, the training of the multimodal recognition model includes:

[0026] Obtain a sample image set of frequently used parts for vehicle damage assessment;

[0027] The sample images in the sample image set are labeled to obtain training data;

[0028] Supervised fine-tuning of the multimodal recognition model is performed using training data.

[0029] Preferably, labeling the sample image includes: labeling whether a specific accessory exists in the sample image, and labeling the location information of the specific accessory in the sample image.

[0030] Preferably, classifying media files includes: identifying and classifying them using a deep learning model based on a visual Transformer.

[0031] Preferably, before classifying the media files, the method further includes performing similarity deduplication on the media files based on feature vectors.

[0032] Preferably, identifying vehicle parts through a trained multimodal recognition model includes: outputting the location information of the vehicle parts in the image, whereby the location information is represented in the form of a bounding box for the parts;

[0033] After associating media files with damage assessment items, the process also includes: scoring multiple images associated with the same vehicle part based on the area ratio of the part's target box in each image, and sorting and outputting the scores.

[0034] Preferably, after generating the operation guidance information, the method further includes:

[0035] Based on the damage assessment items in the structured data, a pre-set parts database is queried to generate the first damage assessment data containing price information;

[0036] The image of the damage assessment report provided by the third party is obtained, and the image is structured by a multimodal visual language model to obtain the second damage assessment data. The second damage assessment data includes the parts information and price information identified from the image.

[0037] Risk items are identified by comparing the first and second loss assessment data.

[0038] Preferably, the first damage assessment data and the second damage assessment data each further include part codes and / or repair method information, and the repair method information in the first damage assessment data is obtained through operation guidance information.

[0039] Preferably, the structured recognition of images using a multimodal visual language model includes:

[0040] The image and preset cue word constraints are input into the model, where the cue word constraints are used to guide the model to extract structured information from predefined fields.

[0041] Preferably, risk items are determined by comparing the first loss assessment data with the second loss assessment data, including:

[0042] The first loss assessment data is compared with the second loss assessment data to identify discrepancies;

[0043] Based on preset business rules, at least some of the discrepancies will be identified as risk items.

[0044] Preferably, the business rules include at least one of the following:

[0045] Parts coding consistency verification rules;

[0046] Parts price deviation threshold rules;

[0047] Matching rules between repair methods and damage levels.

[0048] According to another aspect of the present invention, a vehicle damage assessment system is provided, comprising:

[0049] The receiving unit is used to receive natural language information describing the vehicle damage.

[0050] The structured data generation unit is used to process natural language information and preset prompt words by calling a large language model to generate structured data containing at least one loss assessment item.

[0051] The operation guidance information generation unit is used to generate operation guidance information corresponding to the loss assessment item based on structured data and output it to the user; and

[0052] The association unit is used to obtain media files related to vehicle damage, associate the media files with the damage assessment items, and generate damage assessment results.

[0053] According to another aspect of the present invention, a vehicle damage assessment electronic device is provided, comprising:

[0054] One or more processors; and

[0055] Storage device for storing one or more programs.

[0056] When one or more programs are executed by one or more processors, the one or more processors implement the methods described above in the embodiments of the present invention.

[0057] According to another aspect of the present invention, a computer program is provided that, when executed by a processor, implements the methods described above in the embodiments of the present invention.

[0058] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described above in the embodiments of the present invention.

[0059] One embodiment of the above invention has the following advantages or beneficial effects: by recognizing natural language through a large language model, generating operational guidance information corresponding to the damage assessment items, and associating media files with the damage assessment items, it is possible to provide intelligent assistance to the damage assessment process, reduce the time for manual input, query, and comparison, speed up the damage assessment process, and thus improve the efficiency of vehicle damage assessment.

[0060] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0061] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0062] Figure 1 This is a schematic diagram of the main process of a vehicle damage assessment method according to an embodiment of the present invention;

[0063] Figure 2 This is an interactive schematic diagram of generating a damage assessment preview form using natural language according to an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of the interface guiding the standard operation process of damage assessment according to an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the classification and association operation of damage assessment images according to an embodiment of the present invention;

[0066] Figure 5 This is a schematic diagram of the damage assessment process according to an embodiment of the present invention;

[0067] Figure 6 This is a schematic diagram of the main modules of a vehicle damage assessment system according to an embodiment of the present invention;

[0068] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0069] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0070] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0071] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0072] To collect data on the usage of our products / services, we will aggregate, analyze, and utilize technically processed user data, and share the processed statistical information with third parties. We will use secure encryption techniques and other methods to ensure that information recipients cannot re-identify specific individuals.

[0073] Figure 1 This is a schematic diagram illustrating the main flow of a vehicle damage assessment method according to an embodiment of the present invention. Figure 1 As shown, the vehicle damage assessment method includes steps S101 to S104.

[0074] like Figure 1 As shown, in step S101, natural language information describing the vehicle damage is received. Natural language information is, for example, the language content of a user (such as a damage assessor). For example, the receiving operation of this natural language information can support three modes, namely, (1) natural speech to generate damage assessment form mode, receiving the vehicle damage situation verbally described by the damage assessor (once), (2) natural chat record mode, continuously listening to the voice content of the damage assessor and the repair shop personnel during the communication process, and (3) speaking and taking pictures mode, receiving the voice information such as the name of the parts and the repair method verbally described by the damage assessor during the taking of pictures. The system can parse the above-mentioned natural language information received in real time to generate a damage assessment preview form (structured data) as described below.

[0075] like Figure 1 As shown, in step S102, based on natural language information and preset prompts, a large language model is invoked for processing to generate structured data containing at least one damage assessment item. The structured data may be, for example, a damage assessment preview form.

[0076] Specifically, the received natural language information is converted into text information. Based on a preset prompt, a large language model is called to perform semantic understanding on the natural language (converted text information) of, for example, a loss adjuster, identify the part name, damage type, and repair method therein, and output data in a structured format such as JSON according to a preset data format for subsequent loss adjustment processing. The large language model (general large language model) has the ability to understand human natural language and can be pre-built with a basic knowledge system in the field of vehicle repair. The general large language model is, for example, qwen3-max (Qianwen large language model), doubao-1-5-pro-32k-250115 (Doubao large language model), deepseek-v3.2 (DeepSeek large language model), etc. Through prompt engineering, the model is guided to output structured results that meet the requirements of the loss adjustment business. The preset prompt can, without changing the parameters of the large language model, guide the inference path, behavior mode, output format, etc. of the model by systematically designing and restricting the input prompt content, so as to obtain controllable, stable, and reproducible output results in a specific loss adjustment scenario. In addition, through the guidance of the prompt, lightweight fine-tuning of the model can be replaced.

[0077] The core calculation of the large language model is based on the formula: P(y∣x)=t∏P(yt∣x,y<t). The text generation of the large language model is essentially a process of gradually generating an output token sequence by maximizing the conditional probability of the next token under the condition of a given input token sequence (i.e., the prompt and the user text), based on its pre-trained internal parameters. Therefore, the essence of prompt engineering is the artificial intervention in the model's "probability space + attention distribution + semantic activation path". The key of this invention does not lie in changing the internal parameters of the model, but in constructing an input token sequence through domain-specific prompt engineering to guide the general model to stably output a structured token sequence (such as JSON format) that meets the predetermined format in the vehicle loss adjustment scenario.

[0078] The text information converted from the natural language information can be combined with the preset prompt, and the combined information can be used as input information to be input into the large language model for processing. For example, according to the format defined by the preset prompt, the text information is combined to form the model input information; the model input information is input into the large language model, and the preset prompt is configured to guide the large language model to output structured data that meets the predetermined format based on the text information. The preset prompt, for example, includes at least one of the following for guiding the large language model: role definition information, task objective description, output format specification, and standardized mapping rules for colloquial expressions in the input text.

[0079] Specifically, prompts may include the following: role definition information, task objective description, standardized mapping rules, background and context, process guidance, constraints and rules, and output format (output format specification).

[0080] The following is an example of a prompt word design (the defined format):

[0081] #Parts information extraction prompts (part name, repair method, damage type)

[0082] PART_INFO_EXTRACTION_PROMPT = """You are an automotive repair information extraction expert. (i.e., role definition)

[0083] Task (i.e., task objective): Extract the following information from the text entered by the user:

[0084] 1. Part name (part_name) - Retain directional terms (front / rear / left / right / inner / outer, etc.)

[0085] 2. Repair Method (repair_method) - Optional values: painting, disassembly and assembly, sheet metal work, external repair, replacement

[0086] Colloquial repair method expression mapping rules:

[0087] - "Change one", "Change to a new one", "Get a new one", "Change to a new one" → Replace

[0088] - "Spray paint", "Spray one", "Spray once" → Spray paint

[0089] - "Fix", "Repair", "Repair", "Restore" → Sheet metal

[0090] 3. Damage type - Optional values: crack, deformation, scratch, fracture, damage

[0091] Colloquialism-based impairment expression mapping rules:

[0092] - "shattered", "damaged", "broken", "broken" → cracked

[0093] - "bent", "flattened", "bent", "flattened", "dented" → deformed

[0094] - "scratched", "rubbed", "wiped" → scraped

[0095] Rules (i.e., constraints and rules):

[0096] 1. If there are multiple parts, extract only the last one. If it is not explicitly mentioned in the text, return null. Output only the part name or null, without any other content. Output only the part name, without any explanation.

[0097] 2. The repair method and damage type must be selected from the optional values. If not explicitly mentioned in the text, return null.

[0098] 3. Prioritize identifying colloquial expressions and mapping them to standard terminology.

[0099] 4. Return plain JSON format, without any explanation or additional content.

[0100] Example:

[0101] - Input: "Headlight paint repair and scratch repair" → Output: {{"part_name": "Headlight", "repair_method": "Paint", "damage_type": "Scratching"}} (i.e., output format)

[0102] Now please extract information from the following text:

[0103] {text}

[0104] Output only JSON, no other content: ""

[0105] The following is for reference. Figure 2 The generation of the damage assessment preview form (structured data) is further described through specific examples.

[0106] In the scenario of assessing vehicle damage, the damage adjuster clicks the voice input button in the mobile terminal application (mobile system) and describes the vehicle damage in natural voice, for example, "The front bumper is deformed and needs to be replaced, the left front headlight is broken and also needs to be replaced, and the right front fender is scratched and needs to be repainted."

[0107] After the claims adjuster finishes voice input, the system converts the claims adjuster's voice content into text, and calls the general large language model (LLM) based on the predetermined prompt words (prompt word engineering) to perform semantic parsing on the text and automatically identify the claims adjustment intention contained therein (such as part name, damage type, repair method, etc.).

[0108] After model analysis, the system extracts the corresponding damage assessment items from the text and generates the following structured results:

[0109] Front bumper - Replacement - Deformed

[0110] Left front headlight - Replacement - Broken

[0111] Right front fender - paint - scratch

[0112] The system returns the following JSON data from the program:

[0113] {

[0114] {"part_name": "Front bumper", "repair_method": "Replacement", "damage_type": "Deformation"}

[0115] {"part_name": "Left front headlight", "repair_method": "Replacement", "damage_type": "Crack"}

[0116] {"part_name": "Right front fender", "repair_method": "paint", "damage_type": "scratching"}

[0117] }

[0118] The structured results described above are displayed to the loss assessor as a one-way preview for use in subsequent loss assessment processes.

[0119] Refer again Figure 1In step S103, based on structured data, operational guidance information corresponding to the damage assessment items is generated and output to the user. For example, after generating a damage assessment preview form, the damage assessor can verify the damage assessment items in the preview form. The operational guidance information includes recommended shooting instructions for specified parts. Specifically, based on the damage assessment items in the structured data, the corresponding part information and / or vehicle information can be determined; based on the part information and / or vehicle information, the corresponding operational guidance information can be retrieved from a predetermined standard operating procedure (SOP) knowledge base. The obtained operational guidance information can be displayed to the user (damage assessor) on, for example, a mobile terminal interface to guide the user in completing standardized damage assessment operations.

[0120] The SOP knowledge base can be pre-set to include, for example, recommended shooting guidelines for specific parts (such as standard shooting positions and shooting angle requirements), fault code collection prompts for electronic parts and corresponding damage judgment criteria, and judgment rules for overall replacement, partial replacement or repair of parts, etc.

[0121] refer to Figure 3 This describes a specific example of generating operation guidance information. Based on the generated damage assessment preview form, the SOP corresponding to the damage assessment item is retrieved from the backend database (SOP knowledge base) according to the vehicle model and part name (e.g., engine hood). The SOP content includes: the photo requirement is a 45-degree side view with a close-up; the fault code prompt is to check the collision sensor; the replacement logic is that if the deformation rate is greater than 30%, replacement is recommended. The above SOP content corresponding to the engine hood is displayed on the mobile terminal.

[0122] This step generates operational guidance information and guides users, which can effectively improve the rationality and accuracy of damage assessment, prevent unnecessary replacements or repairs, and thus reduce claims risks.

[0123] Refer again Figure 1 In step S104, media files related to vehicle damage are acquired, and these media files are associated with the damage assessment items to generate a damage assessment result. Media files may include, for example, videos or images. These media files may be collected based on the operation guidance information generated in step S103 above. Media files may also include files collected by the user without following the operation guidance.

[0124] In step S104, the acquired media files can undergo deduplication based on feature vector similarity. Image similarity algorithms can be used to analyze the similarity of images in the acquired media file set (e.g., damaged images) to identify and remove highly similar redundant images. Specifically, each image is vectorized through a feature extraction network to generate a corresponding image feature vector; then, the cosine distance between the feature vectors of any two images is calculated to measure image similarity, where a smaller cosine distance indicates higher similarity between the two images. When the detected image similarity exceeds a preset threshold, the two images are determined to be a similar image set. Furthermore, computer vision (CV) algorithms are used to evaluate image quality indicators such as image sharpness and exposure in this similar image set, automatically retaining the image with the highest sharpness and discarding the rest, thus achieving image deduplication and quality optimization (sharpness optimization). The deduplicated media files can then undergo the classification process described later. Alternatively, the above deduplication process can be omitted, and the acquired media files can be directly classified.

[0125] Media files can be categorized, and images belonging to the vehicle photo category can be filtered out. The media file categorization process includes identification and classification using a trained deep learning model. For example, a visual Transformer-based image classification model can be used to identify the category of damage assessment-related images. The construction and training process of the deep learning model includes the following operations (1) to (4): (1) Collect a set of sample images related to damage assessment (e.g., images related to historical damage assessment cases, with a collection of approximately 6,000 images). The collected sample images can be divided into a training set (e.g., approximately 4,000 images) and a validation set (e.g., approximately 2,000 images); (2) Manually label and classify the collected sample images. For example, they can be classified as: document images (e.g., vehicle registration certificate, driver's license, ID card, etc.), appraisal report images, accident liability determination images, vehicle photos, etc.; (3) Construct a classification model. The backbone module of the classification model is based on the open-source DINO pre-trained visual large model. The classifier head module (Classification Header Module) built based on MLP (Multilayer Perceptron) is connected as an adapter. DINO is based on ViT (Vision in Transformer). The visual base model constructed by the architecture contains, for example, about 800 million parameters, which can effectively extract image features; (4) Using the labeled sample images, on a multi-card Nvidia A100-80G GPU server, the model is trained using SGD (stochastic gradient descent) based on the PyTorch deep learning framework. In addition, various image augmentation methods can be used during model training, including random flipping, random cropping, random rotation, random adjustment of image brightness and contrast, etc., to improve the generalization ability of the model. At the same time, the training parameters (batch size, learning rate, etc.) are appropriately adjusted in multiple rounds of training, so as to obtain the model with the highest accuracy on the validation set.

[0126] Furthermore, media file classification is not limited to classification through deep learning models as described above; the acquired media files can be directly classified based on type (e.g., manual classification). However, the model-based media file classification process in the above embodiments enables rapid image retrieval and location in subsequent loss assessment processes (e.g., review, loss verification, or compensation verification), improving overall processing efficiency.

[0127] In step S104, for vehicle photo images filtered through the classification operation, a trained multimodal recognition model is used to identify vehicle parts within them. The identified vehicle parts are then matched with the damage assessment items to establish an association. That is, after completing image classification, for vehicle photo images, the automatic association between vehicle photos and parts items in the damage assessment report is further implemented.

[0128] While existing open-source multimodal large models based on the Transformer structure possess certain automotive parts recognition capabilities, they are insufficient to directly meet the needs of damage assessment operations and suffer from the following shortcomings: (1) the types of recognizable parts are limited, making it difficult to cover high-frequency (e.g., TOP50) parts in damage assessment scenarios, especially with weak recognition capabilities for internal components (non-exterior parts); (2) the spatial positioning accuracy of parts is insufficient, easily leading to problems such as indistinguishment between left and right, or front and back. In view of the above-mentioned shortcomings of the existing technology, this invention performs secondary training on the open-source multimodal large model to obtain a trained multimodal recognition model, thereby enabling effective and accurate identification and association of vehicle parts.

[0129] Specifically, the multimodal recognition model of the present invention is, for example, a model trained using supervised fine-tuning techniques. Training the multimodal recognition model may include: acquiring a sample image set of frequently occurring parts in vehicle damage assessments; for example, collecting images of the most frequently occurring parts (e.g., the top 50) in historical damage assessment reports; labeling the sample images in the sample image set to obtain training data; wherein, for example, labeling whether a specific part exists in the sample image (constructing positive and negative example labels); and labeling the location information of the specific part in the sample image (e.g., labeling in the form of bounding boxes); wherein at least 2000 sample images can be labeled for each type of part. Thus, training data containing (e.g., more than 120,000 images) of labeled records is obtained, and this data can be divided into a training set and a validation set.

[0130] The model training also includes supervised fine-tuning of the multimodal recognition model using labeled training data. Specifically, based on the labeled training data, the original model is trained using large-scale model SFT (supervised fine-tuning) optimization techniques. First, a multimodal visual-language model based on concept features is constructed. The backbone module uses the open-source SAM3 multimodal large-scale model's visual-language encoder to extract visual and linguistic features respectively. A multilayer perceptron (MLP) is used to build a projection-fusion module for visual and linguistic features after the backbone module. Finally, an MLP is used to construct the classification module. Then, using the training data, the model is trained on a multi-GPU Nvidia A100-80G server using the PyTorch deep learning framework and employing SGD (stochastic gradient descent). During training, various image augmentation methods can be used, including random cropping, random rotation, and random adjustment of image brightness and contrast, to improve the generalization ability of the model. At the same time, training parameters (batch size, learning rate, etc.) can be appropriately adjusted in multiple rounds of training to obtain the model with the highest accuracy on the validation set, thereby generating a multimodal recognition model suitable for loss assessment business scenarios.

[0131] After associating vehicle images with parts items on the damage assessment report, the associated images can be scored and sorted when multiple images correspond to the same part. Specifically, when calling the multimodal recognition model, the location information of the vehicle part in the image can be output, represented as a bounding box. After associating media files with damage assessment items, for multiple images associated with the same vehicle part, a score is calculated based on the area ratio of the bounding box of the part in each image, and the scores are then sorted and output.

[0132] Specifically, for example, images can be scaled proportionally to a preset size range. When the multimodal recognition model for accessories is invoked, the target bounding box information of the accessory is obtained, the area ratio of the target bounding box within the central region of the image is calculated, and the image is scored based on the area ratio; the larger the area ratio, the higher the image score. Based on the scoring results, multiple images associated with the same accessory are sorted and displayed. For example, the damage assessment results can include multiple sorted associated images.

[0133] Figure 4The diagram illustrates the automatic classification and association process for damage assessment images. First, images in the original image pool (acquired media images) undergo deduplication based on vector similarity. Then, a CNN convolutional network is used to classify the deduplicated images. After classification and filtering, a multimodal large model is used to identify accessories. Finally, through a scoring mechanism, the images are automatically associated with the damage assessment items, and the preview results are displayed in sorted order by score, serving as the damage assessment result.

[0134] Furthermore, after generating the operation guidance information in step S103, the process may also include generating an initial damage assessment report (first damage assessment data) and comparing it with an estimated damage report (second damage assessment data). Specifically, based on the damage assessment items in the structured data, a pre-set parts database can be queried to generate first damage assessment data containing price information; an image of an estimated damage report provided by a third party can be obtained, and the image can be structured and recognized using a multimodal visual language model to obtain second damage assessment data, which includes parts information and price information identified from the image; risk items are determined by comparing the first and second damage assessment data. The first and second damage assessment data may each contain parts codes and repair method information. The repair method information in the first damage assessment data can be obtained through the operation guidance information.

[0135] Specifically, after confirming the damage assessment items, the system automatically retrieves the corresponding part codes, prices, and repair hours from the backend parts database based on the part names and repair methods in the damage assessment preview form, generating an initial damage assessment form (first damage assessment data) containing price information. The damage assessor can then scan a damage assessment form (second damage assessment data, such as a computer-generated or handwritten form) provided by a third party (e.g., a repair shop) using a mobile terminal. The system uses a multimodal visual language model to perform structured recognition on the damage assessment form image, extracting information such as part names, part codes, repair methods, and prices. This information is then automatically compared with the initial damage assessment form to identify discrepancies and potential risks, thus assisting the damage assessor in accurately assessing vehicle damage (e.g., on-site negotiation and price negotiation).

[0136] Multimodal visual language models include qwen-vl-ocr (Qianwen OCR-VL multimodal model), qwen3-vl-plus (Qianwen 3-VL Plus multimodal model), gemini-3-pro-preview (Google Gemini3 multimodal large model), and gpt-5.2 (OpenAI gpt5.2 multimodal large model), etc.

[0137] Multimodal visual language models can perform structured image recognition by inputting images of third-party damage assessment reports along with pre-defined cue word constraints into the model. The pre-defined cue word constraints guide the model to extract structured information from predefined fields. Since damage assessment reports from different third parties (repair shops) vary significantly in format, especially handwritten reports which often lack fixed headers and are poorly written, pre-designing the cue words and providing clear feature constraints for various recognition elements before model invocation can improve the model's compatibility and recognition accuracy across different types of damage assessment reports.

[0138] Examples of prompt word constraints are as follows: For the Vehicle Identification Number (VIN) in the vehicle identification information, the following constraints are defined: the VIN consists of 17 letters and numbers, excluding the letters I, O, and Q, and is typically labeled in the damage assessment report image as "VIN", "Vehicle Identification Number", "Chassis Number", or "VIN" fields; For the part name, the following constraints are defined: the part name is characterized by a string composed of Chinese characters representing the vehicle part name, corresponding to fields such as "Materials Used", "Part Name", "Part Name", or "Description" in the table, but excluding the part code field.

[0139] Through constraints such as the aforementioned cue words, the multimodal visual language model outputs recognition results according to, for example, a pre-defined JSON data structure, without outputting other irrelevant content, thereby obtaining structured loss assessment data (second loss assessment data).

[0140] The prompt words and the image of the third-party damage assessment report are input into the multimodal visual language model. The model is invoked in the following way:

[0141] # Calling OpenAI compatible API

[0142] response = client.chat.completions.create(

[0143] model=config['model'],

[0144] messages=[

[0145] {

[0146] "role": "user",

[0147] "content": [

[0148] {

[0149] "type": "text",

[0150] "text": prompt

[0151] },

[0152] {

[0153] "type": "image_url",

[0154] "image_url": {

[0155] "url": f"data:{mime_type};base64,{base64_image}"

[0156] }

[0157] } ]

[0159] }

[0160] ],

[0161] max_tokens=8192 )

[0163] Here, prompt is a pre-designed prompt word, base64_image is the base64 encoding of the damage assessment form, and model is the name of the model, such as qwen-vl-ocr.

[0164] Here is an example of the model returning JSON:

[0165] {

[0166] {"name":"Front Bumper Removal and Installation", "price":"3276", "type":"labor", "work_type":"removal and installation", "part_code":""}

[0167] {"name":"Right front fender painting", "price":"8000", "type":"labor", "work_type":"painting", "part_code":""}

[0168] {"name":"Middle Frame", "price":"27500", "type":"part", "work_type":"Replacement","part_code":"52137308754"}

[0169] {"name":"Front Bumper Upper Body", "price":"964", "type":"part", "work_type":"Replacement", "part_code":"51117379434"}

[0170] {"name":"Left Front Fog Light", "price":"1200", "type":"part", "work_type":"Replacement", "part_code":"63177248911-A1"}

[0171] }

[0172] In addition, determining risk items by comparing the first loss assessment data with the second loss assessment data may include: comparing the first loss assessment data with the second loss assessment data to identify discrepancies; and, based on preset business rules, identifying at least a portion of the discrepancies as risk items.

[0173] Business rules may include at least one of the following: Parts code consistency verification rule, whether the parts codes in the first damage assessment data and the second damage assessment data are consistent; Parts price deviation threshold rule, whether the price deviation of parts in the first damage assessment data and the second damage assessment data exceeds a predetermined threshold; Repair method and damage degree matching rule, whether the repair method and damage degree of the first damage assessment data and the second damage assessment data match. Risk items may include, for example: Parts code risk, such as the parts code on the repair shop's damage estimate differing from the actual VIN code, potentially leading to under-insurance and over-equipment risk; Parts price risk, such as the parts price on the repair shop's damage estimate being higher than the official parts price collected from the price database in the damage assessment, or the price difference exceeding a predetermined value; Parts replacement standard risk, such as parts requiring replacement in the repair shop's damage estimate, while the damage assessment, based on the SOP process, determines they are repairable; Over-repair risk due to complete parts replacement, such as all parts requiring replacement in the repair shop's damage estimate, while the damage assessment, based on the SOP process and the composition of parts in the assembly, determines they can be partially replaced.

[0174] The risk items identified by comparing the first and second loss assessment data can be output and displayed to the user. After editing or adjusting the loss assessment information based on the risk items, a loss assessment report is generated. The aforementioned image association operation is then performed based on the loss assessment items in this report.

[0175] Figure 5 A schematic flowchart of a vehicle damage assessment method according to an embodiment of the present invention is shown. Figure 5As shown, the damage assessment process includes four stages. Stage 1: Generating a preview form using natural language. Natural language input includes three modes: natural voice input, natural chat mode, and simultaneous setup and shooting mode. The natural language information input in these modes is converted into text information. A large language model is used to identify the assessor's intentions, and natural language processing (NLP) is used to standardize the identified part names using word segmentation and retrieval techniques, thus generating a damage assessment preview form. Stage 2: Standard Operating Procedure (SOP)-guided generation of the damage assessment form. Information collection is guided by an SOP knowledge base. Stage 3: Valuation form identification and comparison with the damage assessment form. An initial damage assessment form is generated through querying. A multimodal large model is used to identify the valuation form image, obtaining the identified valuation form. The two are compared to highlight risks, and the final damage assessment form is edited based on these risks. After deduplication and classification of the collected images, a supervised optimization model is used to identify the images and associate them with the assessed parts, thus obtaining the damage assessment result.

[0176] The vehicle damage assessment method according to embodiments of the present invention can reduce the time spent on manual data entry, querying, and comparison by introducing intelligent assistance from large models, thereby accelerating the damage assessment process and improving the efficiency of vehicle damage assessment. Through standardized and structured processing, the completeness and consistency of damage assessment information collection are improved, reducing human error and thus increasing the accuracy of damage assessment. Through natural interaction, intelligent guidance, and automatic organization and association of damage assessment images, the workload of damage assessors is reduced, improving the efficiency of subsequent damage verification and providing reliable evidence. Furthermore, through SOP guidance and a difference comparison mechanism, risk control in the damage assessment process is brought forward to the on-site damage assessment stage, avoiding expanded damage and excessive repairs, effectively reducing compensation costs. Moreover, it provides complete, standardized, and traceable electronic damage assessment evidence for insurance claims.

[0177] Figure 6 This is a schematic diagram of the main modules of a vehicle damage assessment system according to an embodiment of the present invention. Figure 6 As shown, the vehicle damage assessment system 600 includes: a receiving unit 601, a structured data generation unit 602, an operation guidance information generation unit 603, and an association unit 604.

[0178] The receiving unit 601 receives natural language information describing the damage to the vehicle.

[0179] The structured data generation unit 602 uses natural language information and preset prompt words to call a large language model for processing, generating structured data containing at least one loss assessment item.

[0180] Specifically, the structured data generation unit 602 converts natural language information into text information; according to the format defined by preset prompt words, it combines the text information to form model input information; and it inputs the model input information into a large language model, wherein the preset prompt words are configured to guide the large language model to output structured data conforming to a predetermined format based on the text information. The preset prompt words may include at least one of the following for guiding the large language model: role definition information, task objective description, output format specifications, and standardized mapping rules for colloquial expressions in the input text.

[0181] The operation guidance information generation unit 603 generates operation guidance information corresponding to the loss assessment item based on structured data and outputs it to the user.

[0182] Specifically, the operation guidance information generation unit 603 determines the corresponding parts information and / or vehicle information based on the damage assessment items in the structured data; based on the parts information and / or vehicle information, it queries and obtains the corresponding operation guidance information from a predetermined standard operation procedure knowledge base. The operation guidance information may include recommended shooting instructions for specified parts.

[0183] The association unit 604 acquires media files related to vehicle damage and associates these media files with the damage assessment items to generate a damage assessment result. The media files can be collected based on operating instructions information.

[0184] Specifically, the association unit 604 can classify media files and filter out images belonging to the vehicle photo category; for vehicle photo images, it can identify vehicle parts in them using a trained multimodal recognition model; and it can match the identified vehicle parts with the damage assessment items to establish an association.

[0185] The multimodal recognition model is trained using supervised fine-tuning techniques. Training the multimodal recognition model can include: acquiring a sample image set of high-frequency vehicle damage assessment parts; labeling the sample images in the sample image set to obtain training data; and using the training data to perform supervised fine-tuning of the multimodal recognition model.

[0186] Annotating sample images can include indicating whether a specific accessory exists in the sample image and annotating the location information of that specific accessory in the sample image. Classifying media files can include performing identification and classification using a deep learning model based on a visual Transformer.

[0187] Before classifying media files, the association unit 604 can also perform similarity deduplication on media files based on feature vectors.

[0188] Identifying vehicle parts using a trained multimodal recognition model can include: outputting the location information of the vehicle parts in an image, represented as bounding boxes. After associating the media files with the damage assessment project, the association unit 604 can also score multiple images associated with the same vehicle part based on the area ratio of the bounding boxes in each image, and output the scores in a sorted manner.

[0189] The vehicle damage assessment system 600 also includes a comparison unit. After generating operation guidance information, the comparison unit queries a pre-set parts database based on the damage assessment items in the structured data to generate first damage assessment data containing price information; it acquires an image of a damage assessment report provided by a third party, performs structured recognition on the image using a multimodal visual language model, and obtains second damage assessment data, which includes parts information and price information identified from the image; by comparing the first damage assessment data with the second damage assessment data, risk items are determined.

[0190] The first and second damage assessment data may each include parts codes and / or repair method information. The repair method information in the first damage assessment data can be obtained from the operating instructions.

[0191] In the comparison unit, structured image recognition using a multimodal visual language model may include: inputting the image and preset cue word constraints into the model, where the cue word constraints guide the model to extract structured information from predefined fields. Determining risk items by comparing first and second damage assessment data may include: comparing the first and second damage assessment data to identify discrepancies; and, based on preset business rules, identifying at least a portion of the discrepancies as risk items. These business rules may include, for example, at least one of the following: parts code consistency verification rules; parts price deviation threshold rules; and repair method and damage severity matching rules.

[0192] The vehicle damage assessment system according to embodiments of the present invention can reduce the time spent on manual data entry, querying, and comparison by introducing intelligent assistance based on large models, thereby accelerating the damage assessment process and improving the efficiency of vehicle damage assessment. Through standardized and structured processing, the completeness and consistency of damage assessment information collection are improved, reducing human error and thus increasing the accuracy of damage assessment. Through natural interaction, intelligent guidance, and automatic organization and association of damage assessment images, the operational burden on damage assessors is reduced, improving the efficiency of subsequent damage verification and providing reliable evidence. Furthermore, through SOP guidance and a difference comparison mechanism, risk control in the damage assessment process is brought forward to the on-site damage assessment stage, avoiding expanded damage and excessive repairs, effectively reducing compensation costs. Moreover, it provides complete, standardized, and traceable electronic damage assessment evidence for insurance claims.

[0193] Figure 7An exemplary system architecture 700 for a vehicle damage assessment method or system to which embodiments of the present invention can be applied is shown.

[0194] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0195] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703.

[0196] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0197] Server 705 can be a server that provides various services, such as a back-end management server (for example only) that supports interactive operations performed by users using terminal devices 701, 702, and 703. The back-end management server can analyze and process data such as received product information query requests, and feed back the processing results (such as operation guidance information - for example only) to the terminal devices.

[0198] It should be noted that the vehicle damage assessment method provided in this embodiment of the invention is generally executed by server 705, and correspondingly, the vehicle damage assessment system is generally set in server 705.

[0199] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0200] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0201] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0202] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0203] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0204] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0206] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a receiving unit, a structured data generation unit, an operation guidance information generation unit, and an association unit. The names of these units do not necessarily limit the specific unit; for example, the receiving unit may also be described as "a unit that receives natural language information describing vehicle damage."

[0207] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: receiving natural language information describing vehicle damage; processing the natural language information and preset prompts using a large language model to generate structured data containing at least one damage assessment item; generating operation guidance information corresponding to the damage assessment item based on the structured data and outputting it to the user; acquiring media files related to vehicle damage and associating the media files with the damage assessment item to generate a damage assessment result.

[0208] According to the technical solution of this invention, the introduction of intelligent assistance based on large models can reduce the time spent on manual data entry, querying, and comparison, thereby accelerating the damage assessment process and improving the efficiency of vehicle damage assessment. Standardized and structured processing improves the completeness and consistency of damage assessment information collection, reduces human error, and thus improves the accuracy of damage assessment. Natural interaction, intelligent guidance, and automatic organization and association of damage assessment images reduce the workload of damage assessors, improve the efficiency of subsequent damage verification, and provide reliable evidence. Furthermore, through SOP guidance and difference comparison mechanisms, risk control in the damage assessment process is brought forward to the on-site damage assessment stage, avoiding expanded damage and excessive repairs, effectively reducing compensation costs. Moreover, it provides complete, standardized, and traceable electronic damage assessment evidence for insurance claims.

[0209] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for assessing vehicle damage, characterized in that, include: Receive natural language information describing the vehicle damage; Based on the natural language information and preset prompt words, a large language model is invoked for processing to generate structured data containing at least one damage assessment item. Based on the structured data, operation guidance information corresponding to the damage assessment item is generated and output to the user; Obtain media files related to vehicle damage, and associate the media files with the damage assessment items to generate damage assessment results.

2. The method according to claim 1, characterized in that, The process of calling a large language model for processing based on the natural language information and preset prompt words includes: Convert the natural language information into text information; According to the format defined by the preset prompt words, the text information is combined to form model input information; The model input information is input into the large language model, wherein the preset prompt words are configured to guide the large language model to output the structured data in a predetermined format based on the text information.

3. The method according to claim 1, characterized in that, The step of generating operational guidance information corresponding to the damage assessment item based on the structured data includes: Based on the damage assessment items in the structured data, determine the corresponding parts information and / or vehicle information; Based on the accessory information and / or the vehicle information, the corresponding operation guidance information is retrieved from a predetermined standard operation procedure knowledge base.

4. The method according to claim 1 or 3, characterized in that, The operation guidance information includes recommended shooting instructions for specific accessories.

5. The method according to claim 1, characterized in that, The media files were collected based on the operation guide information.

6. The method according to claim 1, characterized in that, Associating the media file with the damage assessment item includes: The media files are categorized, and images belonging to the vehicle photo category are selected. For the vehicle photo type of image, the vehicle accessories in it are identified by a trained multimodal recognition model; The identified vehicle parts are matched with the damage assessment items to establish a correlation.

7. The method according to claim 6, characterized in that, The training of the multimodal recognition model includes: Obtain a sample image set of frequently used parts for vehicle damage assessment; The sample images in the sample image set are labeled to obtain training data; The multimodal recognition model is then fine-tuned under supervision using the training data.

8. The method according to claim 7, characterized in that, The annotation of the sample image includes: annotating whether a specific accessory exists in the sample image, and annotating the location information of the specific accessory in the sample image.

9. The method according to claim 6, characterized in that, The classification of media files includes: identification and classification using a deep learning model based on visual Transformer.

10. The method according to claim 6, characterized in that, Before classifying the media files, the method further includes performing similarity deduplication on the media files based on feature vectors.

11. The method according to claim 6, characterized in that, The step of identifying vehicle accessories using a trained multimodal recognition model includes: outputting the location information of the vehicle accessories in the image, wherein the location information is represented in the form of accessory target boxes; After associating the media file with the damage assessment item, the method further includes: scoring multiple images associated with the same vehicle accessory based on the area ratio of the accessory's target box in each image, and sorting and outputting the scores.

12. The method according to claim 1, characterized in that, After generating the operation guidance information, the following is also included: Based on the damage assessment items in the structured data, a pre-set parts database is queried to generate first damage assessment data containing price information; The image of the damage assessment report provided by a third party is obtained, and the image is structured and recognized using a multimodal visual language model to obtain second damage assessment data. The second damage assessment data includes accessory information and price information identified from the image. Risk items are determined by comparing the first loss assessment data with the second loss assessment data.

13. A vehicle damage assessment system, characterized in that, include: The receiving unit is used to receive natural language information describing the vehicle damage. The structured data generation unit is used to process the natural language information and preset prompt words by calling a large language model to generate structured data containing at least one loss assessment item. The operation guidance information generation unit is used to generate operation guidance information corresponding to the damage assessment item based on the structured data and output it to the user; as well as The association unit is used to acquire media files related to the vehicle damage, associate the media files with the damage assessment items, and generate damage assessment results.

14. A vehicle damage assessment electronic device, characterized in that, include: One or more processors; as well as Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-12.

15. A computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-12.

16. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-12.

Images