A road disease intelligent identification and evaluation method based on a multi-modal large model and an instance segmentation algorithm

Abstract

The application discloses a kind of based on multi-modal big model and instance segmentation algorithm's road disease intelligent identification evaluation method, belong to road health monitoring field.The method includes: collecting the disease image data of road appearance, constructs original data set;Road appearance disease identification model is trained based on instance segmentation (Mask-RCNN etc.) deep learning method;Multi-modal prompt engineering framework based on parameter state transition is constructed, and field self-adapting question and answer data set is generated;Collect historical disease data, professional knowledge and evaluation standard related to road disease, construct professional knowledge compilation framework, and generate multi-modal big model disease evaluation data set;Based on the collaborative optimization method of multi-modal big model fusion specification compliance function, the training result is optimized, and the field special big model is formed;Based on the special big model of fine-tuning, an end-to-end road disease intelligent identification evaluation system is constructed.Through the method, the problem that traditional road appearance detection method depends on artificial experience and is low in efficiency can be solved, and a professional road condition evaluation system is constructed using a big model.

Description

Technical Field

[0001] This invention relates to an intelligent identification and assessment method for road defects based on a multimodal large model and instance segmentation algorithm, belonging to the field of road health monitoring. Background Technology

[0002] Road defect detection is a crucial aspect of road maintenance and traffic safety management. Traditional methods primarily rely on manual inspections or image acquisition by vehicle-mounted equipment followed by professional analysis, resulting in low efficiency and difficulty in meeting the detection needs of large-scale road networks. While automated detection technologies based on computer vision have improved efficiency to some extent, they are still limited by complex environmental factors such as lighting variations and occlusion, making it difficult to accurately extract both minute cracks and large potholes. Furthermore, existing technologies largely focus on defect identification, lacking a scientific quantification of the severity of defects (such as the impact of crack depth on structural load-bearing capacity, and the correlation between pothole area and traffic risk). The assessment process heavily depends on human experience, making it difficult to establish standardized decision-making criteria.

[0003] In recent years, deep learning technology has made significant progress in instance segmentation, especially with the emergence of multimodal large models, which have provided new solutions for road defect identification in complex scenarios. By fusing multi-source data such as vision, infrared, and LiDAR, and combining advanced algorithms such as supervised learning, multimodal large models can more robustly extract defect features and reduce reliance on labeled data. However, the application of multimodal large models in road defect detection is still limited, and general-purpose models have significant drawbacks: their outputs are prone to deviating from human professional expectations, making it difficult to meet the core requirements of accuracy and reliability in road maintenance scenarios.

[0004] A typical prior art technique, CN118918430A, discloses a method for constructing a bridge structural damage assessment model, a damage assessment method, and a system. This includes: acquiring multiple sets of damage data from different building structures to construct a real-label dataset, and acquiring multiple sets of unlabeled bridge structural damage data to construct a filter dataset. Multiple adversarial examples are generated based on the constructed real-label dataset to construct an adversarial example dataset. Various detection models for identifying structural damage are trained using the real-label dataset. The trained detection models identify the pseudo-labels corresponding to the sample data in the filter dataset, thereby constructing a pseudo-label dataset. Based on the real-label dataset, the pseudo-label dataset, and the adversarial example dataset, an adversarial distillation method is used to transfer the knowledge of all detection models to the bridge damage assessment model. This ensures the generalization ability of the trained bridge damage assessment model while reducing the model's dependence on the labeled bridge structural damage data. However, this model lacks improvements in terms of human-expected interaction.

[0005] CN118675074A discloses a road defect identification method and system based on artificial intelligence, belonging to the field of artificial intelligence technology. Specifically, it includes: collecting road image and video data through mobile inspection vehicles or drones, and autonomously and dynamically labeling road defect targets using an RDS system; selecting a multi-scale feature fusion and attention mechanism model as the base model, and training it using knowledge distillation technology combined with a ViT large model; improving the model's recognition performance by subdividing the defect dataset and introducing an adaptive weight allocation mechanism; deploying the trained model to the data processing system of an inspection vehicle or fixed camera, and ensuring the stable operation of the system through online updates and performance monitoring mechanisms, achieving efficient, accurate, and real-time road defect identification. However, this model does not improve the response format and cannot be evaluated.

[0006] To address the aforementioned problems, this invention provides a method for intelligent identification and assessment of road defects based on a multimodal large model and instance segmentation algorithm. Summary of the Invention

[0007] To address the problems of low efficiency and difficulty in quantifying the severity of road defects in existing road defect detection methods, this invention provides an intelligent road defect identification and assessment method based on a multimodal large model and instance segmentation algorithm.

[0008] In the first aspect, this invention proposes an intelligent identification and assessment method for road defects based on a multimodal large model and an instance segmentation algorithm, comprising the following steps:

[0009] Step 1: Collect image data of road surface defects to construct the original dataset;

[0010] Preferably, road surface damage image data are collected through vehicle-mounted or mobile devices, covering typical damage types such as cracks, potholes, and ruts, to construct an original dataset.

[0011] The image data of the disease is one or more of cracks, potholes, and ruts, and the cracks include one or more types such as transverse, longitudinal, and mesh-like.

[0012] Preferably, the vehicle-mounted equipment is equipped with a high-resolution industrial camera, which simultaneously acquires road surface images while traveling at a constant speed along the road, covering typical defects such as cracks and potholes. For complex areas (such as curves and intersections), drones can be used for multi-angle supplementary shooting to ensure the completeness of defect details. The data acquisition process covers different lighting and weather conditions, and records geographical location and environmental parameters. After preliminary screening to remove blurry or duplicate samples, the raw data forms a road surface defect dataset containing typical defects such as cracks (lateral, longitudinal, and mesh-like), potholes, and ruts.

[0013] Step 2: Train a road surface defect identification model based on instance segmentation deep learning method;

[0014] Preferably, the road surface defect identification model is trained based on instance segmentation (Mask-RCNN, etc.) deep learning methods.

[0015] Preferably, based on the road defect dataset constructed in step 1, an instance segmentation algorithm such as Mask R-CNN is used to train a dedicated identification model for road surface defects.

[0016] Preferably, by loading pre-trained weights and optimizing them for road defects, the model can learn the morphological features of defects such as cracks, potholes, and ruts.

[0017] Preferably, after training, the model is input with actual collected road images. The system can automatically detect the presence of defects in the images and output the precise location and quantity of defects, as well as geometric parameters (such as crack length and pothole area). Simultaneously, the model supports visualization output (e.g., ...). Figure 2 The diseased areas are visually displayed using masked annotations, providing a structured data foundation for subsequent assessment and analysis.

[0018] Step 3: Construct a multimodal prompting engineering framework based on parameter state transition to generate a domain-adaptive question-answering dataset;

[0019] Preferably, the purpose of constructing a domain-adaptive question-and-answer format prompt dataset is primarily to extract key disease parameter information from the input data during the training of a multimodal large model and to generate standardized prompt text, providing structured data support for subsequent hazard assessment. The prompt dataset adopts the SFT (Supervised Fine-Tuning) data format and contains two types of data: one is parameter completeness verification data, which is in the form of "image-question-answer (prompt text)"; the other is state transition memory chain data, which is in the form of "memory-question-prompt text".

[0020] In some embodiments, the image and query portions of each data point in the "parameter completeness verification data" format should include an image of the road to be detected and related parameter information (road length, lane width, and pavement material, etc.). When the parameter information in the query is incomplete, the answer will automatically prompt the user to fill in the missing items, for example:

[0021] After inputting the image of the road to be detected, ask: "This is a road in a certain area of ​​Guangzhou. The road is about 200 meters long and 3 meters wide. Can you help me assess the condition of this road?"

[0022] Answer: "The parameter information you provided is incomplete. Please provide information on the road surface material."

[0023] The "state transition memory chain data" format stores the incomplete dialogue chain with the aforementioned parameters as a memory unit, serving as the historical dialogue portion of the new data. The user's input after supplementing the parameters becomes the new question, and the system outputs a prompt text that integrates all parameters. For example:

[0024] Historical Dialogue:

[0025] Question: This is a road in a certain area of ​​Guangzhou. The road is about 200 meters long and 3 meters wide. Can you help me assess the condition of this road?

[0026] Answer: The parameter information you provided is incomplete. Please provide information about the road surface material.

[0027] Question: "The road has a cement concrete surface."

[0028] The prompt text reads: "Location: A road in a certain area of ​​Guangzhou; Road length: 200 meters; Road width: 3 meters; Road surface material: cement concrete."

[0029] When the parameters in the "state transition memory chain data" are complete, the system directly outputs the structured parameter text for subsequent steps.

[0030] Step 4: Collect at least historical road damage data, professional knowledge data related to road damage, and assessment standard data related to road damage; construct a road damage knowledge compilation framework; and generate a multimodal large-scale model damage assessment dataset.

[0031] Preferably, a large-scale road damage assessment model is constructed. The purpose is to integrate historical damage data with relevant professional standards to give the model the ability to scientifically quantify the severity of road damage. Relevant standard and specification documents, such as the "Highway Technical Condition Assessment Standard" and the "Highway Asphalt Pavement Maintenance Technical Specification," are collected to form a textual knowledge base.

[0032] The disease assessment dataset uses two data formats: SFT and pre-trained. The pre-trained data format includes documents from the knowledge base as text paragraphs, with each data entry corresponding to an independent knowledge unit. The SFT data format presents each data entry in the pre-trained dataset in a question-and-answer format, for example:

[0033] One of the data entries in the pre-training dataset is: "Text: When the width of a transverse crack is >5mm, it should be assessed as a Level III hazard and epoxy resin grouting treatment should be used." The corresponding SFT data format is:

[0034] Question: There is a transverse crack that is wider than 5mm. Will it pose a threat to the road?

[0035] Answer: This crack needs to be assessed as a Level III hazard and requires grouting with epoxy resin.

[0036] By using two data formats, the question-and-answer format is linked to regulatory provisions and real-world scenarios, which strengthens the big model's memory of professional knowledge and improves its dynamic decision-making capabilities.

[0037] Step 5: Based on the collaborative optimization method of the multimodal large model fusion canonical compliance function, the training results are optimized to form a dedicated large model;

[0038] Preferably, the adaptive question-and-answer prompt dataset constructed in step 3 is integrated with the disease assessment dataset in step 4, and supervised fine-tuning (SFT) is performed using existing multimodal large models (such as GPT-4V, LLaVA, etc.). By inputting data containing road images, disease parameters (such as crack length and pavement material), and interactive questions, the model learns to generate standardized prompt text and assessment suggestions.

[0039] Preferably, to make the large model's answers more consistent with human preferences and engineering standards, a reward model is introduced to optimize the training results (e.g., Figure 4 At the same time, a normalized conformance metric function is introduced into the reward model:

[0040] ;

[0041] in To determine the parameter matching degree, the crack length parameter is used for judgment. Does it comply with the regulations? The threshold is 1 if it meets the criteria, and 0 otherwise. This represents the normalization factor to avoid differences in numerical dimensions. and These represent the rationality scores of the model prediction scheme and the standard scheme, respectively. To standardize compliance weights, a larger value indicates a greater emphasis on parameter compliance (such as whether the crack length exceeds the limit). This is a weighting factor for cost reasonableness; the higher the value, the more attention is paid to the reasonableness of the repair plan. and The sum of the two is 1. This is the cost sensitivity coefficient; the larger the value, the more severe the penalty for result deviation. For cost sensitivity coefficient, when When it approaches 0, the scheme deviation is completely ignored; when Approaching When the value is 0, the exponent term approaches 0, and in practical engineering applications it is usually taken to be between 0.1 and 2.0.

[0042] Three candidate answers are generated for a question from a dataset using three different temperature coefficients (0.2, 0.7, and 1.0) of a multimodal large model. Each answer is scored by an operator, with scores based on three dimensions: linguistic reasonableness, accuracy of format, and operability (1-5 points). The arithmetic mean of the scores across these three dimensions is used as the final human score. Simultaneously, the... The function value is verified; if the manual score is consistent with... If the absolute difference of the linear mapping values ​​exceeds 0.5, a review mechanism is triggered, and the parameters of the reward model are optimized using the mean squared error loss function (MSE) so that it can automatically quantify the quality of the response.

[0043] The trained reward model is combined with a multimodal large model, and the generation strategy is iteratively adjusted through the proximal policy optimization (PPO) algorithm. For example, when the model generates "a crack depth of 30mm requires surface sealing treatment," the reward model determines its score to be 1.2 (out of 5) based on the standard, driving the large model to adjust its parameters, and finally outputs "Level III hazard requires deep grouting," which meets the standard. Through this process, the model gradually learns to generate high-scoring answers.

[0044] Step 6: Based on a dedicated large model, construct an end-to-end intelligent identification and assessment system for road defects.

[0045] Preferably, based on the road surface defect identification model trained in step 2 and the multimodal large model optimized in step 5, an end-to-end intelligent road defect identification and evaluation method is constructed (e.g., Figure 5 After the user or staff inputs the image to be detected and related questions, the large model first performs an integrity check on the input parameters:

[0046] If a parameter is missing, the missing item will be reported in real time and the user will be asked to fill it in until the parameter information is complete.

[0047] If the parameters are complete, the parameter information is automatically extracted to generate a prompt text, and the input image and prompt text are stored in the storage module via a script. This prompt text is not fed back to the user or staff.

[0048] The images in the storage module are input into the road surface defect recognition instance segmentation model trained in step 2 to predict the number, type, and size parameters (such as crack length and pothole area) of defects in the images. At the same time, the multimodal large model call process is triggered by the script file, combining the defect prediction results with prompt text, and using the defect knowledge base constructed in step 4 to evaluate the road condition in the images, and generate an evaluation report, which is then fed back to the user or staff.

[0049] Secondly, the present invention also claims protection for a road defect intelligent identification system, including a road defect intelligent identification module, the module being configured to evaluate road defects using the road defect intelligent identification and evaluation method based on multimodal large model and instance segmentation algorithm described in the first aspect.

[0050] Thirdly, the present invention also claims protection for an electronic device, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the intelligent road defect identification and assessment method of the first aspect.

[0051] Fourthly, the present invention also claims protection for a computer-readable storage medium having stored thereon computer instructions that, when executed, perform the intelligent road defect identification and assessment method based on multimodal large model and instance segmentation algorithm described in the first aspect to assess road defects.

[0052] The beneficial effects of this invention are as follows:

[0053] The multimodal large model applied in this invention can overcome the limitations of complex environments, while realizing a dynamic interactive optimization mechanism to ensure that the output results not only meet engineering standards, but also adapt to maintenance strategies in different regions.

[0054] By integrating the technical advantages of instance segmentation models and multimodal large models, an end-to-end intelligent road defect identification and assessment method was constructed. This method can automatically collect images of road surface defects, accurately identify defect types and locations using deep learning models, and achieve professional hazard assessment and decision-making recommendations through a fine-tuned multimodal large model.

[0055] This invention significantly improves the automation level of disease detection and the scientific rigor of assessment, solving the problems of reliance on manual experience and low efficiency in traditional methods. Simultaneously, by constructing a professional question-and-answer and assessment dataset, the large model possesses domain knowledge reasoning capabilities, providing reliable data support for road maintenance.

[0056] This method can not only adapt to the detection needs in complex environments, but also optimize model performance through continuous learning, which has important practical value for promoting the intelligent development of road inspection. Attached Figure Description

[0057] Figure 1 A framework for intelligent identification and assessment methods for road defects;

[0058] Figure 2 The structure of the road surface defect identification model;

[0059] Figure 3 Three data formats for multimodal large model question answering hint datasets;

[0060] Figure 4 For fine-tuning of multimodal large models and optimization of reward models;

[0061] Figure 5 The process of running the large model is input by the user or staff. Detailed Implementation

[0062] The specific embodiments of the present invention are described in detail below with reference to the technical solutions and accompanying drawings.

[0063] A road defect intelligent identification and assessment method based on multimodal large model and instance segmentation algorithm includes the following implementation steps:

[0064] Step 1: Collect image data of road surface defects using vehicle-mounted or mobile devices, covering typical defect types such as cracks, potholes, and ruts, and construct the original dataset.

[0065] In some embodiments, this invention employs a vehicle-mounted inspection device and a mobile terminal to collaboratively collect multimodal road defect images. The vehicle-mounted device is equipped with a high-resolution industrial camera, which simultaneously collects road surface images while traveling at a constant speed along the road, covering typical defects such as cracks and potholes. For complex areas (such as curves and intersections), drones can be used for multi-angle supplementary photography to ensure the completeness of defect details. The data collection process covers different lighting and weather conditions, and records geographical location and environmental parameters. After preliminary screening to remove blurry or duplicate samples, the raw data forms a road surface defect dataset containing typical defects such as cracks (lateral, longitudinal, and mesh-like), potholes, and ruts.

[0066] Step 2: Train a road surface defect identification model based on instance segmentation (Mask-RCNN, etc.) deep learning methods.

[0067] In some embodiments, based on the road defect dataset constructed in step 1, a dedicated road defect recognition model is trained using instance segmentation algorithms such as Mask R-CNN. By loading pre-trained weights and optimizing them for road defect characteristics, the model can learn the morphological features of defects such as cracks, potholes, and ruts. After training, the model is input into the actual collected road images, and the system can automatically detect the presence of defects in the images and output the precise location and quantity of defects, as well as geometric parameters (such as crack length, pothole area, etc.). Simultaneously, the model supports visualization output (e.g., ...). Figure 2 The diseased areas are visually displayed using masked annotations, providing a structured data foundation for subsequent assessment and analysis.

[0068] Step 3: Construct a multimodal prompting engineering framework based on parameter state transition to generate a domain-adaptive question-answering dataset.

[0069] In some embodiments, constructing a prompt dataset in the form of domain-adaptive question-and-answer is primarily aimed at extracting key disease parameter information from input data during the training of a multimodal large model and generating standardized prompt text, thus providing structured data support for subsequent hazard assessment. The prompt dataset adopts the SFT (Supervised Fine-Tuning) data format and contains two types of data: one is parameter completeness verification data, which is in the form of "image-question-answer (prompt text)"; the other is state transition memory chain data, which is in the form of "memory-question-prompt text".

[0070] In some embodiments, the image and query portions of each data point in the "parameter completeness verification data" format should include an image of the road to be detected and related parameter information (road length, lane width, and pavement material, etc.). When the parameter information in the query is incomplete, the answer will automatically prompt the user to fill in the missing items, for example:

[0071] In some embodiments, after inputting an image of the road to be detected, the user asks: "This is a road in a certain area of ​​Guangzhou. The road is approximately 200 meters long and 3 meters wide. Can you help me assess the condition of this road?"

[0072] Answer: "The parameter information you provided is incomplete. Please provide information on the road surface material."

[0073] In some embodiments, the "state transition memory chain data" format stores the incomplete dialogue chain with the aforementioned parameters as a memory unit, serving as the historical dialogue portion of new data. The user's input after supplementing the parameters becomes a new question, and the system outputs a prompt text that integrates all parameters. For example:

[0074] Historical Dialogue:

[0075] Question: This is a road in a certain area of ​​Guangzhou. The road is about 200 meters long and 3 meters wide. Can you help me assess the condition of this road?

[0076] Answer: The parameter information you provided is incomplete. Please provide information about the road surface material.

[0077] Question: "The road has a cement concrete surface."

[0078] The prompt text reads: "Location: A road in a certain area of ​​Guangzhou; Road length: 200 meters; Road width: 3 meters; Road surface material: cement concrete."

[0079] In some embodiments, when the parameters in the "state transition memory chain data" are complete, the system directly outputs structured parameter text for subsequent steps (such as...). Figure 3 ).

[0080] Step 4: Collect historical road damage data, relevant professional knowledge and assessment standards, construct a professional knowledge compilation framework, and generate a multimodal large-scale model road damage assessment dataset.

[0081] In some embodiments, a large-scale road damage assessment model is constructed to enable the model to scientifically quantify the severity of road damage by integrating historical damage data and relevant professional standards. Relevant standard and specification documents, such as the "Highway Technical Condition Assessment Standard" and the "Highway Asphalt Pavement Maintenance Technical Specification," are collected to form a text knowledge base. The damage assessment dataset uses two data formats: SFT and pre-training. The pre-training data format includes the documents in the knowledge base as text paragraphs, with each data point corresponding to an independent knowledge unit. The SFT data format presents each data point in the pre-training dataset in a question-and-answer format, for example:

[0082] In some embodiments, one of the data entries in the pre-training dataset is: "Text: When the width of a transverse crack is > 5 mm, it needs to be assessed as a Level III hazard and epoxy resin grouting treatment is required." The corresponding SFT data format is:

[0083] Question: There is a transverse crack that is wider than 5mm. Will it pose a threat to the road?

[0084] Answer: This crack needs to be assessed as a Level III hazard and requires grouting with epoxy resin.

[0085] By using two data formats, the question-and-answer format is linked to regulatory provisions and real-world scenarios, which strengthens the big model's memory of professional knowledge and improves its dynamic decision-making capabilities.

[0086] Step 5: Based on the collaborative optimization method of multimodal large model fusion specification compliance function, optimize the training results to form a domain-specific large model.

[0087] In some embodiments, the adaptive question-answering prompt dataset constructed in step 3 is integrated with the disease assessment dataset in step 4, and supervised fine-tuning (SFT) is performed using existing multimodal large models (such as GPT-4V, LLaVA, etc.). By inputting data containing road images, disease parameters (such as crack length and pavement material), and interactive questions, the model learns to generate standardized prompt text and assessment suggestions.

[0088] In some embodiments, to make the large model's responses more consistent with human preferences and engineering standards, a reward model is introduced to optimize the training results (e.g., ...). Figure 4 At the same time, a normalized conformance metric function is introduced into the reward model:

[0089]

[0090] in To determine the parameter matching degree, the crack length parameter is used for judgment. Does it comply with the regulations? The threshold is 1 if it meets the criteria, and 0 otherwise. This represents the normalization factor to avoid differences in numerical dimensions. and These represent the rationality scores of the model prediction scheme and the standard scheme, respectively. To standardize compliance weights, a larger value indicates a greater emphasis on parameter compliance (such as whether the crack length exceeds the limit). This is a weighting factor for cost reasonableness; the higher the value, the more attention is paid to the reasonableness of the repair plan. and The sum of the two is 1. For cost sensitivity coefficient, when When it approaches 0, the scheme deviation is completely ignored; when Approaching When the exponential term approaches 0, the penalty for deviation becomes more severe, typically between 0.1 and 2.0 in practical engineering applications. Using three different temperature coefficients (0.2, 0.7, 1.0) of a multimodal large model, three candidate answers are generated for a question in a dataset. Each answer is scored by an operator, with scores based on three dimensions: linguistic reasonableness, accuracy of format, and operability (1-5 points). The arithmetic mean of the scores across these three dimensions is used as the final human score. Simultaneously, the... The function value is verified; if the manual score is consistent with... If the absolute difference of the linear mapping values ​​exceeds 0.5, a review mechanism is triggered, and the parameters of the reward model are optimized using the mean squared error loss function (MSE) so that it can automatically quantify the quality of the response.

[0091] In some embodiments, a trained reward model is combined with a multimodal large model, and the generation strategy is iteratively adjusted using a proximal policy optimization (PPO) algorithm. For example, when the model generates "a crack depth of 30mm requires surface sealing," the reward model determines its score to be 1.2 (out of 5) based on the standard, driving the large model to adjust its parameters, ultimately outputting "Level III hazard requires deep grouting," which meets the standard. Through this process, the model gradually learns to generate high-scoring answers.

[0092] Step 6: Based on the finely tuned dedicated large model, construct an end-to-end intelligent identification and evaluation system for road defects.

[0093] Based on the road surface defect identification model trained in step 2 and the multimodal large model optimized in step 5, an end-to-end intelligent road defect identification and assessment method is constructed (e.g., Figure 5 ).

[0094] In some embodiments, after the user or staff inputs the image to be detected and related questions, the large model first performs an integrity check on the input parameters:

[0095] If a parameter is missing, the missing item will be reported in real time and the user will be asked to fill it in until the parameter information is complete.

[0096] If the parameters are complete, the parameter information is automatically extracted to generate a prompt text, and the input image and prompt text are stored in the storage module via a script. This prompt text is not fed back to the user or staff.

[0097] The images in the storage module are input into the road surface defect recognition instance segmentation model trained in step 2 to predict the number, type, and size parameters (such as crack length and pothole area) of defects in the images. At the same time, the multimodal large model call process is triggered by the script file, combining the defect prediction results with prompt text, and using the defect knowledge base constructed in step 4 to evaluate the road condition in the images, and generate an evaluation report, which is then fed back to the user or staff.

[0098] In some embodiments, the present invention also claims protection for a road defect intelligent identification system, including a road defect intelligent identification module, the module being configured to evaluate road defects using a road defect intelligent identification and evaluation method based on a multimodal large model and instance segmentation algorithm, as described in some embodiments.

[0099] In some embodiments, the present invention also claims protection for an electronic device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a road defect intelligent identification and assessment method in some embodiments.

[0100] In some embodiments, the present invention also claims protection for a computer-readable storage medium having stored thereon computer instructions that, when executed, perform a road defect intelligent identification and assessment method based on a multimodal large model and instance segmentation algorithm in some embodiments to assess road defects.

[0101] The above embodiments are merely preferred exemplary embodiments of this application, but the protection scope of the present invention is not limited thereto. Those skilled in the art should understand that any development or substitution that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention shall fall within the protection scope and disclosure scope of the present invention.

Claims

1. A method for intelligent identification and assessment of road defects based on a multimodal large model and instance segmentation algorithm, characterized in that: Includes the following steps: Step 1: Collect image data of road surface defects to construct the original dataset; Step 2: Train a road surface defect identification model based on instance segmentation deep learning method; Step 3: Construct a multimodal prompting engineering framework based on parameter state transition to generate a domain-adaptive question-answering dataset; Step 4: Collect historical road disease data, professional knowledge data related to road diseases, and assessment standard data related to road diseases; construct a road disease knowledge compilation framework; and generate a multimodal large model disease assessment dataset. Step 5: Based on the collaborative optimization method of the multimodal large model fusion canonical compliance function, the training results are optimized to form a dedicated large model; Step 6: Based on a dedicated large model, construct an end-to-end intelligent identification and assessment system for road defects; In step 5, a reward model is introduced to optimize the training results. This reward model incorporates a normalization metric function, which has the following expression: ； in To determine the parameter matching degree, the crack length parameter is used for judgment. Does it comply with the regulations? The threshold value is 1 if it meets the criteria, and 0 otherwise. This represents the normalization factor to avoid differences in numerical dimensions. and These represent the rationality scores of the model-predicted scheme and the standard scheme, respectively. To standardize compliance weights, a larger value indicates a greater emphasis on parameter compliance; As a weight for cost reasonableness, and The sum of the two is 1; Cost sensitivity coefficient Values ​​range from 0.1 to 2.

0.

2. The intelligent identification and assessment method for road defects based on a multimodal large model and instance segmentation algorithm according to claim 1, characterized in that: The image data of the disease is one or more of cracks, potholes, and ruts, and the cracks include one or more types such as transverse, longitudinal, and mesh-like.

3. The intelligent identification and assessment method for road defects based on a multimodal large model and instance segmentation algorithm according to claim 1, characterized in that: In step 3, the large model dataset question-and-answer format dataset is constructed using three data formats: "image-question-answer", "image-question-hint text", and "memory-question-hint text".

4. The intelligent identification and assessment method for road defects based on a multimodal large model and instance segmentation algorithm according to any one of claims 1-3, characterized in that: In step 4, the collected professional knowledge data and evaluation standard data related to road defects are processed using SFT dataset and pre-trained dataset, respectively, to enhance the professional knowledge capabilities and dynamic decision-making capabilities of the large model.

5. A road defect intelligent identification and assessment method based on a multimodal large model and instance segmentation algorithm according to any one of claims 1-3, characterized in that: In step 5, a normative compliance quantification function is introduced into the reward model. Combined with human scoring of three dimensions—language rationality, normative accuracy, and operability—the quality of the large model's answers is automatically quantified, allowing the model to gradually learn and generate high-scoring answers.

6. The intelligent identification and assessment method for road defects based on a multimodal large model and instance segmentation algorithm according to any one of claims 1-3, characterized in that: In step 6, the multimodal large model and the road surface defect identification algorithm are combined to construct a road condition identification and evaluation system.

7. A road defect intelligent identification and assessment method based on a multimodal large model and instance segmentation algorithm according to any one of claims 1-3, characterized in that: After the user or staff inputs the image to be detected and related questions, the large model first performs an integrity check on the input parameters: If a parameter is missing, the missing item will be reported in real time and the user will be asked to fill it in until the parameter information is complete. If the parameters are complete, the parameter information is automatically extracted to generate a prompt text, and the input image and prompt text are stored in the storage module via a script. This prompt text is not fed back to the user or staff.

8. A road defect intelligent identification and assessment method based on a multimodal large model and instance segmentation algorithm according to any one of claims 1-3, characterized in that: The adaptive question-answering hint dataset constructed in step 3 is integrated with the disease assessment dataset in step 4, and supervised fine-tuning is performed using GPT-4V or LLaVA.

9. A road defect intelligent identification system, characterized in that: The system includes a road defect intelligent identification module, which is configured to evaluate road defects using the road defect intelligent identification and evaluation method based on a multimodal large model and instance segmentation algorithm as described in any one of claims 1-8.

Images