Road disease intelligent identification and evaluation method based on multi-modal large model and instance segmentation algorithm

Abstract

The invention discloses a road disease intelligent identification and evaluation method based on a multi-modal large model and an instance segmentation algorithm, and belongs to the field of road health monitoring. The method comprises the following steps: acquiring disease image data of road appearance, and constructing an original data set; training a road apparent disease recognition model based on an instance segmentation (Mask-RCNN and the like) deep learning method; constructing a multi-modal prompt engineering framework based on parameter state transition, and generating a field adaptive question and answer data set; historical disease data and professional knowledge and evaluation standards related to road diseases are collected, a professional knowledge compiling framework is constructed, and a multi-modal large model disease evaluation data set is generated; optimizing a training result based on a collaborative optimization method of a multi-modal large model fusion specification conformity function to form a large model special for the field; and constructing an end-to-end road disease intelligent identification and evaluation system based on the fine-tuned special large model. By means of the method, the problems that a traditional road appearance detection method depends on artificial experience and is low in efficiency can be solved, and meanwhile a professional road condition evaluation system is built through a large model.

Description

TECHNICAL FIELD

[0001] The present application relates to a road disease intelligent identification and evaluation method based on a multi-modal large model and an instance segmentation algorithm, and belongs to the field of road health monitoring. BACKGROUND

[0002] Road disease detection is an important part of road maintenance and traffic safety management. Traditional methods mainly rely on manual inspection or image collection by vehicle-mounted equipment, which is then analyzed by professionals. This method is inefficient and difficult to meet the detection needs of large-scale road networks. Although computer vision-based automated detection technology has improved efficiency to some extent, it is still limited by complex environmental factors such as light changes and obstructions, making it difficult to accurately extract both minor cracks and large-sized potholes. In addition, existing technologies focus on disease identification and lack scientific quantification of disease severity (such as the impact of crack depth on structural bearing capacity and the correlation between pothole area and traffic risk), and the evaluation process relies heavily on human experience, making it difficult to form standardized decision-making criteria.

[0003] In recent years, deep learning technology has made significant progress in the field of instance segmentation, especially the emergence of multi-modal large models, which provides a new solution for road disease identification in complex scenarios. By integrating visual, infrared, and laser radar data, and combining advanced algorithms such as supervised learning, multi-modal large models can more robustly extract disease features and reduce dependence on labeled data. However, current multi-modal large models have limited application in road disease detection, and general models have significant shortcomings: their output results are prone to deviate from human professional expectations, making it difficult to meet the core needs of accuracy and reliability in road maintenance scenarios.

[0004] CN118918430A, a typical prior art, discloses a bridge structure damage assessment model construction method, damage assessment method and system, which includes: obtaining multiple different building structure damage data to construct a real label data set, and obtaining multiple unlabeled bridge structure damage data to construct a screening data set. According to the constructed real label data set, multiple anti-samples are generated to construct an anti-sample data set. The real label data set is used to train multiple detection models for identifying structural damage. The trained detection models are used to identify the pseudo-labels corresponding to the sample data in the screening data set, thereby constructing a pseudo-label data set. Based on the real label data set, the pseudo-label data set, and the anti-sample data set, an anti-distillation method is used to transfer the knowledge of all detection models to a bridge damage assessment model. This ensures that the trained bridge damage assessment model has generalization ability while reducing the dependence of model training on bridge structure damage annotation data. However, this model lacks improvements in human expectation interaction.

[0005] CN118675074A discloses a road disease identification method and system based on artificial intelligence, belonging to the technical field of artificial intelligence, which specifically includes: collecting road image and video data through a mobile inspection vehicle or a drone, and using an RDS system to autonomously and dynamically label road disease targets; selecting a multi-scale feature fusion and attention mechanism model as a base model, and training it through a knowledge distillation technology combined with a ViT large model; improving the recognition performance of the model by subdividing the disease data set and introducing an adaptive weight distribution mechanism; deploying the trained model to a data processing system of the inspection vehicle or a fixed camera, and ensuring the stable operation of the system through an online updating and performance monitoring mechanism, thereby realizing efficient, accurate and real-time road disease identification. However, the model does not improve the answer form and cannot be evaluated.

[0006] Based on the above problems, the present application provides a road disease intelligent identification evaluation method based on a multi-modal large model and an instance segmentation algorithm. SUMMARY

[0007] In order to solve the problems of low efficiency and difficulty in quantitatively evaluating the disease risk of the existing road disease detection method, the present application provides a road disease intelligent identification evaluation method based on a multi-modal large model and an instance segmentation algorithm.

[0008] In a first aspect, the present application provides a road disease intelligent identification evaluation method based on a multi-modal large model and an instance segmentation algorithm, comprising the following steps: Step 1, collecting road surface disease image data to construct an original data set; Preferably, the road surface disease image data is collected by a vehicle-mounted or mobile device, covering typical disease types such as cracks, pits, and ruts, to construct an original data set.

[0009] The disease image data is one or more of cracks, pits, and ruts, and the cracks include one or more of horizontal, vertical, and networked.

[0010] Preferably, the vehicle-mounted device is equipped with a high-resolution industrial camera, which synchronously collects road surface images at a uniform speed along the road, covering typical diseases such as cracks and pits. For complex areas such as curves and intersections, a drone can be used for multi-angle retakes to ensure the integrity of disease details. Different light and weather conditions are covered during the collection process, and geographical location and environmental parameters are recorded. After preliminary screening to remove blurred or duplicate samples, the original data form a road surface disease data set containing typical diseases such as cracks (horizontal, vertical, and networked), pits, and ruts.

[0011] Step 2, training a road surface disease identification model based on an instance segmentation deep learning method; Preferably, the road surface disease recognition model is trained based on instance segmentation (Mask-RCNN, etc.) deep learning method.

[0012] Preferably, the road surface disease recognition model is trained based on instance segmentation (Mask-RCNN, etc.) deep learning method.

[0013] Preferably, the model can learn the morphological characteristics of diseases such as cracks, potholes, and ruts by loading pre-trained weights and optimizing for road disease characteristics.

[0014] Preferably, after training, the actual collected road images are input into the model, and the system can automatically detect whether there are diseases in the images and output the precise location, number, and other geometric parameters (such as crack length, pothole area, etc.) of the diseases. At the same time, the model supports visual output (such as Figure 2 ), which intuitively displays the disease area in the form of mask annotation, providing a structured data basis for subsequent evaluation and analysis.

[0015] Step 3: Construct a multi-modal prompt engineering framework based on parameter state migration to generate a domain adaptive question-answering big data set; Preferably, by constructing a prompt data set in the form of domain adaptive question-answering, the key disease parameter information is extracted from the input data during the training of the multi-modal large model, and a standardized prompt text is generated to provide structured data support for subsequent hazard assessment. The prompt data set adopts SFT (Supervised Fine-Tuning) data format, which includes two types of data: one is parameter completeness verification data, which is in the form of "image-question-answer (prompt text)"; the other is state migration memory chain data, which is in the form of "memory-question-prompt text".

[0016] In some embodiments, the image and question part of each data in the form of "parameter completeness verification data" should contain the road image to be detected and related parameter information (road length, lane width, and road surface material, etc.). When the parameter information in the question is incomplete, the answer will automatically ask the user to supplement the missing items, for example: After inputting the road image to be detected, the question is: "This is a certain road in a certain area of Guangzhou City, with a total length of about 200 meters and a road width of 3 meters. Can you help me assess the condition of this road?" The answer is: "The parameter information you provided is incomplete. Please provide the material information of the road surface." The "state migration memory chain data" form is to set the above parameter incomplete dialogue chain as a memory unit storage, as the historical dialogue part of the new data, and the user's input after supplementing the parameters as a new question, and the system outputs all the parameters as a prompt text. For example: Historical dialogue: "Question: This is a certain road in a certain area of Guangzhou City, with a total length of about 200 meters and a width of 3 meters. Can you help me assess the condition of this section of road? Answer: The parameter information you provided is incomplete. Please provide the road surface material information." Question: "The road is a cement concrete pavement." Prompt text: "Geographical location: a certain road in a certain area of Guangzhou City; Road length: 200 meters; Road width: 3 meters; Road surface material: cement concrete." When the parameters in the "state transition memory chain data" are complete, the system directly outputs the structured parameter text, which is provided to subsequent steps.

[0017] Step 4, collect at least historical disease data, professional knowledge data related to road diseases, and evaluation standard data related to road diseases, build a road disease knowledge compilation framework, and generate a multi-modal large model disease assessment data set; Preferably, a disease assessment large model is built, aiming to integrate historical disease data and related professional specifications to give the large model the ability to scientifically quantify the road disease hazard level. Collect relevant standard specification files such as "Highway Technical Condition Assessment Standard" and "Highway Asphalt Pavement Maintenance Technical Specification" to form a text knowledge base.

[0018] The disease assessment data set adopts SFT and pre-training two data formats. The pre-training data format includes files in the knowledge base in the form of text paragraphs, with each data corresponding to an independent knowledge unit. The SFT data format presents each data in the pre-training data set in the form of questions and answers, for example: One of the data in the pre-training data set is: "Text: When the width of transverse cracks is greater than 5mm, it needs to be assessed as III level hazard and needs to be treated with epoxy resin crack pouring." "Question: Now there is a transverse crack with a width greater than 5mm, will it cause harm to the road? Answer: The crack needs to be assessed as III level hazard and needs to be treated with epoxy resin crack pouring." Through the two data formats, the question and answer form is associated with the specification provisions and the actual scene, strengthening the large model's memory of professional knowledge while improving its dynamic decision-making ability.

[0019] Step 5, based on the multi-modal large model, the collaborative optimization method of the specification compliance function is used to optimize the training results, forming a special large model; Preferably, the adaptive question and answer prompt dataset constructed in step 3 is integrated with the disease assessment dataset in step 4, and a multi-modal large model (such as GPT-4V, LLaVA, etc.) is used for supervised fine-tuning (SFT). By inputting data containing road images, disease parameters (such as crack length, pavement material), and interactive questions, the model learns to generate standardized prompt text and evaluation suggestions.

[0020] Preferably, in order to make the answers of the large model more in line with human preferences and engineering specifications, a reward model (RewardModel) is introduced to optimize the training results (such as Figure 4 ), and a specification compliance quantification function is introduced in the reward model: ; Where is the parameter matching degree, which judges whether the crack length parameter meets the specification threshold, which is 1 if it meets the range, otherwise 0. represents the normalization factor to avoid dimensional differences. and represent the rationality score of the model's predicted scheme and the rationality score of the standard scheme, respectively. is the specification compliance weight, the larger the value, the more emphasis on parameter compliance (such as whether the crack length is out of range); is the cost rationality weight, the larger the value, the more attention to the rationality of the repair scheme, and The sum of the two is 1. is the cost sensitivity coefficient, the larger the value, the stricter the punishment for the deviation of the result. is the cost sensitivity coefficient, when tends to 0, the scheme deviation is completely ignored; when tends to , the exponential term tends to 0, usually between 0.1 and 2.0 in actual engineering applications.

[0021] Three different temperature coefficients (0.2, 0.7, 1.0) of the multi-modal large model are used to generate three candidate answers to the question in the data. The operator scores each answer, including language rationality, specification accuracy, and operability (1-5 points), and takes the arithmetic mean of the three dimensions as the final artificial score. At the same time, the function value is calculated for verification. If the absolute difference between the artificial score and the linear mapping value exceeds 0.5, the review mechanism is triggered, and the mean square error loss function (MSE) is used to optimize the parameters of the reward model, so that it can automatically quantify the quality of the answer.

[0022] The trained reward model is combined with the multi-modal large model, and the generation strategy is iteratively adjusted through the proximal policy optimization (PPO) algorithm. For example, when the model generates "crack depth 30mm adopts surface sealing treatment", the reward model determines that its score is 1.2 (full score 5) based on the specification, drives the large model to adjust the parameters, and finally outputs the standard "III level hazard needs deep crack pouring". Through this process, the model gradually learns to generate high-score answers.

[0023] Step 6, based on the special large model, an end-to-end road disease intelligent identification and evaluation system is constructed.

[0024] Preferably, based on the road surface disease identification model trained in step 2 and the multi-modal large model optimized in step 5, an end-to-end road disease intelligent identification and evaluation method is constructed (such as Figure 5 ). After the user or the staff inputs the image to be detected and the related problems, the large model first checks the integrity of the input parameters: If the parameters are missing, the missing items are fed back in real time and the user is required to supplement them until the parameter information is complete; If the parameters are complete, the parameter information is automatically extracted to generate a prompt text, and the input image and the prompt text are stored in the storage module through a script, and the prompt text is not fed back to the user or the staff.

[0025] The image in the storage module is input into the road surface disease identification instance segmentation model trained in step 2 to predict the number, type and size parameters (such as crack length, pit area) of the disease in the image. At the same time, the multi-modal large model calling process is triggered by using a script file, the disease prediction result is combined with the prompt text, and the road condition in the image is evaluated in combination with the disease knowledge base constructed in step 4, and an evaluation report is generated and fed back to the user or the staff.

[0026] In a second aspect, the present application also claims a road disease intelligent identification system, comprising a road disease intelligent identification module, which is configured to adopt the road disease intelligent identification and evaluation method based on the multi-modal large model and the instance segmentation algorithm of the first aspect to evaluate the road disease.

[0027] In a third aspect, the present application also claims an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the road disease intelligent identification and evaluation method of the first aspect when executing the computer program.

[0028] In a fourth aspect, the present application also claims a computer readable storage medium having computer instructions stored thereon, which, when executed, perform the road disease intelligent identification and evaluation method based on the multi-modal large model and the instance segmentation algorithm of the first aspect to evaluate the road disease.

[0029] The beneficial effects of the present application are: The multi-modal large model applied in the present application can overcome the limitations of complex environments, while realizing a dynamic interactive optimization mechanism, ensuring that the output results not only meet the engineering standards, but also adapt to different regional maintenance strategies.

[0030] By integrating the technical advantages of the instance segmentation model and the multi-modal large model, an end-to-end road disease intelligent identification and evaluation method is constructed. This method can automatically collect road surface disease image data, accurately identify disease types and locations using deep learning models, and realize professional hazard evaluation and decision-making suggestions through the fine-tuned multi-modal large model.

[0031] The present application significantly improves the automation level of disease detection and the scientificity of evaluation, solving the problem of low efficiency and dependence on manual experience of traditional methods. At the same time, by constructing a professional question and answer and evaluation dataset, the large model has the ability of domain knowledge reasoning, providing reliable data support for road maintenance.

[0032] This method not only adapts to the detection needs in complex environments, but also optimizes the model performance through continuous learning, which has important practical value for promoting the intelligent development of road inspection. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The road disease intelligent identification and evaluation method step framework; Figure 2 The road surface disease identification model structure; Figure 3 Three data forms of the multi-modal large model question and answer prompt dataset; Figure 4 Multi-modal large model fine-tuning and reward model optimization; Figure 5 User or staff input and large model running process. DETAILED DESCRIPTION

[0034] The specific embodiments of the present application will be described in detail below in combination with the technical solutions and drawings.

[0035] A road disease intelligent identification and evaluation method based on a multi-modal large model and an instance segmentation algorithm, comprising the following implementation steps: Step 1, collect road surface disease image data through a vehicle-mounted or mobile device, covering typical disease types such as cracks, pits, and ruts, and construct an original dataset.

[0036] In some embodiments, the present application employs a vehicle-mounted inspection device and a mobile terminal to cooperatively collect multi-modal road disease images. The vehicle-mounted device is equipped with a high-resolution industrial camera and synchronously collects road surface images while driving at a constant speed, covering typical diseases such as cracks and potholes. For complex areas such as curves and intersections, a drone can be used for multi-angle retakes to ensure the integrity of disease details. The collection process covers different lighting and weather conditions and records geographic location and environmental parameters. After preliminary screening to remove blurred or duplicate samples, the raw data form a road surface disease dataset containing cracks (transverse, longitudinal, and network), potholes, and other typical diseases.

[0037] Step 2, train a road surface disease recognition model based on instance segmentation (Mask-RCNN, etc.) deep learning method.

[0038] In some embodiments, based on the road disease dataset constructed in step 1, a road surface disease-specific recognition model is trained using Mask R-CNN and other instance segmentation algorithms. By loading pre-trained weights and optimizing for road disease characteristics, the model can learn the morphological features of diseases such as cracks, potholes, and ruts. After training, the actual collected road images are input into the model, and the system can automatically detect whether there are diseases in the image and output the precise location, number, and other geometric parameters of the disease (such as crack length, pothole area, etc.). At the same time, the model supports visual output (such as Figure 2 ), which intuitively displays the disease area in mask annotation form, providing a structured data foundation for subsequent evaluation and analysis.

[0039] Step 3, construct a multi-modal prompt engineering framework based on parameter state migration to generate a domain-adaptive question-answering dataset.

[0040] In some embodiments, by constructing a prompt dataset in the form of domain-adaptive question-answering, the main purpose is to train multi-modal large models to extract key disease parameter information from input data and generate standardized prompt text, providing structured data support for subsequent hazard assessment. The prompt dataset uses SFT (Supervised Fine-Tuning) data format, containing two types of data: one is parameter completeness verification data, which is in the form of "image-question-answer (prompt text)"; the other is state migration memory chain data, which is in the form of "memory-question-prompt text".

[0041] In some embodiments, the image and question part of each data in the "parameter completeness verification data" form should contain the road image to be detected and related parameter information (road length, lane width, and road surface material, etc.). When the parameter information in the question is incomplete, the answer will automatically ask the user to supplement the missing items, for example: In some embodiments, after inputting the road image to be detected, the question is asked: "Is this a certain road in a certain area of Guangzhou City, about 200 meters long and 3 meters wide? Can you help me assess the condition of this road?" Answer: "The parameter information you provided is incomplete. Please provide the road surface material information." In some embodiments, the "state transition memory chain data" form is to set the above parameter incomplete dialogue chain as a memory unit storage, as the historical dialogue part of the new data, and the user's input after supplementing the parameters as a new question. The system output integrates all parameters as a prompt text. For example: Historical dialogue: "Question: Is this a certain road in a certain area of Guangzhou City, about 200 meters long and 3 meters wide? Can you help me assess the condition of this road? Answer: The parameter information you provided is incomplete. Please provide the road surface material information." Question: "The road is a cement concrete pavement." Prompt text: "Geographical location: a certain road in a certain area of Guangzhou City; Road length: 200 meters; Road width: 3 meters; Road surface material: cement concrete." In some embodiments, when the parameters in the "state transition memory chain data" are complete, the system directly outputs the structured parameter text, which is provided to the subsequent steps (such as Figure 3 ).

[0042] Step 4, collect historical disease data, professional knowledge related to road disease, and evaluation standards, build a professional knowledge compilation framework, and generate a multi-modal large model disease evaluation data set.

[0043] In some embodiments, the purpose of building a disease evaluation large model is to give the large model the ability to scientifically quantify the road disease hazard level by integrating historical disease data and related professional specifications. Collect relevant standard specification files such as "Highway Technical Condition Evaluation Standard" and "Highway Asphalt Pavement Maintenance Technical Specification" to form a text knowledge base. The disease evaluation data set adopts SFT and pre-training two data formats. The pre-training data format includes files in the form of text paragraphs in the knowledge base, and each data corresponds to an independent knowledge unit. The SFT data format presents each data in the pre-training data set in the form of a question and answer, for example: In some embodiments, one of the data in the pre-training data set is: "Text: When the width of a transverse crack is greater than 5mm, it needs to be evaluated as a III-level hazard and needs to be treated with epoxy resin crack pouring." The corresponding SFT data format is: "Question: Now there is a transverse crack with a width greater than 5mm, will it cause harm to the road? Answer: The crack needs to be rated as a III-level hazard and requires epoxy resin for crack filling treatment. By associating the question and answer forms with the actual scene through two data formats, the professional knowledge memory of the large model is strengthened, and its dynamic decision-making ability is improved.

[0044] Step 5: Based on the multi-modal large model fusion specification compliance function collaborative optimization method, the training results are optimized to form a domain-specific large model.

[0045] In some embodiments, the adaptive question and answer prompt dataset constructed in step 3 is integrated with the disease assessment dataset in step 4, and the existing multi-modal large model (such as GPT-4V, LLaVA, etc.) is used for supervised fine-tuning (SFT). By inputting data containing road images, disease parameters (such as crack length, road surface material), and interactive questions, the model learns to generate standardized prompt text and evaluation suggestions.

[0046] In some embodiments, in order to make the answers of the large model more in line with human preferences and engineering specifications, a reward model (Reward Model) is introduced to optimize the training results (such as Figure 4 ), and at the same time, a specification compliance quantification function is introduced into the reward model:

[0047] where is the parameter matching degree, which judges whether the crack length parameter meets the specification threshold value, the compliance range is 1, otherwise 0. represents the normalization factor to avoid dimensional differences. and represent the rationality score of the model prediction scheme and the standard scheme rationality score, respectively. is the specification compliance weight, the larger the value, the more emphasis on parameter compliance (such as whether the crack length is over the limit); is the cost rationality weight, the larger the value, the more attention to the rationality of the repair scheme, and the sum of the two is 1. is the cost sensitivity coefficient, when tends to 0, the scheme deviation is completely ignored; when tends to If the temperature coefficient is 0.2, the exponential term approaches 0, and the penalty for deviation from the result is more severe. In practical engineering applications, the value is usually between 0.1 and 2.0. The three different temperature coefficients (0.2, 0.7, 1.0) of the multi-modal large model are used to generate three candidate answers to a question in a data set. The operator scores each answer, which includes language reasonableness, standard accuracy, and operability (1-5 points). The arithmetic mean of the scores of the three dimensions is taken as the final artificial score, and the absolute difference between the artificial score and the linear mapping value is calculated If the absolute difference exceeds 0.5, the review mechanism is triggered, and the mean square error loss function (MSE) is used to optimize the parameters of the reward model to automatically quantify the quality of the answer.

[0048] In some embodiments, the trained reward model is combined with the multi-modal large model to iteratively adjust the generation strategy through the proximal policy optimization (PPO) algorithm. For example, when the model generates “crack depth 30mm adopts surface sealing treatment”, the reward model determines that the score is 1.2 (full score 5) based on the standard, and drives the large model to adjust the parameters, and finally outputs the standard-compliant “Ⅲ-level hazard requires deep crack pouring”. Through this process, the model gradually learns to generate high-score answers.

[0049] Step 6: Based on the fine-tuned special-purpose large model, an end-to-end road disease intelligent identification and evaluation system is constructed.

[0050] Based on the road surface disease identification model trained in step 2 and the multi-modal large model optimized in step 5, an end-to-end road disease intelligent identification and evaluation method (such as Figure 5 ) is constructed.

[0051] In some embodiments, after the user or worker inputs the image to be detected and the related question, the large model first performs integrity check on the input parameters: If the parameters are missing, the missing items are fed back in real time and the user is required to supplement them until the parameter information is complete; If the parameters are complete, the parameter information is automatically extracted to generate a prompt text, and the input image and the prompt text are stored in the storage module through a script. The prompt text is not fed back to the user or worker.

[0052] The image in the storage module is input into the road surface disease identification instance segmentation model trained in step 2 to predict the number, type and size parameters (such as crack length, pit area) of the disease in the image. At the same time, the multi-modal large model calling process is triggered by using a script file to combine the disease prediction results with the prompt text, and to combine step 4 to build a disease knowledge base to evaluate the road conditions in the image and generate an evaluation report to feed back to the user or worker.

[0053] ​In some embodiments, the present application also claims a road disease intelligent identification system, comprising a road disease intelligent identification module, the module is configured to adopt a road disease intelligent identification evaluation method based on a multi-modal large model and an instance segmentation algorithm in some embodiments to evaluate road diseases.

[0054] In some embodiments, the present application also claims an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the road disease intelligent identification evaluation method in some embodiments when executing the computer program.

[0055] In some embodiments, the present application also claims a computer readable storage medium, which stores computer instructions, and the computer instructions execute the road disease intelligent identification evaluation method based on a multi-modal large model and an instance segmentation algorithm in some embodiments to evaluate road diseases when running.

[0056] The above embodiments are only preferred exemplary embodiments of the present application, but the protection scope of the present application is not limited thereto, and those skilled in the art should understand that any development or replacement within the technical scope disclosed by the present application can be easily thought out by those skilled in the art, and all of them belong to the protection scope and disclosure scope of the present application.

Claims

1. A method for intelligent identification and assessment of road defects based on a multimodal large model and instance segmentation algorithm, characterized by: The steps include: Step 1: Collect road surface damage image data and construct the original data set; Step 2: Train a road surface damage recognition model based on instance segmentation deep learning method; Step 3: Build a multimodal prompt engineering framework based on parameter state migration to generate a large dataset of domain-adaptive question answering. Step 4: Collect at least historical disease data, professional knowledge data related to road diseases, and assessment standard data related to road diseases, build a road disease knowledge compilation framework, and generate a multimodal large-scale model disease assessment dataset; Step 5: Based on the collaborative optimization method of the multimodal large model fusion specification compliance function, the training results are optimized to form a dedicated large model; Step 6: Build an end-to-end road disease intelligent identification and assessment system based on a dedicated large model.

2. The method for intelligent road damage identification and assessment based on a multimodal large model and an instance segmentation algorithm according to claim 1, characterized in that: The damage image data is one or more of cracks, potholes, and ruts, and the cracks include one or more of horizontal, vertical, and reticular types.

3. According to claim 1, a method for intelligent identification and assessment of road damage based on a multimodal large model and an instance segmentation algorithm is characterized by: In step 3, the large model dataset question-answer format dataset is constructed using three data formats: "image-question-answer", "image-question-prompt text", and "memory-question-prompt text".

4. The method for intelligent identification and assessment of road damage based on a multimodal large model and an instance segmentation algorithm according to any one of claims 1 to 3, characterized in that: In step 4, the collected professional knowledge data related to road diseases and the evaluation standard data related to road diseases are respectively used as SFT data sets and pre-training data sets to enhance the professional knowledge capabilities and dynamic decision-making capabilities of the large model.

5. The method for intelligent identification and assessment of road damage based on a multimodal large model and an instance segmentation algorithm according to any one of claims 1 to 3, characterized in that: In step 5, a function for quantifying normative compliance is introduced into the reward model. Combined with manual scoring in three dimensions, namely, language rationality, normative accuracy, and operability, the quality of the large model's answers is automatically quantified, allowing the model to gradually learn to generate high-scoring answers.

6. The method for intelligent identification and assessment of road damage based on a multimodal large model and an instance segmentation algorithm according to any one of claims 1 to 3, characterized in that: In step 6, the multimodal large model and the road surface defect recognition algorithm are combined to construct a road condition recognition and assessment system.

7. The method for intelligent identification and assessment of road damage based on a multimodal large model and an instance segmentation algorithm according to any one of claims 1 to 3, characterized in that: In step 5, a reward model is introduced to optimize the training results. A specification compliance metric function is introduced into the reward model. The specification compliance metric function has the following expression: ； in For parameter matching, determine the crack length parameter Whether it complies with the specifications Threshold, if it meets the range, it is 1, otherwise it is 0; Represents a normalization factor to avoid numerical dimension differences; and They represent the rationality score of the model prediction scheme and the rationality score of the standard scheme respectively; is the normative compliance weight, where a larger value indicates a greater emphasis on parameter compliance; is the cost rationality weight, and The sum of the two is 1; is the cost sensitivity coefficient, The value ranges from 0.1 to 2.

0.

8. The method for intelligent identification and assessment of road damage based on a multimodal large model and an instance segmentation algorithm according to any one of claims 1 to 3, characterized in that: After the user or staff inputs the image to be tested and the related questions, the large model first performs a completeness check on the input parameters: If a parameter is missing, the missing item will be fed back in real time and the user will be asked to supplement it until the parameter information is complete; If the parameters are complete, the parameter information is automatically extracted to generate a prompt text, and the input image and the prompt text are stored in a storage module through a script. The prompt text is not fed back to the user or staff.

9. The method for intelligent identification and assessment of road damage based on a multimodal large model and an instance segmentation algorithm according to any one of claims 1 to 3, characterized in that: Integrate the adaptive question-answering prompt dataset constructed in step 3 with the disease assessment dataset in step 4, and use GPT-4V or LLaVA for supervised fine-tuning.

10. A road hazard intelligent identification system, characterized by: It includes a road damage intelligent recognition module, which is configured to use the road damage intelligent recognition and evaluation method based on a multimodal large model and an instance segmentation algorithm as described in any one of claims 1 to 9 to evaluate road damage.

Images