Pipeline defect detection and report generation method based on multi-modal alignment and weak supervised learning

Abstract

The invention discloses a multi-modal alignment and weak supervised learning-based pipeline defect detection and report generation method, which comprises the following steps of: performing data cleaning and standardization processing on pictures in a pre-acquired pipeline defect report and corresponding text information, and labeling the processed pictures, converting the processed picture and text information into high-dimensional semantic vector representation to obtain a text vector and a picture vector, and establishing a data set in combination with the annotation of the picture; based on the data set, training a multi-modal vector aligned large language model, and realizing defect detection and report generation to obtain a defect detection result and a report; performing validity verification and expert feedback correction on the defect detection result and report; the pipeline defect detection and report generation task can be efficiently and accurately completed, and the testing efficiency and the intelligent level in the defect repairing process are improved.

Description

Technical Field

[0001] This invention belongs to the field of industrial intelligent inspection technology, and in particular relates to a method for pipeline defect detection and report generation based on multimodal alignment and weakly supervised learning. Background Technology

[0002] In the context of rapid urbanization, urban water pipe networks, as a crucial infrastructure of urban water supply systems, directly impact the quality of life for urban residents and the sustainability of urban operations through their operational safety and maintenance efficiency. Due to their wide distribution, deep burial, and complex operating environment, urban water pipes are highly susceptible to various defects such as corrosion, cracks, and leaks during long-term service. Failure to detect and address these defects promptly can lead to widespread water supply disruptions, water waste, and even serious secondary disasters like road collapses, posing significant challenges to urban management.

[0003] Currently, defect detection in urban water pipe networks mainly relies on manual inspections, partial excavations, and traditional image processing methods. Manual inspections are inefficient and highly subjective, making it difficult to achieve comprehensive, real-time monitoring of large-scale underground pipe networks. Traditional image processing and detection methods, on the other hand, have high requirements for the standardization and completeness of data annotation. In practical applications, historical defect reports and images often suffer from inconsistent annotations, redundant information, and image watermarks, resulting in low data utilization and affecting the training effect and generalization ability of detection models.

[0004] With the development of deep learning and large language models, multimodal models based on vision-language alignment have shown great potential in fields such as image understanding, automatic annotation, and report generation. However, for real-world scenarios like urban water pipe networks, there are still many challenges in efficiently cleaning and standardizing historical defect data, making full use of weak annotations and multi-source information, and improving the automation, intelligence, and interpretability of defect detection. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning that can automatically generate structured reports.

[0006] Technical solution: The present invention provides a method for pipeline defect detection and report generation based on multimodal alignment and weakly supervised learning, comprising:

[0007] (1) Data cleaning and standardization are performed on the images and corresponding text information in the pre-acquired pipeline defect reports, and the processed images are labeled. The processed images and text information are converted into high-dimensional semantic vector representations to obtain text vectors and image vectors. A dataset is established by combining the image labels.

[0008] (2) Based on the dataset, train a large language model with multimodal vector alignment to achieve defect detection and report generation, and obtain defect detection results and reports;

[0009] (3) Verify the validity of the defect detection results and reports and make corrections based on expert feedback.

[0010] Further, step (1) includes:

[0011] (11) Extract images and corresponding text information from the pre-acquired pipeline defect report. The text information includes the defect category and defect level. Perform data cleaning on the extracted images and text information. Specifically, the data cleaning is as follows: remove non-natural elements from the images, filter invalid or defect-related information from the text information and extract information corresponding to the defect type and defect level. The non-natural elements include watermarks, manual annotations, arrows and text.

[0012] (12) Standardize the cleaned images and text information, and automatically annotate the processed images to obtain the image annotations; the annotations include binary masks, defect bounding boxes, corresponding defect categories and defect levels;

[0013] (13) Using a multimodal embedding model, the image and text information obtained in step (12) are converted into high-dimensional semantic vector representations, and a dataset is established; the high-dimensional semantic vector representations are as follows:

[0014]

[0015]

[0016] in, It is a text vector; This is a text embedding model used to convert text information into a continuous vector representation; The text is the standardized version; It is an image vector; It is an image encoder, a convolutional neural network used to extract image features; For images containing annotations.

[0017] Furthermore, step (12) of automatically annotating the processed image includes:

[0018] A multimodal model using BERT and ResNet-50 was trained for classification using images with known defect categories. For the trained model, a normalized image was input, and the feature map of the last convolutional layer and classification weights were extracted. The contribution of each spatial location to the classification result was calculated, generating a heatmap. The heatmap was then normalized and binarized using the threshold function in OpenCV to obtain a binary mask. Regions with a value of 1 in the binary mask were defined as automatically identified defect regions. Next, the findContours function was used to extract all contours from the mask, and the boundingRect function was used to calculate the minimum bounding rectangle for each contour to obtain the defect bounding box.

[0019] Further, step (2) includes:

[0020] (21) Construct a multimodal input template to integrate the text vectors, image vectors and image annotations in the dataset into a template-formatted multimodal input;

[0021] (22) The multimodal input is fed into the large language model for inference to generate defect detection results and reports; the defect detection results and reports both contain required fields, including defect type, defect level, defect location and confidence level; the defect location is represented by a defect bounding box;

[0022] (23) Extract and verify whether the generated defect detection results and reports meet the preset pipeline inspection standards and report format requirements. Specifically, check whether the required fields of the defect detection results and reports are complete, and whether the defect detection results and reports are in the preset format. If they do not meet the requirements, the generation is deemed invalid and regenerated.

[0023] Further, step (22) includes:

[0024] First, the multimodal input from step (21) is loaded in batches and divided into multiple batches. Each batch contains text vectors, image vectors and image annotations, including defect type, defect level, defect bounding box and binary mask.

[0025] Feature fusion is performed by directly concatenating image vectors and text vectors to form fused features. These fused features are then input into a multilayer perceptron for defect type classification and defect location regression to obtain defect detection results.

[0026] Input the fused features into the text generation model to generate a report.

[0027] Furthermore, step (22) also includes: training the model before generating defect detection results and reports. During the training process, cross-entropy loss, L1 loss and text generation loss are jointly optimized, wherein cross-entropy loss is used for classification supervision, L1 loss is used for localization supervision, and text generation loss is used for report generation.

[0028] Further, step (3) includes:

[0029] (31) Verify the validity of the defect detection results and reports, including the following:

[0030] (311) Field integrity check: Check whether the defect detection result contains all required fields; the required fields include: defect type, defect level, defect location and confidence level;

[0031] (312) Format and numerical validity verification: including confirming that the defect bounding box is a quadruple and the coordinates are within the image size range; confirming that the confidence level is a decimal between 0 and 1; confirming that the report content is not empty;

[0032] (313) The defect detection results and reports after inspection and verification are compared with manual annotations or expert opinions, and the best result is selected to obtain the verified defect detection results and reports; the comparison method is as follows: when the IoU between the defect bounding box in the defect detection results and reports after inspection and verification and the manually annotated defect bounding box is higher than 0.5, it is determined that the defect detection results and reports are complete and meet the actual needs of the project.

[0033] (32) Submit the verified defect detection results and reports to the pipeline engineer for manual review and correction, and collect feedback information from the manual review.

[0034] Furthermore, S320 also includes: if no problems are found during manual review, the model results are adopted and the feedback information is used for continuous learning; otherwise, the model results are manually corrected, the corrected samples are added to the incremental training queue, and the manual review steps are iteratively executed to continuously train the model.

[0035] Furthermore, in step (21), the multimodal input template includes two parts: system role definition and user input. The system role definition is used to specify the professional role of the model, and the user input includes context information, defect report content, defect image and detection and report generation instructions.

[0036] Furthermore, in step (22), the parameter settings of the large language model include temperature parameters, sampling parameters, maximum length parameters, word frequency penalty parameters, and existence penalty parameters.

[0037] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. By automating data cleaning and labeling standardization, it significantly improves data quality and reduces manual intervention and subjective errors by utilizing automatic denoising and labeling normalization processes to address watermarks, non-standard manual annotations, and other noise issues in images. Simultaneously, by combining natural language processing and large-scale modeling techniques, it normalizes and unifies the labels for defect types and other information using synonyms, unifying non-standard defect descriptions into standard labels, overcoming the problems of high consistency and error-proneness associated with manual annotation in traditional methods. 2. By fusing image and text information, it achieves automatic pipeline defect localization through contrastive learning and cross-modal attention mechanisms, overcoming the high dependence of traditional methods on high-quality datasets. 3. Automated data cleaning and labeling standardization significantly reduce manual workload, making it suitable for large-scale, batch applications. Attached Figure Description

[0038] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0039] Figure 2 Figure for an embodiment of the present invention;

[0040] Figure 3 This is an unprocessed image illustration from an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the processed image according to an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0043] As attached Figure 1 As shown in the figure, the pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning in this embodiment includes:

[0044] S100: Perform data cleaning and normalization on the images and corresponding text information in the pre-acquired pipeline defect reports, and label the processed images. Transform the processed images and text information into high-dimensional semantic vector representations to obtain text vectors and image vectors. Combine the image annotations to establish a dataset.

[0045] S200: Based on the dataset, train a large language model with multimodal vector alignment to achieve defect detection and report generation, and obtain defect detection results and reports;

[0046] S300: Verify the validity of defect detection results and reports and make corrections based on expert feedback.

[0047] Specifically, in step S100, the images and text information in the pipeline defect report are cleaned, and the cleaned images and text information are standardized. The processed images are then automatically labeled, and subsequently converted into high-dimensional semantic vector representations using a multimodal embedding model. These vector representations are then stored in a multimodal training set to provide data support for subsequent model training. The pipeline defect report is as follows: Figure 2 As shown. Specifically, it is achieved through the following sub-steps:

[0048] S110: Pipeline Defect Data Cleaning: Filter xxxx defect data entries from existing pipeline defect reports. Each data entry includes the defect type (defect_type), defect level (defect_level), and its associated defect image. The specific processing method is as follows:

[0049] (1) Collect pipeline defect reports, extract text (defect level, defect type) and images from the pipeline defect reports, and handle unprocessed images such as Figure 3 As shown;

[0050] (2) The cleaning process includes two parts: image cleaning and text information cleaning.

[0051] Image cleaning: OpenCV is used for color segmentation and edge detection to initially identify elements such as watermarks, hand-drawn annotations, arrows, and text in the image. Erosion, dilation, and opening / closing operations are employed to enhance the segmentation effect of the target region. A mask is generated for the detected watermark or annotation areas. Finally, the LaMa model is used to repair the masked areas, restoring the original image texture and removing interference.

[0052] Text cleaning: Remove special characters and invalid content, and encode to UTF-8.

[0053] S120: Standardize the cleaned images and text information. The processed images are as follows: Figure 4 As shown.

[0054] Specifically, it is divided into text information standardization and image standardization:

[0055] Text information normalization: By maintaining a standard label mapping table, synonyms or different expressions are unified into standard labels. For words that do not appear in the mapping table, we use the BERT model to calculate the semantic similarity between the word and the standard label, and select the closest label for normalization.

[0056] defect_type_map = {

[0057] "Foreign object penetration"

[0058] "Undulations": "Undulations",

[0059] "Obstacles": "Obstacles",

[0060] "Ruptures": "Ruptures",

[0061] "Disconnections": "Disconnections",

[0062] "Corrosion": "Corrosion",

[0063] "Misalignments": "Misalignments",

[0064] "Deformations": "Deformations",

[0065] "Tree roots": "Tree roots",

[0066] "Branch pipe blind connections": "Branch pipe blind connections",

[0067] "Deposits": "Deposits",

[0068] "Leaks": "Leaks",

[0069] "Remaining walls": "Remaining walls",

[0070] "dam foundations": "dam foundations",

[0071] "Scale buildup": "Scale buildup",

[0072] "Joint material detachment": "Joint material detachment"

[0073] }

[0074] defect_level_map = {

[0075] "Level 1": "1",

[0076] "Level 2": "2",

[0077] "Level 3": "3",

[0078] "Level 4": "4"

[0079] }

[0080] For words not appearing in the mapping table, use the following similarity algorithm:

[0081]

[0082] Where w represents the word to be normalized, and l i For standard label representation, V w =BERT(w) represents the word vector, V li =BERT(l i Let ) be the vector of the i-th standard label, and BERT(x) be the BERT encoder.

[0083] Image standardization: Unify the image format to JPG and use OpenCV to standardize the image size and resolution for easier processing.

[0084] Automated annotation: A multimodal model using BERT + ResNet-50 is trained on images with known defect categories. For the trained model, a normalized image is input, the last convolutional feature map and classification weights are extracted, the contribution of each spatial location to the classification result is calculated, and a heatmap (activation map) is generated. The heatmap highlights the regions the model deems most likely to be defects. The heatmap is then normalized and binarized using the OpenCV threshold function to obtain a binary mask. Regions with a value of 1 in the mask are automatically inferred defect regions. Next, the findContours function is used to extract all contours from the mask. Then, the boundingRect function is used to calculate the minimum bounding rectangle for each contour, resulting in the standard defect bounding box (bbox(x_min, y_min, x_max, y_max)).

[0085] S130: Input the processed text information and image into BLIP-2 (Bootstrapped Language-Image Pretraining). BLIP-2 extracts image and text features through a visual encoder and a text encoder, respectively, and uses a cross-modal attention mechanism in the fusion layer to achieve deep semantic alignment and feature fusion.

[0086] (1) Input of text information and images together: Concatenate fields such as defect type and defect level into a natural language description, such as "crack, minor", and input it together with the processed defect image into the BLIP-2 multimodal encoder.

[0087] (2) Multimodal feature extraction and fusion: The BLIP-2 model extracts image features through a visual encoder and text features through a text encoder. It achieves depth alignment and feature fusion of images and text in the fusion layer and outputs a unified multimodal semantic vector.

[0088] (3) Data structuring and storage: The processed text, images, fused multimodal feature vectors, and related metadata (mask, bbox, etc.) are organized into a structured JSON format to facilitate subsequent multimodal model training and downstream task applications.

[0089] All experiments were conducted on an Ubuntu 20.04 LTS environment, with the experimental machine equipped with 32GB of memory, an Intel Xeon Gold 5318Y CPU, and an NVIDIA A30 GPU. Data processing scripts were written in Python 3.8.

[0090] In step S200, based on the normalized dataset, the multimodal aligned large language model is fine-tuned to achieve defect detection, localization, and report generation. This is specifically achieved through the following sub-steps:

[0091] S210: Constructing a Multimodal Input Template: Constructing a clear prompt template requires integrating the current standardized text information, images, and their annotations into a standardized multimodal input format for input into the large language model. Specifically, this involves integrating core content such as defect type, defect level, and defect images from the current text information to form a complete context, and using Markdown syntax to distinguish different modules such as system prompts, defect reports, and task instructions.

[0092] By combining current pipeline defect information with automated annotation results and referencing best practices of multimodal large models such as BLIP-2, the system prompts assign the role of "Senior Pipeline Defect Detection Expert" to the large language model to enhance professional understanding. At the same time, the template specifies that information such as defect type, defect level, and defect report should be output in JSON format. The model is required to perform self-checks on the output results to ensure the completeness of the output content and the standardization of the format, and to ensure that all fields are filled in. This achieves a closed loop of the entire process from information integration and format verification to role allocation, operation guidance, and result verification.

[0093] The prompt template is shown below:

[0094] Prompt Template

[0095] {"Role": "System","Content": """ You are a senior pipeline defectdetection expert. Your task is to identify defect types, assess defectseverity, locate defect areas (e.g., bbox), and generate structuredinspection reports based on the pipeline defect reports and images providedby users. Please ensure that the output is complete, accurate, and formattedaccording to specifications..

[0096] """}

[0097] {"Role": "User","Content":"""# Context:

[0098] Pipeline ID: [PIPE_ID]

[0099] Inspection Time: [INSPECTION_TIME]

[0100] Location: [LOCATION]

[0101] Inspector: [INSPECTOR]

[0102] Historical Suggestions: [HISTORY_SUGGESTION]

[0103] # Defect report:

[0104] ##Name:

[0105] [DEFECT_NAME]

[0106] ##Description:

[0107] [DEFECT_DESCRIPTION]

[0108] ##Severity:

[0109] [DEFECT_LEVEL]

[0110] ## Defect Image:

[0111] [IMAGE_PATH or BASE64]

[0112] ## Defect Annotation (Optional):

[0113] bbox:[x_min, y_min, x_max, y_max]

[0114] # Inspection and Report Generation Instructions:

[0115] 1. Detect the defect type in the image and output standardized labels(e.g., crack, corrosion, etc.).

[0116] 2. Determine the defect severity level (e.g., minor, moderate,severe).

[0117] 3. Locate the defect area and output the bounding box (bbox)coordinates.

[0118] 4. Provide the detection confidence level (a decimal between 0 and1).

[0119] 5. Generate a structured inspection report, including pipeline ID,defect type, severity level, location description, and recommended actions.

[0120] 6. Please verify that the output content is complete and the formatis standardized, ensuring that all fields are populated.

[0121] # Output format requirements:

[0122] Please output the result in the following JSON format:

[0123] ‘’‘

[0124] {

[0125] "defect_type": "",

[0126] "defect_level": "",

[0127] "defect_bbox": [x_min, y_min, x_max, y_max],

[0128] "confidence": 0.0,

[0129] "report": {

[0130] "pipe_id": "",

[0131] "defect": "",

[0132] "level": "",

[0133] "location": "",

[0134] "suggestion": ""

[0135] }

[0136] }

[0137] ‘‘’

[0138] """

[0139] }

[0140] S220: First, load the high-dimensional semantic vectors output from the S130 stage in batches. Each batch (the batch size is set according to the actual situation, and the batch is automatically generated by DataLoader in the training code) contains text vectors (v_text), image vectors (v_img), and annotations (defect type, bounding box, mask).

[0141] In the feature fusion stage, image vectors and text vectors are directly concatenated to form a unified multimodal feature representation. This multimodal feature representation is input into a multilayer perceptron (MLP) for defect type classification and defect location regression (bbox or mask prediction). The defect location is represented by a defect bounding box.

[0142] In the report generation task, fused features are further input into the text generation model to automatically generate a structured detection report. During training, cross-entropy loss is used for classification supervision, L1 loss for localization supervision, and text generation loss for report generation; all three are jointly optimized.

[0143] S230: Extract and preliminarily verify the generated output: Extract the content enclosed in ```, preliminarily check whether it meets the pipeline inspection standards and report format requirements, and check: whether the fields are complete; whether the data is output in JSON format; if not, the generation is deemed invalid and regenerated.

[0144] In step S300, the defect detection results and structured reports automatically generated by the model are rigorously validated to ensure they comply with industry standards for pipeline defect detection, and the model performance is continuously optimized through expert feedback.

[0145] S310: Perform automated validity verification of the defect detection results and reports generated by the model. The verification content includes...

[0146] (1) Field integrity check: Write a Python script to automatically check the output JSON / structured results to ensure that all required fields are included: defect type (defect_type), defect location (bbox or mask), confidence, and structured report (report), including pipeline ID, defect type, level, location description, recommendations, etc.

[0147] (2) Format and numerical validity check: Write a Python script to traverse all outputs, check the format and numerical range of each item, and automatically record and throw a warning when an anomaly is found.

[0148] bbox validity: Check that the bbox is a quadruple ([x_min, y_min, x_max, y_max]) and that the coordinates are within the image size range.

[0149] Confidence range: Ensure that the confidence is a decimal number between 0 and 1.

[0150] Report content is not empty: All report fields (such as pipeline ID, recommendations, etc.) are not empty.

[0151] (3) Comparison with manual annotations or expert opinions: The IoU (Intersection over Union) between the model output bounding box and the manually annotated bounding box is calculated. An IoU higher than 0.5 is considered a valid detection. Then, the defect type predicted by the model is compared with the manual annotations / expert opinions to see if they are consistent. If there are multiple rounds of generation, the result with the highest IoU, the highest confidence, or the most consistent with the expert opinion is selected as the final output.

[0152] (4) Results recording and anomaly handling: Automatically record all verification results, including anomaly types, correction suggestions, etc., to facilitate subsequent statistics and traceability.

[0153] Abnormal output is automatically marked, which can trigger a manual review process.

[0154] S320: Submit the test results and report to the pipeline engineer for manual review and correction, and collect expert feedback. The specific process is as follows:

[0155] The system sends the detection results and reports to the pipeline engineer. The engineer reviews the defect type, defect location, and report content, and corrects any errors found directly (e.g., adjusting bounding boxes, modifying defect types, or supplementing report content). The system records all manual corrections, including before-and-after comparisons, reasons for corrections, and expert opinions. High-quality manually corrected samples are automatically included in the "expert annotation set" as a data source for subsequent incremental model training or fine-tuning.

[0156] As can be seen from the above embodiments, the present invention can automatically and efficiently process noisy images by combining defect reports and large language model reasoning capabilities, completely overcoming pipeline inspection and report generation tasks, greatly improving the testing efficiency and accuracy in the pipeline defect repair process, and has good engineering application value and promotion prospects.

Claims

1. A method for pipeline defect detection and report generation based on multimodal alignment and weakly supervised learning, characterized in that, include: (1) Data cleaning and standardization are performed on the images and corresponding text information in the pre-acquired pipeline defect reports, and the processed images are labeled. The processed images and text information are converted into high-dimensional semantic vector representations to obtain text vectors and image vectors. The dataset is established by combining the image labels. (2) Based on the dataset, train a large language model with multimodal vector alignment to achieve defect detection and report generation, and obtain defect detection results and reports; (3) Verify the validity of the defect detection results and reports and make corrections based on expert feedback.

2. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 1, characterized in that, Step (1) includes: (11) Extract images and corresponding text information from the pre-acquired pipeline defect report. The text information includes the defect category and defect level. Perform data cleaning on the extracted images and text information. Specifically, the data cleaning is as follows: remove non-natural elements from the images, filter invalid or defect-related information from the text information and extract information corresponding to the defect type and defect level. The non-natural elements include watermarks, manual annotations, arrows and text. (12) Standardize the cleaned images and text information, and automatically annotate the processed images to obtain the image annotations; the annotations include binary masks, defect bounding boxes, corresponding defect categories and defect levels; (13) Using a multimodal embedding model, the image and text information obtained in step (12) are converted into high-dimensional semantic vector representations, and a dataset is established; the high-dimensional semantic vector representations are as follows: ； ； in, It is a text vector; This is a text embedding model used to convert text information into a continuous vector representation; The text is the standardized version; It is an image vector; It is an image encoder, a convolutional neural network used to extract image features; For images containing annotations.

3. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 2, characterized in that, The automated annotation of the processed image in step (12) includes: A multimodal model using BERT and ResNet-50 was trained for classification using images with known defect categories. For the trained model, a normalized image was input, and the feature map of the last convolutional layer and classification weights were extracted. The contribution of each spatial location to the classification result was calculated, generating a heatmap. The heatmap was then normalized and binarized using the threshold function in OpenCV to obtain a binary mask. Regions with a value of 1 in the binary mask were defined as automatically identified defect regions. Next, the findContours function was used to extract all contours from the mask, and the boundingRect function was used to calculate the minimum bounding rectangle for each contour to obtain the defect bounding box.

4. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 1, characterized in that, Step (2) includes: (21) Construct a multimodal input template to integrate the text vectors, image vectors and image annotations in the dataset into a template-formatted multimodal input; (22) The multimodal input is fed into the large language model for inference to generate defect detection results and reports; the defect detection results and reports both contain required fields, including defect type, defect level, defect location and confidence level; the defect location is represented by a defect bounding box; (23) Extract and verify whether the generated defect detection results and reports meet the preset pipeline inspection standards and report format requirements. Specifically, check whether the required fields of the defect detection results and reports are complete, and whether the defect detection results and reports are in the preset format. If they do not meet the requirements, the generation is deemed invalid and regenerated.

5. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 4, characterized in that, Step (22) includes: First, the multimodal input from step (21) is loaded in batches and divided into multiple batches. Each batch contains text vectors, image vectors and image annotations, including defect type, defect level, defect bounding box and binary mask. Feature fusion is performed by directly concatenating image vectors and text vectors to form fused features. These fused features are then input into a multilayer perceptron for defect type classification and defect location regression to obtain defect detection results. Input the fused features into the text generation model to generate a report.

6. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 5, characterized in that, The step (22) further includes: training the model before generating defect detection results and reports. During the training process, cross-entropy loss, L1 loss and text generation loss are jointly optimized, wherein cross-entropy loss is used for classification supervision, L1 loss is used for localization supervision, and text generation loss is used for report generation.

7. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 1, characterized in that, Step (3) includes: (31) Validate the defect detection results and reports, including the following: (311) Field integrity check: Check whether the defect detection result contains all required fields; the required fields include: defect type, defect level, defect location and confidence level; (312) Format and numerical validity verification: including confirming that the defect bounding box is a quadruple and the coordinates are within the image size range; confirming that the confidence level is a decimal between 0 and 1; confirming that the report content is not empty; (313) The defect detection results and reports after inspection and verification are compared with manual annotations or expert opinions, and the best result is selected to obtain the verified defect detection results and reports; the comparison method is as follows: when the IoU between the defect bounding box in the defect detection results and reports after inspection and verification and the manually annotated defect bounding box is higher than 0.5, it is determined that the defect detection results and reports are complete and meet the actual needs of the project. (32) Submit the verified defect detection results and reports to the pipeline engineer for manual review and correction, and collect feedback information from the manual review.

8. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 7, characterized in that, S320 also includes, if no problems are found during manual review, adopting the model results and using the feedback information for continuous learning; Otherwise, the model results are manually corrected, and the corrected samples are added to the incremental training queue. The manual review process is then iteratively executed to continuously train the model.

9. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 4, characterized in that, In step (21), the multimodal input template includes two parts: system role definition and user input. The system role definition is used to specify the professional role of the model, and the user input includes context information, defect report content, defect image and detection and report generation instructions.

10. The pipeline defect detection and report generation method based on multimodal alignment and weakly supervised learning according to claim 4, characterized in that, In step (22), the parameter settings of the large language model include temperature parameters, sampling parameters, maximum length parameters, word frequency penalty parameters, and existence penalty parameters.

Images