A manhole cover defect classification method and system
By fine-tuning the visual language model using multiple reward functions and a group relative strategy optimization algorithm, the interpretability and semantic understanding problems of manhole cover defect detection are solved, achieving interpretability and efficient scalability of manhole cover defect detection, which is suitable for municipal facility management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG NORMAL UNIV
- Filing Date
- 2026-03-26
- Publication Date
- 2026-06-09
AI Technical Summary
Existing CNN/YOLO-based manhole cover defect detection solutions suffer from problems such as lack of interpretability of black-box output, lack of semantic understanding, limited generalization ability, need to retrain new categories, and single training paradigm, making it difficult to meet the actual needs of municipal facility management.
A manhole cover defect classification method is adopted. By defining multiple reward functions for classification accuracy, format standardization and description quality, and combining the group relative strategy optimization algorithm and low-rank adaptation technology, the visual language model is fine-tuned and trained to generate an interpretable thought chain reasoning process and classification results.
It achieves interpretability, strong semantic understanding capability, good scalability and robustness in manhole cover defect detection, reduces training resource requirements, and improves the applicability and efficiency of the model in actual deployment.
Smart Images
Figure CN122176409A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically providing a method and system for classifying defects in manhole covers. Background Technology
[0002] Urban manhole covers are an important component of underground pipe networks. Their defects, such as missing, displaced, broken, warped, sunken, or bulging covers, directly threaten the safety of pedestrians and vehicles. Traditional manhole cover inspections rely primarily on manual labor, which suffers from low efficiency, limited coverage, and inconsistent subjective judgment. With the development of smart city and IoT technologies, automated manhole cover defect detection solutions based on computer vision are gradually emerging.
[0003] Current mainstream solutions are based on convolutional neural networks (CNNs) and object detection models (such as the YOLO series) to achieve automatic classification or detection of manhole cover defects through end-to-end supervised learning. These methods have achieved good results in specific scenarios, but they also have a series of inherent limitations.
[0004] Existing CNN / YOLO-based manhole cover defect detection solutions have the following shortcomings: (1) Black-box output, lacking interpretability. Traditional deep learning models directly output category labels or bounding boxes, without explaining the basis for their judgments. In municipal facility management scenarios, inspectors and management departments need to understand the reasons for their judgments to confirm the reliability of the inspection results. A purely black-box output method is difficult to meet the needs of practical applications.
[0005] (2) Lack of semantic understanding. CNN / YOLO models only learn the mapping relationship from images to labels and do not have the ability to understand the semantics of defect types. For example, the model cannot distinguish the semantic difference between "holes caused by missing manhole covers" and "cracks caused by manhole covers being raised", which limits the model's ability to make judgments in boundary cases.
[0006] (3) Limited generalization ability. Traditional classification models are highly dependent on the distribution of training data. When the deployment environment (lighting conditions, road surface material, manhole cover style, etc.) differs from the training data, the model performance drops significantly.
[0007] (4) New category expansion requires retraining. When new defect types emerge or the classification system needs to be adjusted, CNN / YOLO models need to collect labeled data again and retrain, which lacks flexibility.
[0008] (5) Single training paradigm. Existing solutions all adopt the supervised fine-tuning (SFT) training paradigm, which cannot utilize the exploratory optimization of reinforcement learning to improve the model's reasoning ability. Summary of the Invention
[0009] In order to overcome the above-mentioned defects, the present invention is proposed to provide a solution or at least a partial solution to the above-mentioned problems.
[0010] In a first aspect, the present invention provides a method for classifying defects in manhole covers, comprising the following steps: acquiring a dataset of manhole cover defect images and labeling each image with a corresponding defect category label; defining a classification accuracy reward function, a format specification reward function, and a description quality reward function, and calculating the final reward value through a weighted combination; wherein, the classification accuracy reward function is used to evaluate the degree of matching between the classification result output by the visual language model and the defect category label; the format specification reward function is used to evaluate the structural integrity of the thought chain labels and answer labels in the output of the visual language model and the effectiveness of the answer label content, and outputs the reward value using a progressive scoring mechanism with at least three levels of discrimination; the description quality reward function is used to extract the reasoning text within the thought chain labels, match it based on predefined visual feature keyword groups corresponding to each defect category, and output continuous reward values according to the matching ratio; employing a group relative strategy optimization algorithm, with the classification accuracy reward function, format specification reward function, and description quality reward function as optimization objectives, to fine-tune the visual language model; inputting the manhole cover image to be detected into the trained model, obtaining the output text containing the thought chain reasoning process and classification result, and extracting the classification label.
[0011] Preferably, the defect category labels include missing displacement, breakage, warping, and subsidence / protrusion; when acquiring the manhole cover defect image dataset, a balanced sampling mode is used to uniformly and randomly extract the same number of samples from each category, or a full sampling mode is used to use all data and oversample a few categories; and stratified sampling is used to divide the training set and validation set according to a preset ratio to ensure that the distribution ratio of each category in the training set and validation set is consistent.
[0012] Preferably, it also includes preset system prompts and user prompts. The preset system prompts set the visual language model as a municipal facility inspection assistant. The user prompts include image placeholders, names of each defect category and their semantic descriptions, and format instructions requiring the visual language model to organize its output using thought chain tags and answer tags.
[0013] Preferably, the classification accuracy reward function evaluates the degree of matching between the classification result output by the visual language model and the defect category label. Specifically, it uses regular expressions to extract the predicted category from the answer label in the visual language model output, extracts the true category from the defect category label, and the reward value is a first reward value when the predicted category is completely consistent with the true category, otherwise the reward value is a second reward value, and the first reward value is greater than the second reward value.
[0014] Preferably, the progressive scoring mechanism of the format specification reward function determines the reward value according to the following scoring tiers: First tier: When the visual language model output has both complete thought chain label and answer label structures and the answer content belongs to the valid category set, the reward value is the first reward value; Second tier: When the visual language model output has complete thought chain label and answer label structures but the answer content does not belong to the valid category set, the reward value is the second reward value, and the second reward value is less than the first reward value; Third tier: When the visual language model output has only thought chain labels or only one of answer labels, the reward value is the third reward value, and the third reward value is less than the second reward value; Fourth tier: When the visual language model output has no label structure, the reward value is the fourth reward value, and the fourth reward value is less than the third reward value.
[0015] Preferably, the description quality reward function outputs continuous reward values according to the matching ratio, specifically: predefine multiple sets of visual feature keywords for each defect category, each set of keywords containing multiple synonyms; extract the reasoning text within the thought chain tags from the visual language model output, and match the reasoning text with the visual feature keyword groups corresponding to the real category; determine the reward value according to the ratio of the number of matched keyword groups to the total number of keyword groups corresponding to the real category, and the reward value is positively correlated with the ratio.
[0016] Preferably, the formula for calculating the final reward value through weighted combination is: R_total=w1×R_acc+w2×R_fmt+w3×R_desc, where R_acc is the classification accuracy reward value, R_fmt is the format standardization reward value, R_desc is the description quality reward value, w1 is the accuracy reward weight, w2 is the format reward weight, w3 is the description quality reward weight, and w1>w3>w2.
[0017] Preferably, when fine-tuning the visual language model, a low-rank adaptation layer is injected next to the weight matrix of the attention layer of the visual language model. The low-rank adaptation layer includes two low-rank matrices, configured by a rank parameter, a scaling factor, and a regularization rate. During training, the group relative policy optimization algorithm generates multiple candidate answers for each input prompt, normalizes the reward value within the same group of candidate answers to obtain an advantage value, updates the group relative policy optimization algorithm parameters based on the normalized advantage value, and constrains the policy update magnitude through a KL divergence penalty term.
[0018] Preferably, the extraction of category tags adopts a three-level hierarchical extraction strategy: the first level uses regular expressions to extract the content within the answer tags; the second level searches for the first non-whitespace string after the preset keyword when the first level fails; the third level takes the entire output text after removing leading and trailing whitespace when the first two levels fail.
[0019] Secondly, this invention provides an intelligent classification system for manhole cover defects, comprising: an image acquisition module for acquiring a dataset of manhole cover defect images and labeling each image with a corresponding defect category label; and a reward function module for defining a classification accuracy reward function, a format specification reward function, and a description quality reward function, and calculating a final reward value through a weighted combination. The classification accuracy reward function is used to evaluate the degree of matching between the classification result output by the visual language model and the defect category label, and the format specification reward function is used to evaluate the structural integrity of the thought chain labels and answer labels in the visual language model output, as well as the validity of the answer label content, and employs at least... The progressive scoring mechanism with three levels of discrimination outputs reward values. The description quality reward function is used to extract the reasoning text within the thought chain labels and match it based on the visual feature keyword groups corresponding to each predefined defect category, outputting continuous reward values according to the matching ratio. The model training module is used to fine-tune the visual language model using a group relative strategy optimization algorithm, with the classification accuracy reward function, format standardization reward function, and description quality reward function as optimization objectives. The inference engine module is used to input the manhole cover image to be detected into the trained visual language model, obtain the output text containing the thought chain reasoning process and classification results, and extract the classification labels.
[0020] The beneficial effects of this invention are: (1) Interpretable reasoning: Due to the adoption of the thinking chain reasoning mechanism and the multiple guidance of the reward function (formatted reward guides the output structure, and descriptive quality reward guides the reasoning content), the model generates while outputting the classification results. <think>The reasoning process of the label package makes the test results traceable, thus facilitating the confirmation and verification of test results by inspection personnel in municipal management scenarios.
[0021] (2) Triple Reward Optimization Mechanism: A combined reward mechanism of accuracy reward, format standardization reward, and description quality reward is designed. Through weighted combination, classification accuracy, output format standardization, and inference description quality are simultaneously optimized. Therefore, while learning correct classification, the model develops standardized thought process reasoning habits and pays attention to the visual features of the correct category. The description quality reward compensates for the sparse signal of the accuracy reward, enabling the model to distinguish between "correct reasoning direction but wrong answer" and "complete guessing," accelerating training convergence. The progressive scoring design of the format reward avoids the problem of sparse reward signals in the early stages of training.
[0022] (3) Efficient parameter fine-tuning: Due to the use of low-rank adaptation technology to inject a low-rank matrix into the model attention layer, only about 0.006% of the model parameters need to be trained. Therefore, compared with full parameter fine-tuning, about 99.994% of the trainable parameters are saved, which significantly reduces the GPU memory and computing resources required for training, so that training can be completed on a single consumer-grade GPU.
[0023] (4) Strong semantic understanding ability: Due to the cross-modal understanding ability based on the pre-trained visual language model, the model can understand the feature differences of different defect types from the semantic level. Therefore, it has a better judgment ability than traditional classification models when the inter-class boundaries are blurred.
[0024] (5) Good scalability: Since the visual language model-based method naturally supports adjusting or expanding the classification system by modifying the prompt words, adding a new defect type only requires updating the category description in the prompt words and the corresponding small number of training samples, without redesigning the network structure.
[0025] (6) Class imbalance handling: Since two strategies, equal sampling and oversampling, are provided to deal with the class imbalance problem, and the group-based normalization mechanism optimized by the group relative strategy is combined, the training bias caused by insufficient samples of a minority class is effectively alleviated.
[0026] (7) Robust label extraction: Due to the adoption of a three-level hierarchical extraction strategy (answer label → keyword → full text catch), effective classification results can still be extracted as much as possible even when the model output format is imperfect, which improves the robustness of the system in actual deployment.
[0027] (8) Complete end-to-end pipeline: Because it provides a complete technical pipeline from data preparation, reward function design, model training, inference engine to evaluation and analysis, as well as a complete system architecture from image acquisition to alarm scheduling, it is easy to implement in engineering. Attached Figure Description
[0028] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein: Figure 1 This is a flowchart illustrating an embodiment of the intelligent classification method for manhole cover defects according to the present invention. Figure 2 A detailed flowchart of the GRPO training loop according to an embodiment of the present invention; Figure 3 This is a scoring diagram of the triple reward function according to an embodiment of the present invention; Figure 4 This is an architecture diagram of the intelligent classification system for manhole cover defects according to an embodiment of the present invention. Detailed Implementation
[0029] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0030] Example 1 like Figure 1-3 As shown, the present invention provides a method for classifying defects in manhole covers, comprising the following steps: S1. Obtain the dataset of manhole cover defect images and label each image with the corresponding defect category label.
[0031] In this embodiment, a dataset of manhole cover defect images is collected, folder names are mapped to standard category labels, samples of each category are sampled and hierarchically divided, and each sample is converted into training data containing image paths, dialogue messages and real labels by combining preset system prompts and user prompts.
[0032] Specifically: The collected images of manhole cover defects include 4 types of defects, totaling 1,536 images, distributed as follows:
[0033] The class imbalance ratio of the dataset is 7.45:1 (maximum class 931 / minimum class 125), which is considered a moderate imbalance scenario.
[0034] Folder names are mapped to standard category labels based on predefined category mapping relationships. CLASS_MAP = { "1": "Missing and Displaced", "2": "Damaged", "3": "Limped", "4": "Sunken and Protruding",} The categories include missing, shifted, broken, warped, and sunken / protruding. Sample processing employs either a balanced sampling mode or a full sampling mode. The balanced sampling mode uniformly and randomly selects the same number of samples from each category (defaulting to the smallest category with 125 samples) to ensure consistent training data across categories. The full sampling mode uses all data and optionally oversamples a minority of classes. The oversampling method involves counting the number of samples in each category, targeting the largest category, and padding any insufficient categories with random repeated sampling to reach the maximum category count.
[0035] A stratified sampling method was used to divide the data into training and validation sets at a preset ratio of 80:20, ensuring that the distribution ratio of each category was consistent between the training and validation sets.
[0036] Example of data partitioning under balanced sampling mode: Training set: 400 records (100 records per class) Validation set: 100 records (25 records per class) Example of data partitioning under full oversampling mode: Training set (before oversampling): approximately 1,228 records Training set (after oversampling): approximately 2,976 records (approximately 744 records per class). Validation set: approximately 308 records Each sample is converted into a standardized format containing a list of image paths, a list of dialogue messages, and ground truth labels. Each sample is then converted into JSONL format required for group relative policy optimization training, with each data entry containing three fields: images: A list of absolute paths to images; messages: A list of conversation messages containing system prompts and user prompts; Solution: Wrapped in <answer>The true category label in the label is used as the basis for evaluating the reward function.
[0037] The system prompts define the model's role as a municipal facility inspection assistant. The user prompts include image placeholder markers, names of each defect category and their semantic descriptions, and format instructions requiring the model to organize its output using thought chain tags and answer tags. The semantic descriptions provide reference information for each category.
[0038] The system prompt is defined as follows: "You are a professional municipal infrastructure inspection assistant, skilled at identifying types of defects in manhole covers." User prompts are defined as: \<image\> Please carefully observe this picture of the manhole cover and determine the type of defect. Optional categories: Missing or Displaced Manhole Covers: Missing or displaced manhole covers, clearly revealing an empty manhole, pose the greatest safety hazard. Damage: The manhole cover is still there, but it is damaged or cracked. Limping: The manhole cover is not damaged, but it is tilted and uneven. Subsidence / Protrusion: If the manhole cover is undamaged but protrudes or is sunken into the ground, or if the surrounding road surface is damaged, please contact us.<think\>< / think\> Write down your analysis process in the text.<answer\>< / answer\> The final category is given in the text. In the user suggestion words, Image placeholders are used, which are replaced by image visual features during inference by the visual language model; each category label is accompanied by a semantic description, providing the model with a reference for category judgment. The system prompt sets the model's role as a "professional municipal facility inspection assistant," activating the model's domain knowledge.
[0039] The following is a sample of the actual generated training data: { "images": ["dataset / 1 / example.jpg"], "messages": [ { "role": "system", "content": "You are a professional municipal facility inspection assistant, skilled in identifying manhole cover defect types."},{ "role": "user", "content": " Please carefully observe this manhole cover image and determine the type of defect. Optional categories: - Missing or Displaced: The manhole cover is missing or displaced; an empty manhole is clearly visible, posing the greatest safety hazard. - Damaged: The manhole cover is still there, but it is damaged or cracked. - Limped: The manhole cover is not damaged, but it is uneven and tilted. - Sunken or Protruding: The manhole cover is not damaged, but it protrudes or dents into the ground, or the surrounding road surface is damaged. Please... <think>< / think> Write down your analysis process in the text. <answer>< / answer> The final category is given in [the code]. <answer> Missing shift< / answer> "} S2. Define the classification accuracy reward function, the format specification reward function, and the description quality reward function, and calculate the final reward value by weighted combination.
[0040] In this embodiment, the scoring process of the classification accuracy reward function is as follows: the predicted category within the answer label is extracted from the model output using a regular expression, and the true category is extracted from the true label using the same regular expression. The reward value is 1.0 when the predicted category is completely consistent with the true category, and 0.0 otherwise.
[0041] The format-specific reward function employs a progressive scoring mechanism, specifically: for each generated result of the model, it is checked whether it conforms to... <think> ...< / think> <answer> ...< / answer> The system outputs a thought process chain format and verifies whether the answer tags are valid categories. The set of valid categories is {missing / shifted, broken, warped, sunken / protruding}. A progressive scoring mechanism is employed. Possess complete <think>and <answer>Tag structure, and <answer>The content belongs to the valid category set → Reward 1.0; It has a complete tag structure, but <answer>Content not in a valid category → Reward 0.5; Only some tags ( <think>or <answer>(One of them) → Reward 0.25; No tag structure → Reward 0.0
[0042] Describe the quality reward function: for each generated result of the model, extract... <think>The thought process text within the tag is checked for the presence of visual descriptive keywords relevant to the actual category. Five sets of visual feature keywords are predefined for each defect category, each set containing multiple synonyms. A continuous reward of 0.0 to 1.0 is awarded based on the keyword set matching ratio. Match all 5 sets of keywords and receive a reward of 1.0. Match 3 sets of keywords, reward 0.6; A reward of 0.2 is given for each matching keyword. none <think>The tag did not match any keywords, reward 0.0.
[0043] The keywords are matched only with keyword groups corresponding to the true category, preventing the model from obtaining high scores by stuffing keywords from all categories. The predefined visual feature keyword groups for each category are as follows:
[0044] The formula for calculating the final reward value by weighted combination of the three reward functions is as follows: R_total= w1 × R_acc + w2 × R_fmt + w3 × R_desc Where R_acc is the classification accuracy reward, R_fmt is the format standardization reward, R_desc is the description quality reward, w1 is the accuracy reward weight, w2 is the format reward weight, and w3 is the description quality reward weight, with w1>w3>w2. For example, the accuracy reward weight is 1.0, the format reward weight is 0.3, and the description quality reward weight is 0.4. The accuracy reward is dominant, ensuring that the model learns the correct classification first. The description quality reward provides dense, continuous gradient signals, guiding the model to focus on the visual features of the correct category during inference; the format reward serves as an auxiliary, guiding the model to form a standard thought chain output structure. The weight design ensures that the total reward for pure keyword stuffing (at most 0.3+0.4=0.7) is still lower than that for correct classification (1.0), preventing the model from relying on description to cheat and neglecting classification accuracy.
[0045] S3. Model training steps: Inject a low-rank adaptation (LoRA) layer into the pre-trained visual language model, train it using the group relative policy optimization algorithm, generate multiple candidate answers for each training prompt, calculate the combined reward using the triple reward function and normalize it within the group, and update the policy model parameters based on the normalized advantage value.
[0046] In this embodiment, Qwen3-VL-2B-Instruct is selected as the base model. This model has a multimodal architecture of visual encoder and language model, supports cross-modal understanding of image and text, has a parameter scale of about 2 billion, and is a lightweight visual language model, suitable for fine-tuning under limited computing resources.
[0047] Two low-rank matrices A and B are injected next to the weight matrix of the attention layer of the base model, where the dimension of A is d×r and the dimension of B is r×d, and r is much smaller than d, so that the number of trainable parameters accounts for only a small part of the total number of parameters. The low-rank adaptation layer is configured through three hyperparameters: rank parameter, scaling factor and regularization rate.
[0048] With the above configuration, the number of trainable parameters accounts for only about 0.006% of the total parameters, which greatly reduces the GPU memory and computing resources required for training.
[0049] The training process of the group relative policy optimization algorithm specifically includes: (a) Group generation: For each input prompt, generate G using the current policy model. （ For example, G=8 ） There are 100 candidate answers. The generation temperature is set to 1.0 to ensure sufficient diversity of candidate answers, and the maximum generation length is 512 tokens.
[0050] (b) Reward calculation: For each candidate answer, the combined reward is calculated using the triple reward function described above; (c) Within-group normalization: Within the G responses of the same group, the reward values are normalized by the mean and standard deviation to obtain the normalized advantage value. The normalized advantage value reflects the relative merit of each response within the group, so that the reward signal is not affected by the magnitude of the absolute reward value.
[0051] (d) Policy optimization: Update the policy model parameters using the normalized advantage value, while constraining the policy update magnitude through the KL divergence penalty term; constrain the policy model to not deviate too far from the reference model (initial policy) to prevent policy collapse. Smaller values encourage a larger policy exploration space.
[0052] The training requires that the batch size be divisible by the number of prompts generated, G.
[0053] S4. Reasoning and Label Extraction Steps: The image of the manhole cover to be detected is combined with the prompt words and input into the trained model for reasoning. A multi-level extraction strategy is used to obtain classification labels from the text generated by the model.
[0054] In this embodiment, the multi-level extraction strategy includes three levels of extraction: the first level uses regular expressions to extract the content within the answer tags; the second level searches for the first non-whitespace string after the preset keyword when the first level fails; and the third level takes the entire output text after removing leading and trailing whitespace when the first two levels fail.
[0055] S5. Evaluation steps: Calculate the precision, recall and F1 score for each category of the inference results, generate a confusion matrix, and detect invalid prediction categories.
[0056] In this embodiment, the evaluation steps specifically include: calculating the overall accuracy and the precision, recall, and F1 score of each category; calculating the macro average of all category indicators; generating a confusion matrix and outputting it in both text table and heatmap formats; automatically detecting and statistically analyzing prediction results that do not belong to the valid category set {missing shift, broken, warped, sunken protrusion}; and identifying abnormal output formats of the model.
[0057] Example 2 like Figure 4 As shown, the present invention provides an intelligent classification system for manhole cover defects, comprising: The image acquisition module is used to acquire a dataset of images of defects in manhole covers and to label each image with the corresponding defect category label; The reward function module defines a classification accuracy reward function, a format specification reward function, and a description quality reward function, and calculates the final reward value through a weighted combination. The classification accuracy reward function evaluates the degree of matching between the model's output classification results and the defect category labels. The format specification reward function evaluates the structural integrity of the thought chain labels and answer labels in the model output and the effectiveness of the answer label content, and outputs the reward value using a progressive scoring mechanism with at least three levels of discrimination. The description quality reward function extracts the reasoning text within the thought chain labels and matches it based on predefined visual feature keyword groups corresponding to each defect category, and outputs continuous reward values according to the matching ratio. The model training module is used to fine-tune the visual language model using a group relative strategy optimization algorithm, with the classification accuracy reward function, format normalization reward function and description quality reward function as optimization objectives. The inference engine module is used to input the image of the manhole cover to be detected into the trained model, obtain the output text containing the reasoning process and classification results, and extract the classification label.
[0058] Furthermore, the system also includes a data preprocessing module for converting acquired images into an input format acceptable to the model, including category label mapping, sampling processing, dataset partitioning, and training data format generation. Specifically, it performs the following operations: scanning the dataset directory structure and mapping folder names to standard category labels; performing balanced sampling or full data processing on samples of each category; using stratified sampling to partition the training and validation sets; converting each sample into standardized format data containing image paths, dialogue messages, and ground truth labels; and performing field integrity checks, image existence checks, and category distribution statistical verification on the generated data.
[0059] Furthermore, the reward function module includes a classification accuracy reward component, a format standardization reward component, and a description quality reward component. The classification accuracy reward component evaluates the classification correctness through exact matching and outputs a binary reward. The format standardization reward component evaluates the output format standardization through progressive scoring and outputs a multi-level reward. The description quality reward component evaluates the quality of reasoning description during the thinking process through predefined category visual feature keyword group matching and outputs a continuous reward. The three reward components are registered through the reward model interface of the training framework and are automatically called during training.
[0060] Furthermore, the model training module is specifically used to perform the following operations: check GPU availability and data file integrity, verify batch constraints of population relative policy optimization, call the training framework to execute the training process of population relative policy optimization combined with low-rank adaptation, save model checkpoints at preset step intervals and retain the most recent checkpoints, and record training logs.
[0061] Furthermore, the inference engine module is specifically used to perform the following operations: load the trained model checkpoints, combine the input image with system prompts and user prompts to construct a standardized inference request, perform forward inference of the model to generate text containing the thought chain inference process, use a three-level hierarchical label extraction strategy to obtain the final classification result, and support two modes: single image inference and batch validation set inference.
[0062] Furthermore, the system also includes an evaluation module for multi-dimensional quantitative evaluation of inference results, generating evaluation metrics and visualization charts. Specifically, it performs the following operations: calculating overall accuracy and precision, recall, and F1 score for each category; calculating macro-average metrics; generating a text table and heatmap of the confusion matrix; generating visual bar charts for each category metric; and detecting and statistically analyzing invalid prediction categories.
[0063] Furthermore, the system also includes an alarm and scheduling module, which converts defect classification results into hierarchical alarms and performs work order dispatch and status management. Specifically, it performs the following operations: generates hierarchical alarms based on defect type and severity, pushes alarm information to the municipal management platform or mobile terminal, automatically dispatches work orders by region and defect type, maintains the manhole cover status database to support historical queries and trend analysis, and provides API interfaces for integration with third-party systems.
[0064] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the original technical features, and the technical solutions resulting from these changes or substitutions will all fall within the scope of protection of the present invention.< / think> < / think> < / answer> < / think> < / answer> < / answer> < / answer> < / think> < / answer> < / think>
Claims
1. A method for classifying defects in manhole covers, characterized in that, Includes the following steps: Obtain a dataset of manhole cover defect images and label each image with the corresponding defect category label; Define a classification accuracy reward function, a format specification reward function, and a description quality reward function, and calculate the final reward value by weighted combination; The classification accuracy reward function is used to evaluate the degree of matching between the classification results output by the visual language model and the defect category labels; the format specification reward function is used to evaluate the structural integrity of the thought chain labels and answer labels in the output of the visual language model and the effectiveness of the answer label content, and outputs reward values using a progressive scoring mechanism with at least three levels of discrimination; the description quality reward function is used to extract the reasoning text within the thought chain labels, match it based on the predefined visual feature keyword groups corresponding to each defect category, and output continuous reward values according to the matching ratio. A group relative strategy optimization algorithm is adopted, with the classification accuracy reward function, format normalization reward function and description quality reward function as optimization objectives, to fine-tune the visual language model. The image of the manhole cover to be detected is input into the trained visual language model to obtain the output text containing the thought chain reasoning process and classification results, and the classification label is extracted.
2. The method according to claim 1, characterized in that, The defect category labels include missing, misaligned, broken, warped, and sunken / protruding. When acquiring the manhole cover defect image dataset, a balanced sampling mode is used to uniformly and randomly extract the same number of samples from each category, or a full sampling mode is used to use all data and oversample a minority of classes. Stratified sampling is used to divide the training set and validation set according to a preset ratio to ensure that the distribution ratio of each category in the training set and validation set is consistent.
3. The method according to claim 1, characterized in that, It also includes preset system prompts and user prompts. The preset system prompts set the visual language model as a municipal facility inspection assistant. The user prompts include image placeholders, names of each defect category and their semantic descriptions, and format instructions that require the visual language model to organize its output using thought chain tags and answer tags.
4. The method according to claim 1, characterized in that, The classification accuracy reward function evaluates the degree of matching between the classification result output by the visual language model and the defect category label. Specifically, it uses regular expressions to extract the predicted category from the answer label in the visual language model output, extracts the true category from the defect category label, and the reward value is the first reward value when the predicted category is completely consistent with the true category, otherwise the reward value is the second reward value, and the first reward value is greater than the second reward value.
5. The method according to claim 1, characterized in that, The progressive scoring mechanism of the specified reward function determines the reward value according to the following scoring ladder: First tier: When the visual language model output has both a complete thought chain label and answer label structure and the answer content belongs to the valid category set, the reward value is the first reward value; Second tier: When the visual language model outputs a complete thought chain label and answer label structure but the answer content does not belong to the valid category set, the reward value is the second reward value, and the second reward value is less than the first reward value; Third tier: When the visual language model output has only one of the thought chain label or only one of the answer label, the reward value is the third reward value, and the third reward value is less than the second reward value; Fourth step: When the visual language model outputs no label structure, the reward value is the fourth reward value, and the fourth reward value is less than the third reward value.
6. The method according to claim 1, characterized in that, The quality reward function describes the output of continuous reward values according to the matching ratio, specifically: For each defect category, multiple sets of visual feature keywords are predefined, and each set of keywords contains multiple synonyms; Extract the reasoning text within the thought chain tags from the output of the visual language model, and match the reasoning text with the visual feature keyword groups corresponding to the real categories; The reward value is determined based on the proportion of the number of matched keyword groups to the total number of corresponding keyword groups in the actual category, and the reward value is positively correlated with the proportion.
7. The method according to claim 1, characterized in that, The formula for calculating the final reward value through weighted combination is as follows: R_total=w1×R_acc+w2×R_fmt+w3×R_desc Where R_acc is the classification accuracy bonus, R_fmt is the format specification bonus, R_desc is the description quality bonus, w1 is the accuracy bonus weight, w2 is the format bonus weight, and w3 is the description quality bonus weight, and w1 > w3 > w2.
8. The method according to claim 1, characterized in that, When fine-tuning the visual language model, a low-rank adaptation layer is injected next to the weight matrix of the attention layer of the visual language model. The low-rank adaptation layer includes two low-rank matrices, which are configured by rank parameter, scaling factor and regularization rate. During training, the group relative policy optimization algorithm generates multiple candidate answers for each input prompt. Within the same group of candidate answers, the reward value is normalized to obtain the advantage value. The parameters of the group relative policy optimization algorithm are updated based on the normalized advantage value, and the update magnitude of the group relative policy is constrained by the KL divergence penalty term.
9. The method according to claim 1, characterized in that, The extraction of category tags adopts a three-level hierarchical extraction strategy: the first level uses regular expressions to extract the content within the answer tags; the second level searches for the first non-whitespace string after the preset keyword when the first level fails; the third level takes the entire output text after removing leading and trailing whitespace when the first two levels fail.
10. A manhole cover defect intelligent classification system, characterized in that, include: The image acquisition module is used to acquire a dataset of images of defects in manhole covers and to label each image with the corresponding defect category label; The reward function module defines a classification accuracy reward function, a format specification reward function, and a description quality reward function, and calculates the final reward value through a weighted combination. The classification accuracy reward function evaluates the degree of matching between the classification results output by the visual language model and the defect category labels. The format specification reward function evaluates the structural integrity of the thought chain labels and answer labels in the visual language model output and the effectiveness of the answer label content, and outputs the reward value using a progressive scoring mechanism with at least three levels of discrimination. The description quality reward function extracts the reasoning text within the thought chain labels and matches it based on predefined visual feature keyword groups corresponding to each defect category, and outputs continuous reward values according to the matching ratio. The model training module is used to fine-tune the visual language model using a group relative strategy optimization algorithm, with the classification accuracy reward function, format normalization reward function and description quality reward function as optimization objectives. The inference engine module is used to input the image of the manhole cover to be detected into the trained visual language model, obtain the output text containing the reasoning process and classification results, and extract the classification labels.