Well lid category diagnosis method and system based on semantic guidance and state space modeling, storage medium and device

Abstract

The invention discloses a manhole cover category diagnosis method, system and device based on semantic guidance and state space modeling, and a storage medium, and the method comprises the following steps: S1, obtaining a training image, and carrying out the preprocessing of the training image, and obtaining a preprocessed training image; s2, training a category diagnosis model based on the preprocessed training image to obtain a trained category diagnosis model; s3, inputting an image to be detected into the trained category diagnosis model; s4, performing multi-scale layered analysis on the input image based on a self-supervised visual encoder DINOv3 to obtain high-resolution image block features; and S5, based on a visual language model, introducing a learnable prompt word template, and mapping the well lid state description into a high-dimensional text semantic vector. The manhole cover state identification method can solve the problems of low manhole cover state identification accuracy and poor environmental adaptability caused by non-robust feature expression, lack of semantic understanding and insufficient long-range dependency relationship modeling in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer vision and intelligent inspection technology, specifically to a method, system, storage medium, and device for diagnosing manhole cover categories based on semantic guidance and state space modeling. Background Technology

[0002] As urban road infrastructure ages and traffic volume increases, the frequency of damaged, displaced, or missing manhole covers is gradually rising. These abnormalities not only pose safety hazards to motor vehicles and pedestrians but can also lead to serious consequences such as drainage system failure and road collapse. Therefore, timely and accurate diagnosis of manhole cover conditions is crucial for ensuring road traffic safety and the stable operation of municipal facilities.

[0003] In recent years, drone inspection technology, with its advantages of high mobility, wide coverage, and fast response speed, has been gradually applied to the intelligent inspection of municipal infrastructure. However, the automated recognition of manhole cover images collected by drones still faces significant challenges, and existing methods mainly have the following limitations: First, the feature representation capability is insufficient, and the environmental robustness and generalization performance are poor. Because manhole covers in aerial images are small in size, the appearance differences between different state categories are subtle, and they are easily affected by environmental factors such as road texture, lighting changes, shadow occlusion, and dirt, leading to a decrease in target recognizability. Traditional image classification models based on convolutional neural networks struggle to extract stable and robust feature representations under such complex conditions, and their recognition accuracy is easily affected by environmental fluctuations. Especially in drone inspections, manhole covers exhibit significant differences in posture, lighting, and angle. Traditional vision models are prone to feature drift when migrating across different areas (such as different road materials or different capture heights), resulting in a significant decrease in classification performance.

[0004] Secondly, the adaptability to dynamic environments and the ability to identify fine-grained states are weak. Images of manhole covers captured by drones are significantly affected by dynamic factors such as weather (rain, snow, fog, backlight), time (day / night), and obstructions (leaves, vehicles, road stains). Because traditional convolutional networks and visual Transformers lack the ability to model dynamic spatiotemporal heterogeneity, their visual features are easily affected by noise; especially fine-grained states such as damage and displacement, where differences often lie in small areas such as crack details, boundary displacement, and local structural deformation. Traditional visual features struggle to reliably capture these, further reducing diagnostic reliability. Furthermore, different abnormal states of manhole covers have different feature dependencies. For example, judging a "missing" state relies more on the overall scene and contextual structure information, while states such as "damaged" and "displaced" rely more on local details, geometric shifts, or subtle texture changes. Existing single-scale or single-type feature extraction methods struggle to simultaneously and effectively capture both global semantic information and local detail features, failing to meet the needs of different state discrimination.

[0005] Secondly, semantic understanding and cross-modal association capabilities are lacking. Traditional single-modal vision models struggle to understand abstract semantic concepts such as "damage" and "displacement," failing to directly link pixel-level visual features to higher-level state semantics, leading to misclassifications in multi-scene and complex background conditions. States like manhole cover displacement and missing parts are semantic-level concepts, but existing vision models, relying solely on pixel-level features, struggle to establish semantic associations with manually defined states. The lack of cross-modal semantic constraints causes the model to easily confuse fine-grained categories; for example, the states of damage and displacement are frequently confused, affecting the final classification accuracy.

[0006] Finally, the modeling ability for global structural relationships is insufficient and computational efficiency is low. Determining the state of manhole covers depends not only on local texture (such as local damage or cracks) but also on global geometric structures (such as the surrounding ground conditions and the boundaries of missing areas). Traditional CNNs struggle to capture long-range dependencies, while existing Transformers, although possessing some long-range modeling capabilities, suffer from computational complexity that increases quadratically with sequence length due to their self-attention mechanism. This makes efficient modeling on high-resolution UAV inspection images difficult, limiting their large-scale deployment in UAV inspection scenarios. Existing deep learning models have limitations in modeling long-range dependencies in images, failing to fully capture the spatial and structural relationships between the manhole cover area and its surrounding environment, thus affecting the overall accuracy of state classification.

[0007] In summary, existing manhole cover status recognition methods are insufficient in terms of feature representation capabilities in complex environments, fine-grained status identification capabilities, dynamic environment adaptability, cross-modal semantic understanding capabilities, and efficient modeling capabilities for global long-range dependencies. These limitations fail to meet the practical needs of intelligent urban road inspection for high-precision and robust automated diagnosis of manhole cover status. Therefore, a new method integrating robust visual features, deep semantic information, and efficient long-sequence modeling capabilities is urgently needed to improve the accuracy, robustness, and practical feasibility of automated manhole cover status classification in UAV inspection scenarios. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, one of the objectives of this invention is to provide a manhole cover category diagnosis method based on semantic guidance and state space modeling, so as to solve the problems of low accuracy and poor environmental adaptability of manhole cover state recognition caused by non-robust feature representation, lack of semantic understanding and insufficient modeling of long-range dependencies in existing technologies.

[0009] The second objective of this invention is to provide a manhole cover category diagnosis system based on semantic guidance and state space modeling, so as to solve the problems of low accuracy and poor environmental adaptability of manhole cover state recognition caused by non-robust feature representation, lack of semantic understanding and insufficient modeling of long-range dependencies in the prior art.

[0010] The third objective of this invention is to provide a storage medium to solve the problems of low accuracy and poor environmental adaptability in manhole cover status recognition caused by non-robust feature representation, lack of semantic understanding, and insufficient modeling of long-range dependencies in the prior art.

[0011] The fourth objective of this invention is to provide a manhole cover category diagnostic device based on semantic guidance and state space modeling, so as to solve the problems of low accuracy and poor environmental adaptability of manhole cover state recognition caused by non-robust feature representation, lack of semantic understanding and insufficient modeling of long-range dependencies in the prior art.

[0012] To achieve one of the objectives of this invention, the following solution is adopted: The manhole cover category diagnosis method based on semantic guidance and state space modeling includes the following steps: S1. Acquire training images and perform preprocessing to obtain preprocessed training images; S2. The category diagnosis model is trained based on the preprocessed training images to obtain the trained category diagnosis model; S3. Input the image to be tested into the trained category diagnostic model; S4. Multi-scale hierarchical analysis of the input image is performed based on the self-supervised visual encoder DINOv3 to obtain high-resolution image patch features; S5. Based on the visual language model, learnable prompt word templates are introduced to map the manhole cover status description into a high-dimensional text semantic vector. S6. Based on the bidirectional cross-attention module, the high-resolution image patch features and the high-dimensional text semantic vector are semantically interacted to obtain multimodal fusion features; S7. Based on the selective state-space model Mamba, the multimodal fusion features are serialized and dynamically fused to output a globally consistent final representation vector. S8. Classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

[0013] Furthermore, in step S1, preprocessing the training images includes: performing data augmentation on the training images, wherein the data augmentation includes reducing image sharpness to simulate a long-distance shooting scene and adding occlusions.

[0014] Furthermore, in step S5, the learnable prompt word template is dynamically generated through gradient learnable parameters to enhance the distinguishability between the semantics of texts with different manhole cover states.

[0015] Further, in step S6, the bidirectional cross-attention module includes: Image-to-text attention mechanism is used to focus visual features based on text information to generate text-guided visual focus features; A text-to-image attention mechanism is used to enhance text features based on visual information to generate visually guided text enhancement features.

[0016] Furthermore, in step S7, the selective state-space model Mamba achieves dynamic fusion of cross-scale features through a selective interactive gating mechanism, thereby improving the ability to capture details of damage, gaps, and displacement.

[0017] Furthermore, in step S8: The dynamic prototype network achieves robust classification of complex manhole cover states by updating the category prototype vector online. The uncertainty-aware classification head estimates the uncertainty of samples based on temperature scaling and energy function, thereby improving the reliability of abnormal state identification.

[0018] Further, in step S8, the obtained state category of the manhole cover is a diagnostic result that includes the state type and corresponding confidence level. The state categories of the manhole cover include normal, damaged, displaced, and missing. And / or, after step S8, the method further includes: S9, filtering the state category results of the manhole cover using a non-maximum suppression algorithm, removing redundant and highly overlapping results, and outputting the final optimized diagnostic results; And / or, deploy the trained category diagnostic model to a drone or mobile terminal device.

[0019] To achieve the second objective of this invention, the following solution is adopted: A manhole cover category diagnostic system based on semantic guidance and state space modeling includes: The training data processing unit is used to acquire training images and perform preprocessing to obtain preprocessed training images. The model training unit is used to train the category diagnosis model based on the preprocessed training images to obtain the trained category diagnosis model. The model application unit is used to input the image to be tested into the trained category diagnosis model; The visual coding unit is used to perform multi-scale hierarchical parsing of the input image based on the self-supervised visual encoder DINOv3 to obtain high-resolution image patch features. The text encoding unit is used to map the manhole cover status description into a high-dimensional text semantic vector based on a visual language model and by introducing learnable prompt word templates. A multimodal interaction unit is used to semantically interact the high-resolution image patch features with the high-dimensional text semantic vector based on a bidirectional cross-attention module to obtain multimodal fusion features; The state-space fusion unit is used to perform serialized state modeling and dynamic fusion of the multimodal fusion features based on the selective state-space model Mamba, and output a globally consistent final representation vector. The classification decision unit is used to classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

[0020] To achieve the third objective of this invention, the following solution is adopted: A storage medium, which is a computer-readable storage medium, stores a computer program thereon, the computer program being a manhole cover category diagnosis program, which, when executed by a processor, implements the steps of the manhole cover category diagnosis method based on semantic guidance and state space modeling as described in one of the objectives of this invention.

[0021] To achieve the fourth objective of this invention, the following solution is adopted: The manhole cover category diagnosis device based on semantic guidance and state space modeling includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is a manhole cover category diagnosis program. When the processor executes the manhole cover category diagnosis program, it implements the steps of the manhole cover category diagnosis method based on semantic guidance and state space modeling as described in one of the objectives of this invention.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention improves feature representation and recognition accuracy in complex environments. By employing a self-supervised visual encoder based on DINOv3 for multi-scale hierarchical parsing, it can simultaneously capture local detail features (such as cracks and textures) and global structural context of manhole cover images. This overcomes the limitations of traditional single-scale feature extraction methods in dealing with different states such as "damage" and "displacement" that depend on local details and "missing" that depend on the global scene. Thus, it can still obtain stable and robust feature representations in complex environments such as road surface texture interference and lighting changes, laying the foundation for high-precision classification.

[0023] 2. This invention enhances the model's ability to understand and align abstract state semantics. By introducing a visual language model and learnable cue word templates, abstract state concepts such as "normal," "damaged," "displaced," and "missing" are transformed into high-dimensional textual semantic vectors. A bidirectional cross-attention module is then used to achieve deep interaction between image features and textual semantics. This process enables semantically guided visual feature focusing, allowing the model to understand and associate high-level state semantics with underlying visual representations. This effectively solves the problem that traditional single-modal visual models struggle to understand abstract semantic concepts and are prone to misjudgment in multiple scenarios.

[0024] 3. This invention enhances the modeling capability for long-range dependencies and global structural relationships. The selective state-space model Mamba is employed to perform sequential state modeling and dynamic fusion of the fused multimodal features. This model can efficiently capture the long-range spatial dependencies and structural associations between the manhole cover area and its surrounding environment, forming a globally consistent representation vector. This overcomes the shortcomings of traditional CNNs in long-range modeling capabilities and the high computational complexity of Transformers on high-resolution images, improving the accuracy of judging the overall state of the manhole cover (especially the "missing" state dependent on the surrounding environment).

[0025] 4. This invention improves the robustness, reliability, and practicality of the diagnostic system. By combining a dynamic prototype network with an uncertainty-aware classification head, fine-grained and robust classification is achieved. The dynamic prototype network can optimize category representations online to adapt to sample distributions in different scenarios; the uncertainty-aware mechanism can assess prediction confidence, identify low-confidence samples, and trigger manual review. This enables the system to maintain high performance not only in challenging scenarios such as occlusion and blurring, but also to possess self-evaluation capabilities, improving its reliability and practicality in actual deployment.

[0026] 5. This invention forms a complete, efficient, and deployable solution. The method, from data preprocessing, multimodal feature extraction and fusion, to advanced sequence modeling and classification decision-making, constitutes a complete automated diagnostic process. This solution effectively integrates the advantages of cutting-edge technologies such as self-supervised learning, visual-language models, and state-space models. While improving performance, the high efficiency of components such as Mamba also ensures the computational feasibility of real-time model operation on drones or edge devices, meeting the comprehensive requirements of high precision, high efficiency, and strong deployability for intelligent urban road inspection. Attached Figure Description

[0027] Figure 1 This is a flowchart of the manhole cover category diagnosis method based on semantic guidance and state space modeling in an embodiment of the present invention; Figure 2 This is a block diagram of the manhole cover category diagnosis system based on semantic guidance and state space modeling in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0029] Example 1 This invention provides a manhole cover category diagnosis method based on semantic guidance and state space modeling. It is a manhole cover state diagnosis method that integrates semantic guidance, sequence state modeling and reliable classification mechanism. It realizes semantic guidance of text to key visual regions through visual language model, uses state space network to model the long-range dependency relationship of multimodal features, and combines dynamic prototype matching and uncertainty perception mechanism to achieve high confidence category discrimination, which significantly improves the accuracy and system reliability of manhole cover category diagnosis in complex scenarios such as occlusion and long distance.

[0030] like Figure 1 As shown in the figure, the manhole cover category diagnosis method based on semantic guidance and state space modeling in this invention includes the following steps: S1. Acquire training images and perform preprocessing to obtain preprocessed training images.

[0031] Understandably, urban manhole cover status recognition often faces complex environmental interference, such as partial occlusion, long-distance shooting, strong reflection, and uneven lighting, leading to the degradation or loss of key visual features. Traditional methods based on single-modal analysis struggle to reliably obtain target features, easily resulting in misjudgments and missed detections. Furthermore, existing models have limited adaptability to diverse scenarios (urban main roads, sidewalks, green belts) and insufficient generalization performance.

[0032] Therefore, in some feasible embodiments, a dynamic data augmentation strategy for real inspection scenarios is introduced, which improves the generalization ability of the model by synthesizing training samples that cover various real-world scene interferences (such as rain and fog obscuration, partial coverage).

[0033] Specifically, the team collected a dataset of manhole cover images covering various typical scenarios, including main urban roads, sidewalks, and underground pipeline entrances, and labeled each image with its status category: normal, missing, damaged, and displaced. The specific data augmentation process included: (1) Simulation of long-distance shooting: Based on different shooting distances and angles, adjust the image contrast and color saturation to approximate the image quality degradation under long-distance shooting; (2) Local occlusion simulation: randomize the foreground occlusion objects (such as fallen leaves, plastic bags, water reflection areas) that are not of the target category, and overlay them on the key areas of the manhole cover with controllable transparency to simulate the actual coverage situation.

[0034] The enhancement process employs Albumentations (an image processing tool), dynamically combining multiple perturbation types and their parameter distributions in each training iteration to ensure the model encounters as many degradation mode combinations as possible. After training, model performance is evaluated using an independent test set containing real-world complex scene images not used in training (e.g., nighttime images, heavily occluded images, and distant small targets). Experimental results show that the model trained with this enhancement strategy exhibits higher recognition integrity and classification consistency under various perturbation conditions, significantly reducing the false negative and false positive rates.

[0035] S2. The category diagnosis model is trained based on the preprocessed training images to obtain the trained category diagnosis model.

[0036] S3. Input the image to be tested into the trained category diagnostic model.

[0037] In some feasible embodiments, pre-trained models and knowledge distillation are utilized to implement the present invention, specifically including the following: To enhance feature learning capabilities, a pre-trained model is introduced: Before training the manhole cover category diagnostic model, a self-supervised pre-training strategy on large-scale datasets such as ImageNet is used to initialize the feature extraction backbone network. Through mask reconstruction or contrastive learning, the model acquires richer and more stable visual representation capabilities under unsupervised conditions, providing a better feature foundation for subsequent manhole cover status recognition tasks.

[0038] A knowledge distillation mechanism is employed to enhance the model's generalization performance: a higher-performance, more accurate teacher model is constructed and used as the guiding network during training. While learning real manhole cover category labels, the student model also references the soft label information output by the teacher model, enabling it to capture finer-grained semantic differences and category boundaries. This significantly improves the diagnostic accuracy and generalization ability for manhole covers in different scenarios and states.

[0039] Model inference stage: After completing the above training process, the image of the manhole cover to be tested is input into the pre-trained and distilled optimized category diagnostic model, and the model can output the category status prediction result of the manhole cover.

[0040] S4. Based on the self-supervised visual encoder DINOv3, the input image is subjected to multi-scale hierarchical analysis to obtain high-resolution image patch features.

[0041] In some feasible implementations, an image encoder is constructed based on the large-scale self-supervised pre-trained visual model DINOv3. This model employs a Vision Transformer architecture, dividing the input image into fixed-size non-overlapping image blocks and extracting high-dimensional semantic features step-by-step through a multi-layer self-attention mechanism. In the initial stage, weight parameters that have undergone self-supervised pre-training on ultra-large-scale unlabeled datasets such as ImageNet-25K are directly loaded, giving the model powerful general visual representation capabilities. Subsequently, the model is fine-tuned in a domain-adaptive manner on a labeled dataset for manhole cover condition diagnosis. This involves freezing some shallow modules to preserve low-level texture perception capabilities, unfreezing only mid-to-deep Transformer blocks, and performing end-to-end optimization with a low learning rate to prevent overfitting under small sample training.

[0042] During the forward propagation process, patch-level feature maps are extracted from different levels: shallow outputs capture local details (such as metal texture and edge sharpness), mid-level outputs respond to component structural features (such as manhole cover shape and bolt distribution), and deep outputs encode global semantic context (such as overall integrity and positional offset trends). After normalization and channel alignment, these multi-scale feature sequences constitute a spatially hierarchical multi-granular visual representation, which can effectively support the subsequent fine-grained identification and localization of subtle anomalies such as cracks, damaged boundaries, and slight displacements in manhole covers.

[0043] S5. Based on the visual language model, learnable prompt word templates are introduced to map the manhole cover status description into a high-dimensional text semantic vector.

[0044] In some feasible embodiments, a pre-trained visual language model generates a textual semantic description related to the state of the manhole cover, and encodes it into a high-dimensional textual feature vector, which serves as a semantic guidance signal for subsequent multimodal fusion. Specifically, a task-oriented fixed prompt word template is designed to stimulate the visual language model's ability to understand the state of the manhole cover. An example prompt word is: "You are a staff member specifically responsible for diagnosing the state of urban manhole covers. Please analyze images of manhole covers taken by drones, give a judgment on the state of the manhole cover (normal, damaged, displaced, missing), and explain the reason within 30 words." This prompt word and the manhole cover image to be classified are input into a visual language model (such as Qwen3-VL or ChatGPT-5), and the model automatically generates the corresponding natural language description, such as: "The manhole cover is made of round cast iron, the surface is intact without cracks, and it is in a normal state" or "The edge of the manhole cover has obvious cracks, and some areas are missing, which belongs to the damaged type." Subsequently, the generated text description is fed into the text encoder BERT and converted into a fixed-dimensional high-dimensional semantic vector. The semantic vectors embody the model's high-level understanding of the manhole cover's appearance, structural integrity, and potential fault types. They effectively represent the semantic boundaries of each category, providing reliable prior textual information for subsequent image-text interaction modules. This approach leverages the contextual reasoning capabilities of a large model to transform image content into structured semantic expressions, achieving a mapping from visual perception to language cognition. The obtained textual features will guide the spatial attention distribution of image features, enhancing the response intensity of key areas and improving the accuracy of manhole cover category discrimination in complex environments.

[0045] S6. Based on the bidirectional cross-attention module, the high-resolution image patch features are semantically interacted with the high-dimensional text semantic vector to obtain multimodal fusion features.

[0046] S7. Based on the selective state-space model Mamba, the multimodal fusion features are serialized, state-modeled, and dynamically fused to output a globally consistent final representation vector.

[0047] S8. Classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

[0048] In some feasible implementations, a bidirectional cross-attention module is constructed to enable deep interaction between image and text features.

[0049] In this process, the image patch features extracted in step S2 are linearly transformed to generate an image query matrix; the text semantic vector generated in step S3 is linearly transformed to obtain a text key matrix and a value matrix; the image query matrix and the text key matrix are multiplied and normalized to generate cross-modal attention weights; these weights are multiplied by the value matrix to obtain semantically guided visual features; these features are then fused with and mapped to the original image features to output an enhanced image representation. Conversely, using the text vector as the query and the image patch features as the key and value, an image-to-text attention operation is performed to update the text representation to better reflect the actual observed content. Through multiple rounds of alternating processing, a joint feature representation of deep image-text fusion is generated.

[0050] To further model the long-range spatial relationships among these fused features, the selective state-space model Mamba is used for their serialization. In this process, the fused features obtained in step S4 are arranged into a long sequence according to their original spatial order. The input is fed into the Mamba module. This module recursively updates the state at each time step using a discretized state equation: , Where the coefficient matrix and It is dynamically generated from the current input, has selective perception capabilities, and can actively filter irrelevant information while retaining key context. Compared with the self-attention mechanism of the traditional Transformer, Mamba effectively captures the long-distance correlation between the overall shape of the manhole cover and its surrounding environment (such as road cracks and object occlusion) while maintaining linear computational complexity, and finally outputs a compact and globally consistent representation vector z.

[0051] Based on this representation vector, a dual-path classification structure combining a dynamic prototype network and an uncertainty perception mechanism was designed to make the final classification decision. The dynamic prototype network maintains a category prototype vector for each category (normal, missing, broken, and shifted), initialized with the mean of samples from each category, and continuously optimized during training using a momentum update strategy. During inference, the cosine similarity between the input features and each category prototype is calculated, and the category with the highest score is selected as the initial prediction result.

[0052] The uncertainty-aware classification head employs a fully connected network structure with Monte Carlo Dropout, performing multiple forward propagations during the inference phase to collect the softmax output at each iteration and calculate the average prediction probability. and its entropy value: If the entropy value exceeds a preset threshold (e.g., 0.8), the model's judgment is considered insufficient, and the sample is marked as low-confidence and manual review is triggered. The system ultimately outputs the manhole cover status category, confidence score, and a flag indicating whether manual intervention is required, forming a reliable diagnostic mechanism that can both automatically identify and self-assess.

[0053] This method demonstrates strong robustness in practical tests, maintaining high recognition accuracy even under severe occlusion or blurred images. Thanks to Mamba's efficient structure and modular design, the entire model can run in real-time on drones or vehicle-mounted edge devices, making it suitable for urban road inspection scenarios. Furthermore, the dynamic prototyping and uncertainty mechanisms provide a solid foundation for future expansion into new categories or the construction of active learning systems.

[0054] This invention provides a manhole cover category diagnosis method based on semantic guidance and state space modeling, used for automated state recognition of manhole cover images acquired by UAV inspections. The identified states include four categories: normal, damaged, displaced, and missing. The method first constructs a training dataset and performs multi-scenario preprocessing. Then, it uses a self-supervised visual encoder DINOv3 to perform multi-scale hierarchical parsing of the input images, extracting high-resolution image patch features. Simultaneously, it uses learnable cue word templates from a visual language model to generate high-dimensional text semantic vectors related to the manhole cover state. Next, a bidirectional cross-attention module is used to achieve semantic alignment and interaction between image features and text features, guiding the model to focus on key areas of the manhole cover. Furthermore, a selective state space model (Mamba) is used to perform serialized state modeling and dynamic fusion of the fused multimodal features, forming a globally consistent representation vector. Finally, a dynamic prototype network combined with an uncertainty-aware classification head completes fine-grained diagnosis of the manhole cover state. This invention significantly improves the accuracy, robustness, and generalization ability of manhole cover state recognition in UAV inspection scenarios, and has the advantages of high computational efficiency and ease of deployment.

[0055] Example 2 like Figure 2 As shown, embodiments of the present invention also provide a manhole cover category diagnostic system based on semantic guidance and state space modeling, including: The training data processing unit is used to acquire training images and perform preprocessing to obtain preprocessed training images. The model training unit is used to train the category diagnosis model based on the preprocessed training images to obtain the trained category diagnosis model. The model application unit is used to input the image to be tested into the trained category diagnosis model; The visual coding unit is used to perform multi-scale hierarchical parsing of the input image based on the self-supervised visual encoder DINOv3 to obtain high-resolution image patch features. The text encoding unit is used to map the manhole cover status description into a high-dimensional text semantic vector based on a visual language model and by introducing learnable prompt word templates. A multimodal interaction unit is used to semantically interact the high-resolution image patch features with the high-dimensional text semantic vector based on a bidirectional cross-attention module to obtain multimodal fusion features; The state-space fusion unit is used to perform serialized state modeling and dynamic fusion of the multimodal fusion features based on the selective state-space model Mamba, and output a globally consistent final representation vector. The classification decision unit is used to classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

[0056] Example 3 This invention also provides a storage medium, which is a computer-readable storage medium storing a computer program thereon. The computer program is a manhole cover category diagnostic program, such as... Figure 1 As shown, when the manhole cover category diagnostic program is executed by the processor, it performs the following steps: S1. Acquire training images and perform preprocessing to obtain preprocessed training images; S2. The category diagnosis model is trained based on the preprocessed training images to obtain the trained category diagnosis model; S3. Input the image to be tested into the trained category diagnostic model; S4. Multi-scale hierarchical analysis of the input image is performed based on the self-supervised visual encoder DINOv3 to obtain high-resolution image patch features; S5. Based on the visual language model, learnable prompt word templates are introduced to map the manhole cover status description into a high-dimensional text semantic vector. S6. Based on the bidirectional cross-attention module, the high-resolution image patch features and the high-dimensional text semantic vector are semantically interacted to obtain multimodal fusion features; S7. Based on the selective state-space model Mamba, the multimodal fusion features are serialized and dynamically fused to output a globally consistent final representation vector. S8. Classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

[0057] Furthermore, in step S1, preprocessing the training images includes: performing data augmentation on the training images, wherein the data augmentation includes reducing image sharpness to simulate a long-distance shooting scene and adding occlusions.

[0058] Furthermore, in step S5, the learnable prompt word template is dynamically generated through gradient learnable parameters to enhance the distinguishability between the semantics of texts with different manhole cover states.

[0059] Further, in step S6, the bidirectional cross-attention module includes: Image-to-text attention mechanism is used to focus visual features based on text information to generate text-guided visual focus features; A text-to-image attention mechanism is used to enhance text features based on visual information to generate visually guided text enhancement features.

[0060] Furthermore, in step S7, the selective state-space model Mamba achieves dynamic fusion of cross-scale features through a selective interactive gating mechanism, thereby improving the ability to capture details of damage, gaps, and displacement.

[0061] Furthermore, in step S8: The dynamic prototype network achieves robust classification of complex manhole cover states by updating the category prototype vector online. The uncertainty-aware classification head estimates the uncertainty of samples based on temperature scaling and energy function, thereby improving the reliability of abnormal state identification.

[0062] Further, in step S8, the obtained state category of the manhole cover is a diagnostic result that includes the state type and corresponding confidence level. The state categories of the manhole cover include normal, damaged, displaced, and missing. And / or, after step S8, the method further includes: S9, filtering the state category results of the manhole cover using a non-maximum suppression algorithm, removing redundant and highly overlapping results, and outputting the final optimized diagnostic results; And / or, deploy the trained category diagnostic model to a drone or mobile terminal device.

[0063] Example 4 This invention also provides a manhole cover category diagnostic device based on semantic guidance and state space modeling, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is a manhole cover category diagnostic program, such as... Figure 1 As shown, when the processor executes the manhole cover category diagnostic program, it performs the following steps: S1. Acquire training images and perform preprocessing to obtain preprocessed training images; S2. The category diagnosis model is trained based on the preprocessed training images to obtain the trained category diagnosis model; S3. Input the image to be tested into the trained category diagnostic model; S4. Multi-scale hierarchical analysis of the input image is performed based on the self-supervised visual encoder DINOv3 to obtain high-resolution image patch features; S5. Based on the visual language model, learnable prompt word templates are introduced to map the manhole cover status description into a high-dimensional text semantic vector. S6. Based on the bidirectional cross-attention module, the high-resolution image patch features and the high-dimensional text semantic vector are semantically interacted to obtain multimodal fusion features; S7. Based on the selective state-space model Mamba, the multimodal fusion features are serialized and dynamically fused to output a globally consistent final representation vector. S8. Classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

[0064] Furthermore, in step S1, preprocessing the training images includes: performing data augmentation on the training images, wherein the data augmentation includes reducing image sharpness to simulate a long-distance shooting scene and adding occlusions.

[0065] Furthermore, in step S5, the learnable prompt word template is dynamically generated through gradient learnable parameters to enhance the distinguishability between the semantics of texts with different manhole cover states.

[0066] Further, in step S6, the bidirectional cross-attention module includes: Image-to-text attention mechanism is used to focus visual features based on text information to generate text-guided visual focus features; A text-to-image attention mechanism is used to enhance text features based on visual information to generate visually guided text enhancement features.

[0067] Furthermore, in step S7, the selective state-space model Mamba achieves dynamic fusion of cross-scale features through a selective interactive gating mechanism, thereby improving the ability to capture details of damage, gaps, and displacement.

[0068] Furthermore, in step S8: The dynamic prototype network achieves robust classification of complex manhole cover states by updating the category prototype vector online. The uncertainty-aware classification head estimates the uncertainty of samples based on temperature scaling and energy function, thereby improving the reliability of abnormal state identification.

[0069] Further, in step S8, the obtained state category of the manhole cover is a diagnostic result that includes the state type and corresponding confidence level. The state categories of the manhole cover include normal, damaged, displaced, and missing. And / or, after step S8, the method further includes: S9, filtering the state category results of the manhole cover using a non-maximum suppression algorithm, removing redundant and highly overlapping results, and outputting the final optimized diagnostic results; And / or, deploy the trained category diagnostic model to a drone or mobile terminal device.

[0070] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A manhole cover category diagnosis method based on semantic guidance and state space modeling, characterized in that, Includes the following steps: S1. Acquire training images and perform preprocessing to obtain preprocessed training images; S2. The category diagnosis model is trained based on the preprocessed training images to obtain the trained category diagnosis model; S3. Input the image to be tested into the trained category diagnostic model; S4. Multi-scale hierarchical analysis of the input image is performed based on the self-supervised visual encoder DINOv3 to obtain high-resolution image patch features; S5. Based on the visual language model, learnable prompt word templates are introduced to map the manhole cover status description into a high-dimensional text semantic vector. S6. Based on the bidirectional cross-attention module, the high-resolution image patch features and the high-dimensional text semantic vector are semantically interacted to obtain multimodal fusion features; S7. Based on the selective state-space model Mamba, the multimodal fusion features are serialized and dynamically fused to output a globally consistent final representation vector. S8. Classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

2. The manhole cover category diagnosis method based on semantic guidance and state space modeling according to claim 1, characterized in that, In step S1, preprocessing the training images includes: performing data augmentation on the training images, wherein the data augmentation includes reducing image sharpness to simulate a long-distance shooting scene and adding occlusions.

3. The manhole cover category diagnosis method based on semantic guidance and state space modeling according to claim 1, characterized in that, In step S5, the learnable prompt word template is dynamically generated through gradient learnable parameters to enhance the distinguishability between the semantics of texts with different manhole cover states.

4. The manhole cover category diagnosis method based on semantic guidance and state space modeling according to claim 1, characterized in that, In step S6, the bidirectional cross-attention module includes: Image-to-text attention mechanism is used to focus visual features based on text information to generate text-guided visual focus features; A text-to-image attention mechanism is used to enhance text features based on visual information to generate visually guided text enhancement features.

5. The manhole cover category diagnosis method based on semantic guidance and state space modeling according to claim 1, characterized in that, In step S7, the selective state-space model Mamba achieves dynamic fusion of cross-scale features through a selective interactive gating mechanism, thereby improving the ability to capture details of damage, gaps, and displacement.

6. The manhole cover category diagnosis method based on semantic guidance and state space modeling according to claim 1, characterized in that, In step S8: The dynamic prototype network achieves robust classification of complex manhole cover states by updating the category prototype vector online. The uncertainty-aware classification head estimates the uncertainty of samples based on temperature scaling and energy function, thereby improving the reliability of abnormal state identification.

7. The manhole cover category diagnosis method based on semantic guidance and state space modeling according to claim 1, characterized in that, In step S8, the obtained state category of the manhole cover is a diagnostic result that includes the state type and corresponding confidence level. The state categories of the manhole cover include normal, damaged, displaced and missing. And / or, after step S8, the method further includes: S9, filtering the state category results of the manhole cover using a non-maximum suppression algorithm, removing redundant and highly overlapping results, and outputting the final optimized diagnostic results; And / or, deploy the trained category diagnostic model to a drone or mobile terminal device.

8. A manhole cover category diagnosis system based on semantic guidance and state space modeling, characterized in that, include: The training data processing unit is used to acquire training images and perform preprocessing to obtain preprocessed training images. The model training unit is used to train the category diagnosis model based on the preprocessed training images to obtain the trained category diagnosis model. The model application unit is used to input the image to be tested into the trained category diagnosis model; The visual coding unit is used to perform multi-scale hierarchical parsing of the input image based on the self-supervised visual encoder DINOv3 to obtain high-resolution image patch features. The text encoding unit is used to map the manhole cover status description into a high-dimensional text semantic vector based on a visual language model and by introducing learnable prompt word templates. A multimodal interaction unit is used to semantically interact the high-resolution image patch features with the high-dimensional text semantic vector based on a bidirectional cross-attention module to obtain multimodal fusion features; The state-space fusion unit is used to perform serialized state modeling and dynamic fusion of the multimodal fusion features based on the selective state-space model Mamba, and output a globally consistent final representation vector. The classification decision unit is used to classify the final representation vector based on the dynamic prototype network and the uncertainty-aware classification head to obtain the state category of the manhole cover.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium on which a computer program is stored. The computer program is a manhole cover category diagnosis program. When the manhole cover category diagnosis program is executed by the processor, it implements the steps of the manhole cover category diagnosis method based on semantic guidance and state space modeling as described in any one of claims 1-7.

10. A manhole cover category diagnostic device based on semantic guidance and state space modeling, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is a manhole cover category diagnostic program. When the processor executes the manhole cover category diagnostic program, it implements the steps of the manhole cover category diagnostic method based on semantic guidance and state space modeling as described in any one of claims 1-7.

Images