Well lid hidden danger integrated identification method and system fusing target detection and segmentation

Abstract

The invention discloses a manhole cover hidden danger integrated identification method and system fusing target detection and segmentation, and the method comprises the steps: carrying out the preprocessing of manhole cover image data, and dividing the manhole cover image data into cells and anchor frames; and secondly, extracting multi-scale features of the preprocessed well lid picture through YOLO v8x, and outputting a well lid prediction result. And a Bagging ensemble learning strategy is adopted, and based on the well lid prediction result, a weighted voting mechanism is utilized to obtain a discrimination result of the well lid hidden danger category. And inputting the preprocessed well lid picture into an encoder module composed of PVTv2-B4 for well ring feature extraction, and outputting a well ring prediction result. And carrying out secondary analysis on the walling crib based on the walling crib prediction result to obtain a judgment result of the type of the hidden danger of the walling crib. And finally, obtaining a hidden danger category by combining the distinguishing results of the well lid and the well ring. According to the invention, the refined identification capability of the well lid state in a complex scene is improved, and accurate outlining and positioning of the well ring boundary are realized.

Description

Technical Field

[0001] The present invention belongs to the field of manhole cover hidden danger identification, and relates to a manhole cover hidden danger integrated identification method and system integrating target detection and segmentation. Background Art

[0002] With the continuous development of society, urban infrastructure construction is becoming more and more perfect. As one of the indispensable infrastructures in the city, the safety of manhole covers has also attracted much attention. However, in recent years, hidden dangers of manhole covers have occurred frequently, and many serious safety accidents are simply caused by manhole cover problems such as manhole cover displacement, damage or loss. The root causes of these manhole cover problems may be unreasonable design, improper installation, material aging, vehicle crushing, etc. These problems may not be quickly discovered by conventional manual inspections or monitoring systems, and it is difficult to take timely repair or replacement measures, which brings great risks to vehicles, pedestrians, communication facilities, etc., and becomes a hidden danger to urban safety. Therefore, real-time and accurate identification of hidden dangers of manhole covers is of great significance to urban public safety management.

[0003] There are various types of manhole covers in cities, and the hidden danger characteristics of manhole covers in different environments vary greatly, which brings great challenges to the identification of hidden dangers in manhole covers. Researchers have never stopped studying the hidden dangers of manhole covers. Changes in physical signals such as vision, vibration, and sound waves are generally used to detect hidden dangers in manhole covers. The recognition accuracy of a single detection sensor is not as good as the joint action of multiple sensors. However, this multi-sensor detection method has high requirements for sensors, installation, and environment, so it is greatly limited in practical applications. Based on this, hidden danger detection methods proposed using image features or vibration features on the surface of manhole covers have gradually developed, but the manually extracted features are affected by prior knowledge, and their adaptability and generalization ability need to be improved, and the optimization process is complicated and cumbersome. At the same time, the influence of various complex factors in the urban environment, such as vehicle flow, climatic conditions, etc., causes the manually extracted features to be easily missed or misdetected in similar complex scenes, making it difficult to ensure recognition accuracy.

[0004] In recent years, there are two main types of research on the identification of hidden dangers in manhole covers: 1. Methods for extracting manhole cover features: mainly analyzing the image or vibration signal features of the manhole cover, and using mathematical methods to extract effective features, combined with machine learning classifiers or artificial neural networks to obtain the final output. 2. Research on deep learning models and algorithms: Most of them extract features by designing different convolutional networks, and some methods pre-process the physical signals of the manhole cover into images and then input them into subsequent classification models for processing. These methods analyze the impact of manhole cover hazards on environmental signals, the impact of different scenarios on the characteristics of manhole cover hazards, and apply heuristic methods or machine learning to make decisions. Among them, deep learning models have been repeatedly proven to have significant advantages in various performance indicators in the identification of manhole cover hazards. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for integrated identification of hidden dangers of manhole covers that integrates target detection and segmentation in view of the deficiencies of the prior art. The method first uses the YOLO (You Only Look Once) v8x model for target detection pre-training, focusing on the identification of different types of manhole covers, while excluding manhole circles as training targets. At the same time, a segmentation model based on the PVTv2-B4-Swin-Transformer cascade U-shaped composite structure is constructed for accurate segmentation of manhole circles. For each anchor frame generated by target detection, three Yolo v8l-cls classification networks are trained using a self-service sampling method. The three classification models make predictions independently, and their results are comprehensively evaluated with the category prediction results of the original target detection model itself. In the final category decision process, the category weight predicted by the target detection model itself is assigned three votes, and the prediction results of each classification model each account for one vote. In this way, weighted voting is performed, and finally the categories of manhole covers and manhole circles are integrated to realize the identification of hidden dangers of manhole covers.

[0006] In a first aspect, the present invention provides an integrated identification method for manhole cover hazards by integrating target detection and segmentation, the method comprising the following steps:

[0007] Object Detection:

[0008] 1. Preprocess the collected manhole cover image data, including size standardization, normalization, and division of cells and anchor frames to meet the needs of subsequent processing.

[0009] 2. The YOLO v8x model is used to extract the multi-scale features of the preprocessed manhole cover images, and multi-scale feature fusion and anchor frame allocation are performed to output the manhole cover prediction results.

[0010] Semantic Segmentation:

[0011] 3. Input the manhole cover image preprocessed in the first step into the encoder module composed of PVTv2-B4 to extract the manhole circle features and output the manhole circle prediction results.

[0012] Ensemble learning:

[0013] 4. The Bagging ensemble learning strategy is adopted to obtain the discrimination results of the manhole cover hidden danger category based on the manhole cover prediction results and the weighted voting mechanism. At the same time, the manhole circle is subjected to a secondary analysis based on the well circle prediction results to obtain the discrimination results of the well circle hidden danger category. The hidden danger category is obtained by combining the two discrimination results.

[0014] Preferably, step 1 specifically comprises:

[0015] 1-1 Adjust all manhole cover images I to a fixed size (H, W, 3), and pad the insufficient parts to maintain the aspect ratio of the image.

[0016] 1-2 Normalize the resized manhole cover image I and map the pixel values ​​to the range [0,1] to obtain the normalized manhole cover image I pre .

[0017] 1-3 The normalized manhole cover image I in step 1-2 pre It is divided into S×S cells, and several anchor boxes are assigned to each cell. The size and proportion of the anchor boxes are set during model training.

[0018] Preferably, step 2 is specifically:

[0019] 2-1 The normalized manhole cover image I in step (1) pre Input into the backbone network of the YOLO v8x model for feature extraction. The backbone of YOLO v8x contains multiple convolutional layers and residual connection modules, which are used to gradually extract local and global features in the image and form multi-scale feature maps at different levels.

[0020] 2-2 The multi-scale feature map in step 2-1 Input to the Neck part of the YOLO v8x model to aggregate features of different scales. Get the feature map F neck .

[0021] 2-3 The aggregated feature map P neck Input into the head network Head of YOLO v8x; Head consists of a detection head and a classification head. The detection head generates bounding box predictions, confidence predictions, and category predictions for each anchor box, and then inputs the classification head for classification, and finally outputs the manhole cover prediction result P class .

[0022] As a preferred embodiment, the specific implementation process of step 3 is as follows:

[0023] 3-1 Save the output features of each layer of the encoder, and concatenate the features of the last three layers and send them to the feature pyramid FPN for feature aggregation. After processing, the aggregated features are concatenated with the features output by the last three layers of encoders to obtain the skip connection features corresponding to the last three layers.

[0024] 3-2 In the decoder part, the skip connection features of the fourth layer are used as the input of the fourth layer decoder, and are output after being processed by the decoder. The skip connection features corresponding to the third layer after convolution are used as the input of the third layer encoder. This is analogous to the output of the first layer decoder. The segmentation truth value is used for supervision and the pixel position aware PPA Loss function is used to update the model parameters. The prediction result output by the updated model is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

[0025] Preferably, the specific implementation process of step 3-1 is as follows:

[0026] 3-1-1 The manhole cover image data is adjusted to a fixed size and then input into the encoder module composed of PVTv2-B4. The encoder module has four layers of PVTv2-B4 encoders, denoted as e x ,x={1,2,3,4}; the manhole cover image is input to encoder e1, which processes it and inputs it to the next layer encoder e2, and so on. The feature map output of each encoder is recorded as

[0027] 3-1-2 The outputs of encoders e2, e3, and e4 Input into FPN for feature aggregation, and the aggregated feature map is recorded as Its width, height, and number of channels are similar to same.

[0028] 3-1-3 Aggregate feature map Perform average pooling and convolution operations to obtain feature maps Its size and Same; feature maps are obtained by bilinear interpolation and convolution at the same time Its size and same; will Respectively Splice to get the skip connection features of the last three layers

[0029] Preferably, the specific implementation process of step 3-2 is:

[0030] The decoder module has four layers of Swin-Transformer decoders, denoted as d x , and the output feature map of each layer of the decoder is recorded as The skip connection features corresponding to the fourth layer Input into the fourth layer decoder d4, and after being processed by the decoder, the fourth layer decoder output feature is obtained Later Perform convolution operation to make it equal to Same as Together they serve as the input of d3. The same is true for d2. d1 only accepts the output of d2 and outputs the feature map after processing Will Use the segmentation truth value S GT The updated model outputs the feature map P ring That is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

[0031] Preferably, step (4) is specifically:

[0032] 4-1 Through bootstrap sampling, three different manhole cover data subsets D1, D2, and D3 are randomly sampled with replacement to train three independent YOLO v8l-cls subnetworks. After each subnetwork is trained, the category of manhole cover hidden dangers is independently predicted. The prediction result of each subnetwork is recorded as

[0033] 4-2 When fusing the prediction results of the three YOLO v8l-cls subnetworks and the initial YOLO v8x model, a weighted voting mechanism is used to set the prediction result weight of the YOLO v8x model to three votes, and the prediction result weight of the three YOLO v8l-cls subnetworks to one vote, and the weighted sum is performed to obtain the discrimination result P of the manhole cover hidden danger category. well , the specific formula is:

[0034]

[0035] 4-3 For the well circle part, the well circle prediction result P output in step (5) is ring After the mask, it is input into another YOLO v8x model for training, and finally the identification result P of the well circle hidden danger category is output. circle .

[0036] 4-4 Obtain the result of manhole cover hidden danger identification P well (well-covered, missing, damaged, uncovered) and the results of the identification of hidden dangers in the well circle P circle After the well ring is intact or has problems, the following rules are used to perform category fusion to obtain the final hidden danger category P final :If P well If the manhole cover is intact and P circle If the wellbore is intact, then P final is the "well circle problem"; if P well If the manhole cover is intact and P circle If "there is a problem with the wellbore", then P final is the "well circle problem"; if P wellIf it is not "the manhole cover is intact", P is not considered circle , P final Equivalent to P well Hazard category.

[0037] In a second aspect, the present invention provides an integrated identification system for manhole cover hazards integrating target detection and segmentation, comprising the following modules:

[0038] The data processing module is used to obtain the image data of manhole cover hidden dangers, pre-process the data according to categories, and divide the cells and anchor frames.

[0039] The target positioning module obtains the manhole cover prediction results and the manhole circle prediction results through the preprocessed manhole cover hidden danger image and the encoder module composed of the YOLOv8x model and PVTv2-B4 respectively.

[0040] The category judgment module adopts an integrated classification network strategy. Based on the prediction results of manhole covers and manhole rings, combined with a weighted voting mechanism, it obtains the discrimination results of the manhole cover hidden danger category and the discrimination results of the manhole ring hidden danger category respectively. Finally, the hidden danger category is obtained by combining the two discrimination results.

[0041] The beneficial effects of the present invention are:

[0042] 1. The present invention adopts the Yolo v8 model for target detection, which improves the model's ability to finely identify the status of manhole covers in complex scenarios. Even in environments with drastic light changes, severe occlusion or complex backgrounds, it can effectively distinguish between various states of manhole covers, such as normal, damaged, or missing.

[0043] 2. This invention combines the ability of PVTv2-B4 to efficiently process multi-scale visual features and the excellent performance of Swin-Transformer in global information modeling and self-attention mechanism. Through the U-shaped architecture, the model can effectively capture the underlying image details while fully refining high-level semantic features to achieve accurate delineation and positioning of the well circle boundary.

[0044] 3. The present invention selects the self-service sampling method to train three Yolo v8l-cls classification networks. The three classification models make predictions independently, and their results are comprehensively evaluated with the category prediction results of the original target detection model itself. Specifically, in the final category decision process, the category weight predicted by the target detection model itself is assigned three votes, and the prediction results of each classification model each account for one vote. In this way, weighted voting is performed to finally determine the precise category label of the manhole cover. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the structure of the present invention;

[0046] Figure 2 Schematic diagram of the segmentation model;

[0047] Figure 3 PR curve of the present invention on the validation set of high-resolution acquisition data. DETAILED DESCRIPTION

[0048] Further analysis is given below in conjunction with specific embodiments and drawings.

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0050] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the steps or units listed in the specification, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0051] In a first aspect, the present invention provides an integrated identification method for manhole cover hazards by integrating target detection and segmentation, the method comprising the following steps:

[0052] Object Detection:

[0053] 1. Preprocess the collected manhole cover image data, including size standardization, normalization, and division of cells and anchor frames to meet the needs of subsequent processing.

[0054] 2. The YOLO v8x backbone network extracts local and global features of the preprocessed manhole cover image through convolutional layers and residual connections to generate corresponding feature maps. The feature map is input to the Neck part for multi-scale feature fusion and anchor frame assignment. Finally, the fused feature map and anchor frame are input to the detection head of the Head part to generate the bounding box prediction, confidence prediction and category prediction of each anchor frame; then it is input to the classification head for classification and outputs the manhole cover prediction result.

[0055] Semantic Segmentation:

[0056] 3. Such as Figure 2 As shown, the manhole cover image preprocessed in the first step is input into the encoder module composed of PVTv2-B4 to extract the manhole circle features and output the manhole circle prediction result.

[0057] 3-1. Save the output features of each layer of the encoder, and concatenate the features of the last three layers into the FPN (Feature Pyramid Network) feature pyramid for feature aggregation. After processing, the aggregated features are concatenated with the features output by the original three layers of encoders to obtain the skip connection features corresponding to the last three layers.

[0058] 3-2. In the decoder part, the skip connection features of the fourth layer are used as the input of the fourth layer decoder, and are processed and output by the decoder. The skip connection features corresponding to the third layer after convolution are used as the input of the third layer encoder, and so on. After the output of the first layer decoder, the segmentation truth value is used for supervision and the PPA (Pixel Position Aware) Loss function is used to update the model parameters. The prediction result output by the updated model is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

[0059] Ensemble learning:

[0060] 4. The Bagging ensemble learning strategy is adopted, that is, three YOLO v8l-cls sub-networks are trained to learn the hidden dangers of manhole covers. Based on the prediction results of manhole covers, the bounding box and classification results are fused using a weighted voting mechanism. At the same time, another YOLOv8x model is used to perform a secondary analysis of the manhole circle and classify the hidden danger categories of the manhole circle based on the prediction results of the manhole circle. Finally, the judgment results of the two models are combined to obtain the hidden danger category.

[0061] Preferably, step (1) is:

[0062] 1-1 Adjust all manhole cover images I to a fixed size (H, W, 3), and pad the insufficient parts to maintain the aspect ratio of the manhole cover image.

[0063] 1-2 Normalize the manhole cover image I and map the pixel values ​​to the range [0,1]. The specific operation is to divide the pixel values ​​of the original image by 255, that is:

[0064]

[0065] Among them I pre Represents the normalized manhole cover image, where the pixel values ​​are scaled to the range [0,1].

[0066] 1-3 The manhole cover image I preprocessed in step 1-2 pre It is divided into S×S cells, and one or more anchor boxes are assigned to each cell. The size and proportion of the anchor box are set during model training.

[0067] Preferably, step (2) is:

[0068] 2-1 The image I that has been preprocessed and resized in step (1) pre Input into the backbone network of the YOLO v8x model for feature extraction. The backbone of YOLO v8 contains multiple convolutional layers and residual connection modules, which are used to gradually extract local and global features in the image and form feature maps at different levels. The process of feature extraction can be expressed as:

[0069]

[0070] Among them, Bone(·) represents the operations of convolution, deconvolution, residual connection, etc. of the backbone network. Represents the feature map at level i.

[0071] 2-2 The multi-scale feature map in step 2-1 Input to the Neck part of the YOLO v8x model to aggregate features of different scales. The Neck part contains the following modules:

[0072] The SPPF (Spatial Pyramid Pooling-Fast) module reduces the feature map to different spatial scales through pooling operations at different scales, thereby obtaining multi-scale information and adapting to targets of different sizes. Input feature map After being processed by the SPPF module, a multi-scale feature map is obtained.

[0073] The PAA (Probabilistic Anchor Assignment) module is used for intelligent anchor box assignment. An adaptive anchor box is assigned to each feature position based on the prediction confidence to accommodate manhole covers of different sizes and shapes. The PAA module selects the optimal anchor box by calculating the probability distribution of each candidate box.

[0074] The PAN (Path Aggregation Network) module further aggregates feature maps at different levels to perform multi-scale feature maps. By integrating multi-level features through upsampling and downsampling operations, the model's ability to detect multi-scale targets is enhanced. The aggregated feature map is F neck , the specific formula is:

[0075]

[0076] where PAN(·) represents the multi-level feature aggregation operation in the PAN module.

[0077] 2-3 The aggregated feature map F neckInput to the head network (Head) of YOLO v8x. The Head consists of two main parts: the detection head (for bounding box regression) and the classification head (for target category prediction).

[0078] The detection head performs bounding box regression on the anchor box through convolution and deconvolution operations, predicting the specific position and size of each anchor box. The output of the detection head is the bounding box parameter B and the target confidence C:

[0079] (B, C) = DH (F neck )

[0080] Among them, B represents the regression value of the bounding box, C represents the target confidence, which is used to determine whether the candidate box contains the manhole cover target; DH(·) represents a series of convolution and deconvolution operations in the detection head.

[0081] The classification head extracts the global features in the feature map through the global average pooling operation, and predicts the category of each detected target, outputting the category prediction as P class .

[0082] The final output of the classification head is the final object detection result of the YOLO v8x model.

[0083] Preferably, step 3-1 is specifically:

[0084] 3-1-1 The manhole cover image data is organized into a uniform size (352*352*3) and then input into the encoder module composed of PVTv2-B4. Specifically: The encoder module has four layers of PVTv2-B4 encoders, denoted as e x ,x={1,2,3,4}. The manhole cover image is input to encoder e1, which processes it and inputs it to the next encoder e2, and so on. The feature map output of each encoder is recorded as

[0085] 3-1-2 The outputs of encoders e2, e3, and e4 in step (3) are Input into FPN for feature aggregation, and the aggregated feature map is recorded as Its width, height, and number of channels are similar to same.

[0086] 3-1-3 Aggregate feature map Perform average pooling and convolution operations to obtain feature maps Its size and Same; feature maps are obtained by bilinear interpolation and convolution Its size and Same. Respectively Splice to get the skip connection features of the last three layers

[0087] Preferably, step 3-2 is specifically:

[0088] The decoder module has four layers of Swin-Transformer decoders, denoted as d x , and the output feature map of each layer of the decoder is recorded as The skip connection features corresponding to the fourth layer Input into the fourth layer decoder d4, and after being processed by the decoder, the fourth layer decoder output feature is obtained Later Perform convolution operation to make it equal to Same as Together they serve as the input of d3. The same is true for d2. d1 only accepts the output of d2 and outputs the feature map after processing Will Use the segmentation truth value S GT The updated model outputs the feature map P ring That is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

[0089] Preferably, step (4) is specifically:

[0090] 4-1 Through bootstrap sampling, three different manhole cover data subsets D1, D2, and D3 are randomly sampled with replacement to train three independent YOLO v8l-cls subnetworks. After each subnetwork is trained, the category of manhole cover hidden dangers is independently predicted. The prediction result of each subnetwork is recorded as

[0091] 4-2 When fusing the prediction results of the three YOLO v8l-cls subnetworks and the initial YOLO v8x model, a weighted voting mechanism is used to set the prediction result weight of the YOLO v8x model to three votes, and the prediction result weight of the three YOLO v8l-cls subnetworks to one vote, and the weighted sum is performed to obtain the discrimination result P of the manhole cover hidden danger category. well , the specific formula is:

[0092]

[0093] 4-3 For the well circle part, the well circle prediction result P output in step (5) is ringAfter the mask, it is input into another YOLO v8x model for training, and finally the identification result P of the well circle hidden danger category is output. circle .

[0094] 4-4 Obtain the result of manhole cover hidden danger identification P well (well-covered, missing, damaged, uncovered) and the results of the identification of hidden dangers in the well circle P circle After the well ring is intact or has problems, the following rules are used to perform category fusion to obtain the final hidden danger category P final :If P well If the manhole cover is intact and P circle If the wellbore is intact, then P final is the "well circle problem"; if P well If the manhole cover is intact and P circle If "there is a problem with the wellbore", then P final is the "well circle problem"; if P well If it is not "the manhole cover is intact", P is not considered circle , P final Equivalent to P well Hazard category.

[0095] In a second aspect, the present invention provides a manhole cover hidden danger identification system for implementing the above method, comprising:

[0096] The data processing module is used to obtain the image data of hidden dangers of manhole covers through crawlers, AIGC, etc. At the same time, the data is divided according to categories and pre-processed such as data enhancement and size normalization, and cells and anchor boxes are divided;

[0097] The target positioning module deploys two independently running neural network architectures at the same time, which are respectively composed of YOLO v8x model and PVTv2-B4 encoder modules for the category detection task of manhole covers and the segmentation and classification task of manhole rings, to obtain the prediction results of manhole covers and manhole rings. Among them, the target detection network realizes the precise positioning and category recognition of various types of manhole covers, and the segmentation network is used to perform pixel-level segmentation of the manhole ring area and further refine its category labeling.

[0098] The category determination module adopts an integrated classification network strategy and combines a weighted voting mechanism to optimize the final decision process. This module integrates the prediction results of multiple classifiers, reasonably screens and adjusts the weights of candidate anchor boxes, and finally outputs a fully integrated and judged category label.

[0099] Example:

[0100] In the process of manhole cover image recognition and classification, it is difficult to ensure the recognition accuracy of the features of manhole covers and manhole rings because the boundaries between manhole covers and manhole rings are difficult to distinguish and manhole covers of the same type are very different. Therefore, in order to better improve the accuracy of the model in identifying and classifying manhole cover categories, the following problems need to be solved: how to achieve multi-target detection and improve the refined classification and generalization capabilities in classification tasks as much as possible.

[0101] Based on this, the embodiment of the present application provides an integrated identification method for manhole cover hidden dangers based on target detection and segmentation. The present invention uses the YOLO v8x model for target detection pre-training, focusing on the identification of different types of manhole covers, while excluding manhole circles as training targets. At the same time, a method based on

[0102] The segmentation model of the cascaded U-shaped composite structure of PVTv2-B4-Swin-Transformer is used to accurately segment the manhole circle. For each anchor frame generated by target detection, three Yolo v8l-cls classification networks are trained using the self-service sampling method. The three classification models make predictions independently, and their results are comprehensively evaluated with the category prediction results of the original target detection model itself. In the final category decision process, the category weight predicted by the target detection model itself is assigned three votes, and the prediction results of each classification model each account for one vote. In this way, weighted voting is performed, and finally the manhole cover and manhole circle categories are integrated to realize the identification of manhole cover hazards.

[0103] Manhole cover hidden danger identification method based on target detection and segmentation, such as Figure 1 The following steps are involved:

[0104] Step (1) deploys two independent neural networks to detect the category of manhole covers and segment and classify the manhole rings. The target detection network is used to locate and identify manhole covers, while the segmentation network focuses on pixel-level segmentation and category refinement of the manhole rings. The specific steps are as follows:

[0105] Step 1: Use the Yolo v8x target detection model architecture to perform supervised learning training on a specific image dataset. The dataset at this stage contains a variety of different types of manhole cover sample images, aiming to enhance the model's ability to accurately detect and locate various manhole cover entities, as well as preliminary category differentiation capabilities through training. Yolo v8x usually uses a combined loss function, which can be expressed as:

[0106] L=ω coord L coord +ω conf L conf +ω class L class

[0107] Where L coord is the position loss, Lconf is the confidence loss, L class is the classification loss, ω coord ,ω conf and ω class is the weight coefficient used to balance these losses.

[0108] Step (2): Using some elements of PVTv2-B4, SwinTransformer and U-Net architecture, a PVTv2-B4-Swin-Transformer cascade U-shaped composite structure is proposed. By utilizing the deep feature extraction and fusion capabilities contained in the model architecture, the accurate depiction of the boundaries of the well area in the image and the accurate judgment of the area to which the internal pixels belong are ensured. The model utilizes the attention mechanism in the Transformer encoder.

[0109]

[0110] Where Q, K and V represent the query vector, key vector and value vector in the attention mechanism respectively. k is the dimension of the key vector.

[0111] Step (3): After Step 2 is completed, the output well circle segmentation mask marks whether each pixel belongs to the target well circle structure. The segmentation mask is multiplied with the well circle image. The pixels of the well circle part in the original image are marked as significant values ​​in the mask and are retained and enhanced in the dot multiplication process, while the pixels of the background and other non-well circle parts are faded or disappeared by multiplying with the lower or zero value of the corresponding position in the mask.

[0112] Step (4) uses an integrated classification network strategy and a re-voting mechanism to integrate the prediction results of multiple classifiers and screen and adjust the weights of candidate areas, ultimately obtaining a category determination of manhole cover hazards. The specific steps are as follows:

[0113] Step 1: For each anchor box generated by the object detection algorithm, a multi-model ensemble learning strategy is adopted. Specifically, three Yolo v8l-cls subnetworks are trained independently, each focusing on predicting the object category in the anchor box. Self-service sampling is used to prevent overfitting, and each classification network is trained with different data subsets.

[0114] Step 2: In the decision fusion stage, four key sources of information are considered: the basic predicted category of the target in each candidate box from the original target detection algorithm, and the prediction output of each of the three Yolo v8l-cls classification networks. Specifically, the initial target detection algorithm is given a higher degree of trust, and its predicted category has the equivalent of three votes in the voting process, while each independently trained classification network has one vote.

[0115]

[0116] where {c i =c} is an indicator function, which takes the value 1 when the condition is met and 0 otherwise.

[0117] Step 3: Use an independent Yolo v8x model to conduct a deep secondary analysis of the image processed by the segmentation network, and the focus is shifted to accurately identifying whether there are quality problems in the well circle. Through in-depth learning of training samples, the model has a high sensitivity to the well circle state and the ability to accurately distinguish. Ultimately, the model can grasp the well circle characteristics in the two states of "circle-good" and "circle-problem" and form an accurate recognition pattern.

[0118] Step 4: Combining Step 1, Step 2 and Step 3, when the manhole cover itself remains intact but there are obvious problems with the surrounding manhole ring, such cases are uniformly classified into the "circle" category. Conversely, for samples with damaged manhole covers, they are classified and labeled accordingly according to their actual degree of damage and type.

[0119] The above model is trained. For the loss function design inside the model, the present invention gives a larger weight to the category prediction branch (classification branch), which is set to 5 to emphasize the training strength of the model in distinguishing different categories. In terms of training strategy, a phased training method is selected. Each training cycle contains 100 iterations (epochs), and a total of two such cycles are executed, that is, a total of 200 rounds of training. Within each cycle, especially in the learning rate adjustment scheme of the segmentation network, cosine annealing is adopted, and the parameters are set to GFLOPs = 92.81MMac, Params = 126.72M, to ensure that the learning rate has a reasonable decline curve during training, which is conducive to the convergence and generalization performance of the model. The initial learning rate is set to the highest value of 5e-5 at the beginning of the first cycle, and decreases according to the cosine annealing law as the training progresses, and drops to the lowest value of 1e-7 at the end of the second cycle. This dynamic learning rate adjustment mechanism helps the model to learn quickly in the early stage of training, and refines the model parameters in the later stage, effectively suppressing the overfitting phenomenon. In order to prove the effectiveness of the model, the target positioning module and category determination module of the model were tested and the PR curve was drawn. Figure 3 .

[0120] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing a program code, and the instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the aforementioned method embodiment, which will not be repeated here. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store program codes.

[0121] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An integrated identification method for manhole cover hazards integrating target detection and segmentation, characterized in that: The following steps are involved: Step 1. Preprocess the collected manhole cover image data and divide it into cells and anchor frames; Step 2. Use the YOLO v8x model to extract the multi-scale features of the preprocessed manhole cover image, perform multi-scale feature fusion and anchor frame allocation, and output the manhole cover prediction result; Step 3. Input the pre-processed manhole cover image into the PVT v The encoder module composed of 2-B4 extracts well circle features, completes semantic segmentation, and outputs well circle prediction results; Step 4. Using the Bagging ensemble learning strategy, based on the manhole cover prediction results, and using the weighted voting mechanism, we can obtain the classification results of the manhole cover hidden danger categories; At the same time, based on the well circle prediction results, the well circle is analyzed again to obtain the discrimination result of the well circle hidden danger category; the hidden danger category is obtained by combining the two discrimination results.

2. The integrated identification method of manhole cover hidden dangers by integrating target detection and segmentation according to claim 1 is characterized in that: The implementation process of step 1 is as follows: Step 1-1: adjust all manhole cover images I to a fixed size (H, W, 3), and pad the insufficient part to maintain the aspect ratio of the manhole cover images; Step 1-2 normalizes the resized manhole cover image I and maps the pixel values ​​to the range [0, 1] to obtain the normalized manhole cover image I pre ; Step 1-3: normalize the manhole cover image I pre It is divided into S×S cells, and several anchor boxes are assigned to each cell.

3. The integrated identification method of manhole cover hidden dangers by integrating target detection and segmentation according to claim 2 is characterized in that: The specific implementation process of step 2 is as follows: 2-1 The normalized manhole cover image I pre The image is input into the backbone network of the YOLO v8x model for feature extraction; the backbone of the YOLO v8x model contains multiple convolutional layers and residual connection modules, which are used to gradually extract local and global features in the image and form multi-scale feature maps at different levels. 2-2 Multi-scale feature map Input to the Neck part of the YOLO v8x model, perform feature aggregation at different scales, and obtain the feature map F neck ; 2-3 The aggregated feature map F neck Input into the head network Head of YOLO v8x; Head consists of a detection head and a classification head. The detection head generates bounding box predictions, confidence predictions, and category predictions for each anchor box, and then inputs them into the classification head for classification, and finally outputs the manhole cover prediction result P class .

4. The integrated identification method of manhole cover hidden dangers by integrating target detection and segmentation according to claim 3 is characterized in that: The specific implementation process of step 3 is as follows: 3-1 Save the output features of each layer of the encoder, and concatenate the features of the last three layers and send them to the feature pyramid FPN for feature aggregation. After processing, the aggregated features are concatenated with the features output by the last three layers of encoders to obtain the skip connection features corresponding to the last three layers; 3-2 In the decoder part, the skip connection features of the fourth layer are used as the input of the fourth layer decoder, and are output after being processed by the decoder. The skip connection features corresponding to the third layer after convolution are used as the input of the third layer encoder. This is analogous to the output of the first layer decoder. The segmentation truth value is used for supervision and the pixel position aware PPA Loss function is used to update the model parameters. The prediction result output by the updated model is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

5. The integrated identification method of manhole cover hidden dangers by integrating target detection and segmentation according to claim 4 is characterized in that: The specific implementation process of step 3-1 is as follows: 3-1-1 The manhole cover image data is adjusted to a fixed size and then input into the encoder module composed of PVTv2-B4. The encoder module has four layers of PVTv2-B4 encoders, denoted as e x , x = {1, 2, 3, 4}; the manhole cover image is input to encoder e1, which processes it and inputs it to the next layer of encoder e2, and so on. The feature map output of each encoder is recorded as 3-1-2 The outputs of encoders e2, e3, and e4 Input into FPN for feature aggregation, and the aggregated feature map is recorded as Its width, height, and number of channels are similar to same; 3-1-3 Aggregate feature map Perform average pooling and convolution operations to obtain feature maps Its size and Same; feature maps are obtained by bilinear interpolation and convolution at the same time Its size and same; will Respectively Splice to get the skip connection features of the last three layers 6. The integrated identification method of manhole cover hidden dangers by integrating target detection and segmentation according to claim 5 is characterized in that: The specific implementation process of step 3-2 is as follows: the decoder module has four layers of Swin-Transformer decoders, denoted as d x , and the output feature map of each layer of the decoder is recorded as The skip connection features corresponding to the fourth layer Input into the fourth layer decoder d4, and after being processed by the decoder, the fourth layer decoder output feature is obtained Later Perform convolution operation to make it equal to Same as Together they serve as the input of d3; the same is true for d2; d1 only accepts the output of d2 and outputs the feature map after processing Will Use the segmentation truth value S GT Supervision is performed and PPA Loss is used to update the model parameters; the feature map P output by the updated model ring That is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

7. The integrated identification method of manhole cover hidden dangers by integrating target detection and segmentation according to claim 6 is characterized in that: The specific implementation process of step 4 is as follows: 4-1 Through self-service sampling, three different manhole cover data subsets D1, D2, and D3 are randomly sampled with replacement, which are used to train three independent YOLO v81-cls subnetworks respectively; after each subnetwork is trained, the category of manhole cover hidden dangers is independently predicted; the prediction result of each subnetwork is recorded as 4-2 When fusing the prediction results of the three YOLO v81-cls subnetworks and the initial YOLO v8x model, a weighted voting mechanism is used to set the prediction result weight of the YOLO v8x model to three votes, and the prediction result weight of the three YOLO v81-cls subnetworks to one vote, and perform weighted summation to obtain the discrimination result P of the manhole cover hidden danger category. well , the specific formula is: 4-3 For the well circle part, the output well circle prediction result P ring After masking, it is input into another YOLO v8x model for training, and finally the discrimination result P of the well circle hidden danger category is output. circle ; 4-4 Obtain the result of manhole cover hidden danger identification P well And the result of identifying the hidden dangers of the wellbore P circle Finally, the category fusion is performed according to the following rules to obtain the final hidden danger category P final :If P well The manhole cover is intact and P circle If the wellbore is intact, then P final is the well circle problem; if P well The manhole cover is intact and P circle If there is a problem with the wellbore, then P final is the well circle problem; if P well If the manhole cover is not intact, P is not considered. circle , P final Equivalent to P well Hazard category.

8. A manhole cover hidden danger integrated identification system integrating target detection and segmentation, used to implement the manhole cover hidden danger integrated identification method according to any one of claims 1 to 7, characterized in that: Includes the following modules: The data processing module is used to obtain the image data of hidden dangers of manhole covers, pre-process the data according to categories, and divide the cells and anchor frames; The target positioning module uses the preprocessed manhole cover hidden danger image to locate the target using the YOLO v8x model and PVT v The encoder module composed of 2-B4 obtains the prediction results of the manhole cover and the manhole circle; The category judgment module adopts an integrated classification network strategy. Based on the prediction results of manhole covers and manhole rings, combined with a weighted voting mechanism, it obtains the discrimination results of the manhole cover hidden danger category and the discrimination results of the manhole ring hidden danger category respectively. Finally, the hidden danger category is obtained by combining the two discrimination results.

Images