A manhole cover hidden danger integrated identification method and system fusing target detection and segmentation

Abstract

The application discloses a manhole cover hidden danger integrated recognition method and system fusing target detection and segmentation, which firstly carries out pretreatment on manhole cover picture data, and divides cells and anchor frames.Secondly, multi-scale features of the pretreated manhole cover picture are extracted through YOLO v8x, and a manhole cover prediction result is outputted.A Bagging integrated learning strategy is adopted, and based on the manhole cover prediction result, a weighted voting mechanism is used to obtain a discrimination result of a manhole cover hidden danger category.Then, the pretreated manhole cover picture is inputted into an encoder module composed of PVTv2-B4 to extract manhole ring features, and a manhole ring prediction result is outputted.Meanwhile, based on the manhole ring prediction result, the manhole ring is analyzed again to obtain a discrimination result of a manhole ring hidden danger category.Finally, the discrimination results of the manhole cover and the manhole ring are combined to obtain a hidden danger category.The application improves the fine recognition ability of the manhole cover state in a complex scene, and realizes accurate outlining and positioning of the manhole ring boundary.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of manhole cover hidden danger identification, and relates to a manhole cover hidden danger integrated identification method and system fusing target detection and segmentation. BACKGROUND

[0002] With the continuous development of society, urban infrastructure construction is increasingly perfect, and the safety of manhole covers, as one of the indispensable infrastructures in the city, is also of great concern. However, in recent years, manhole cover hidden dangers have occurred frequently, and many serious safety accidents have been caused by the displacement, damage or loss of manhole covers. The root of these manhole cover problems may be due to unreasonable design, improper installation, material aging, vehicle crushing, etc. These problems may not be quickly discovered by conventional manual inspection or monitoring systems, and it is difficult to take repair or replacement measures in time, which brings great risks to vehicles, pedestrians, communication facilities, etc., and becomes a hidden danger to urban safety. Therefore, real-time and accurate identification of manhole cover hidden dangers is of great significance to urban public safety management.

[0003] The types of manhole covers in the city are various, and the characteristics of manhole covers in different environments differ greatly, which brings great challenges to the identification of manhole cover hidden dangers. Researchers have never stopped studying manhole cover hidden dangers, and changes in visual, vibration, sound wave and other physical signals are generally used for detection of manhole cover hidden dangers. The identification accuracy of a single detection sensor is not as good as that of multiple sensors working together. However, this multi-sensor detection method has high requirements for sensors, installation and environment, and therefore is greatly limited in practical application. Based on this, manhole cover hidden danger detection methods using image features or vibration features on the surface of the manhole cover have gradually developed, but the manually extracted features are affected by prior knowledge, and their adaptability and generalization ability need to be improved, and the optimization process is complex and tedious. At the same time, the influence of various complex factors in the urban environment, such as vehicle flow, climate conditions, etc., causes the manually extracted features to be prone to missed detection or false detection in similar complex scenes, making it difficult to ensure the identification accuracy.

[0004] In recent years, there have been two main types of research on manhole cover hidden danger identification: 1. Feature extraction method of manhole cover: mainly analyzing the image or vibration signal features of the manhole cover, and extracting effective features using mathematical methods, combining with machine learning classifiers or artificial neural networks to get the final output. 2. Research on deep learning models and algorithms: most of them use different convolutional networks for feature extraction, and some methods preprocess the physical signals of the manhole cover into images and input them into subsequent classification models for processing. These methods analyze the influence of manhole cover hidden dangers on environmental signals, the influence of different scenes on manhole cover hidden danger features, and apply heuristic methods or machine learning for decision-making, among which deep learning models have been proven to have significant advantages in various performance indicators in manhole cover hidden danger identification. SUMMARY

[0005] The object of the present application is to overcome the deficiencies of the prior art and provide a manhole cover hidden danger integrated recognition method and system that fuses target detection and segmentation. The method first uses a YOLO (You Only Look Once) v8x model for target detection pre-training, focusing on different types of manhole cover recognition, while excluding the well circle as a training target. At the same time, a segmentation model based on a PVTv2-B4-Swin-Transformer cascade U-shaped composite structure is constructed for accurate segmentation of the well circle. For each anchor box generated by the target detection, three Yolo v8l-cls classification networks are trained using a bootstrap sampling method. The three classification models independently predict and evaluate the results with the original target detection model's own class prediction results. In the final class decision-making process, the class weight predicted by the target detection model itself is given three votes, while each classification model's prediction result accounts for one vote. In this way, weighted voting is performed to finally fuse the manhole cover and well circle classes to recognize manhole cover hidden dangers.

[0006] In a first aspect, the present application provides a manhole cover hidden danger integrated recognition method that fuses target detection and segmentation, comprising the following steps:

[0007] Target detection:

[0008] 1. Preprocess the collected manhole cover picture data, including size standardization, normalization processing, and dividing cells and anchor boxes to meet the needs of subsequent processing.

[0009] 2. Extract the multi-scale features of the preprocessed manhole cover picture through the YOLO v8x model, perform multi-scale feature fusion and anchor box assignment, and output the manhole cover prediction results.

[0010] Semantic segmentation:

[0011] 3. Input the preprocessed manhole cover picture in the first step into the encoder module composed of PVTv2-B4 to extract well circle features, and output the well circle prediction results.

[0012] Integrated learning:

[0013] 4. Using the Bagging integrated learning strategy, based on the manhole cover prediction results, using the weighted voting mechanism, the discrimination results of the manhole cover hidden danger class are obtained; at the same time, based on the well circle prediction results, the well circle is analyzed again to obtain the discrimination results of the well circle hidden danger class; combining the two discrimination results obtains the hidden danger class.

[0014] As a preferred, step 1 is specifically:

[0015] 1-1 Adjust all manhole cover images I to a fixed size (H, W, 3) and pad the insufficient part (Padding) to maintain the aspect ratio of the image.

[0016] 1-2 Normalize the size-adjusted manhole cover image I to map the pixel values to the range [0, 1] to obtain the normalized manhole cover image I pre .

[0017] 1-3 Divide the normalized manhole cover image I pre in step 1-2 into SxS cells, and assign several anchor boxes in each cell. The size and proportion of the anchor boxes are set during model training.

[0018] As a preferred, step 2 is specifically:

[0019] 2-1 Input the normalized manhole cover image I pre in step (1) into the backbone network of the YOLO v8x model for feature extraction. The backbone of YOLO v8x contains multiple convolutional layers and residual connection modules to gradually extract local and global features in the image, forming multi-scale feature maps at different levels

[0020] 2-2 Input the multi-scale feature maps in step 2-1 into the Neck part of the YOLO v8x model for feature aggregation of different scales. Obtain the feature map F neck .

[0021] 2-3 Input the aggregated feature map P neck into the head network Head of YOLO v8x; Head consists of a detection head and a classification head. The detection head generates boundary box prediction, confidence prediction and class prediction for each anchor box, and then inputs the classification head for classification. Finally, output the manhole cover prediction result P class .

[0022] As a preferred, the specific implementation process of step 3 is as follows:

[0023] 3-1 Save the output features of each layer of the encoder, and send the features of the last three layers to the feature pyramid FPN for feature aggregation. The aggregated features are processed and concatenated with the features output by the last three layers of the encoder, respectively, to obtain the corresponding skip connection features of the last three layers.

[0024] 3-2In the decoder part, the skip connection features of the fourth layer are taken as the input of the fourth layer decoder, and after the decoder processing, the output is taken as the input of the third layer corresponding to the convolution after the skip connection features of the third layer, and the first layer decoder output is taken in turn, supervised by the segmentation true value and using the pixel position perception PPA Loss loss function to update the model parameters, and the prediction result output by the updated model is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

[0025] As preferred, the step 3-1 is specifically implemented as follows:

[0026] 3-1-1After adjusting the manhole cover image data to a fixed size, it is input into the encoder module composed of PVTv2-B4, and the encoder module has four layers of PVTv2-B4 encoders, denoted as e x ,x={1,2,3,4};The manhole cover image is input into the encoder e1, and after processing, it is input into the next layer of encoder e2, and the feature map output of each encoder is denoted as

[0027] 3-1-2The outputs of the encoders e2, e3, and e4 are input into the FPN for feature aggregation, and the aggregated feature map is denoted as The width, height, and channel number are the same as .

[0028] 3-1-3The aggregated feature map is subjected to average pooling and convolution operation to obtain the feature map , which has the same size as ; at the same time, the feature map is obtained through bilinear interpolation and convolution, which has the same size as ; and is spliced with , respectively, to obtain the skip connection features of the last three layers

[0029] As preferred, the step 3-2 is specifically implemented as follows:

[0030] The decoder module has four layers of Swin-Transformer decoders, denoted as d x , and the output feature map of each layer of decoder is denoted as The fourth layer corresponding to the skip connection feature is input into the fourth layer decoder d4, and after the decoder processing, the fourth layer decoder output feature is obtained is subjected to convolution operation to make its size the same as , and combined with ​together as input of d3. d2 is the same. d1 only accepts the output of d2, processes it and outputs feature maps will be with segmentation ground truth S GT is supervised and model parameter is updated using PPALoss. The output feature maps P ring is the final prediction result of semantic segmentation part, outputting the prediction result of well circle.

[0031] As preferred, step (4) is specifically:

[0032] 4-1 By Bootstrap Sampling, three different manhole cover data subsets D1, D2 and D3 are randomly sampled with replacement, which are used to train three independent YOLO v8l-cls subnetworks respectively. After the training of each subnetwork is completed, independent prediction of manhole cover hidden danger categories is performed. The prediction result of each subnetwork is denoted as

[0033] 4-2 When fusing the prediction results of the three YOLO v8l-cls subnetworks and the initial YOLO v8x model, a weighted voting mechanism is used, the prediction result of the YOLO v8x model is weighted as three votes, and the prediction results of the three YOLO v8l-cls subnetworks are weighted as one vote, and weighted summation is performed to obtain the discrimination result P well of manhole cover hidden danger categories. The specific formula is:

[0034]

[0035] 4-3 For the well circle part, the well circle prediction result P ring output in step (5) is input into another YOLO v8x model after mask, and finally the discrimination result P circle of well circle hidden danger categories is output.

[0036] 4-4 After obtaining the discrimination result P well (manhole cover intact, manhole cover missing, manhole cover damaged, manhole cover not covered) of manhole cover hidden danger and the discrimination result P circle (well circle intact, well circle problem) of well circle hidden danger, the final hidden danger category P final is obtained by classifying fusion according to the following rules: if P well is “manhole cover intact” and P circle is “well circle intact”, then P final is “well circle problem”; if P well is “manhole cover intact” and P circle is “well circle has problem”, then P final is “well circle problem”; if P wellNot "well", P circle , P final is equivalent to P well hidden danger category.

[0037] In a second aspect, the present application provides a manhole cover hidden danger integrated recognition system that fuses target detection and segmentation, comprising the following modules:

[0038] A data processing module is configured to obtain manhole cover hidden danger image data, pre-process the data according to categories, and divide cells and anchor boxes.

[0039] A target positioning module is configured to obtain manhole cover prediction results and well ring prediction results from an encoder module composed of a YOLOv8x model and a PVTv2-B4 through pre-processed manhole cover hidden danger images.

[0040] A category determination module is configured to obtain manhole cover hidden danger category determination results and well ring hidden danger category determination results based on the manhole cover prediction results and the well ring prediction results by using an integrated classification network strategy and combining a weighted voting mechanism, and finally obtain a hidden danger category by fusing the two determination results.

[0041] The present application has the following advantages:

[0042] 1. The present application uses a Yolo v8 model for target detection, which improves the model's fine-grained recognition ability of manhole cover states in complex scenes. Even in environments with severe light changes, severe occlusions, or complex backgrounds, the model can effectively distinguish various states of manhole covers such as normal, damaged, and missing.

[0043] 2. The present application combines the PVTv2-B4's ability to efficiently process multi-scale visual features with the Swin-Transformer's outstanding performance in global information modeling and self-attention mechanisms. Through a U-shaped architecture, the model can effectively capture low-level image details while fully extracting high-level semantic features, achieving precise outlining and positioning of well ring boundaries.

[0044] 3. The present application selects a self-sampling method to train three Yolo v8l-cls classification networks. These three classification models independently predict and evaluate their results with the original target detection model's own category prediction results. Specifically, in the final category decision-making process, the target detection model itself predicts a category weight of three votes, while each classification model's prediction result accounts for one vote. In this way, weighted voting is performed to finally determine the accurate category label of the manhole cover. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 Structure diagram of the present application;

[0046] Figure 2 A structural diagram of a segmentation model;

[0047] Figure 3 P-R curve of the present application on a high-resolution data verification set. DETAILED DESCRIPTION

[0048] Further analysis will be made below in combination with specific embodiments and the drawings.

[0049] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] The terms “comprise” and “have” and any variations thereof mentioned in the embodiments of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the steps or units listed in the specification, but can optionally further comprise other steps or units not listed, or can optionally further comprise other steps or units inherent to the process, method, product or device.

[0051] In a first aspect, the present application provides a manhole cover hidden danger integrated recognition method combining target detection and segmentation, which comprises the following steps:

[0052] Target detection:

[0053] 1. Preprocessing the collected manhole cover picture data, including size standardization, normalization processing, and dividing cells and anchor boxes to adapt to the needs of subsequent processing.

[0054] 2. The local and global features of the preprocessed manhole cover picture are extracted by the convolution layer and residual connection of the backbone network of YOLO v8x to generate the corresponding feature map. The feature map is input to the Neck part for multi-scale feature fusion and anchor box assignment. Finally, the fused feature map and anchor box are input to the detection head of the Head part to generate the boundary box prediction, confidence prediction and class prediction of each anchor box; then input to the classification head for classification, output the manhole cover prediction result.

[0055] Semantic segmentation:

[0056] 3. As shown in Figure 2 , the preprocessed manhole cover picture in the first step is input into the encoder module composed of PVTv2-B4 to extract the well circle features, and output the well circle prediction result.

[0057] 3-1. The output features of each layer of the encoder are saved, and the features of the last three layers are spliced and sent to the FPN (Feature Pyramid Network) feature pyramid for feature aggregation. The aggregated features are processed and spliced with the original features of the last three layers of the encoder, respectively, to obtain the corresponding skip connection features of the last three layers.

[0058] 3-2. In the decoder part, the skip connection features of the fourth layer are used as the input of the fourth layer decoder, and after decoding, the output is used as the input of the third layer encoder after convolution. In this way, the first layer decoder outputs, and the segmentation ground truth is used for supervision and the PPA (Pixel Position Aware) loss function is used for model parameter update. The prediction result output by the updated model is the final prediction result of the semantic segmentation part, and the well cover prediction result is output.

[0059] Ensemble learning:

[0060] 4. Bagging ensemble learning strategy is adopted, that is, three YOLO v8l-cls subnetworks are trained for well cover hazard learning, based on the well cover prediction result, the weighted voting mechanism is used to fuse the bounding box and classification result; at the same time, another YOLOv8x model is used to predict the well circle based on the well circle prediction result, and the well circle hazard category is classified. Finally, the hazard category is obtained by combining the determination results of the two models.

[0061] As preferred, step (1) is specifically:

[0062] 1-1. Adjust all well cover pictures I to a fixed size (H, W, 3), and pad the insufficient part (Padding) to maintain the aspect ratio of the well cover image.

[0063] 1-2. Normalize the well cover image I to map the pixel value to the range [0, 1], and the specific operation is to divide the pixel value of the original image by 255, that is:

[0064]

[0065] where I pre represents the normalized well cover image, and the pixel value is scaled to the range [0, 1].

[0066] 1-3. Divide the well cover image I pre preprocessed in step 1-2 into SxS cells, and assign one or more anchor boxes in each cell. The size and ratio of the anchor box are set during model training.

[0067] As preferred, step (2) is specifically:

[0068] 2-1 The image I that has been preprocessed and resized in step (1) pre The input is fed into the backbone network of the YOLO v8x model for feature extraction. The YOLO v8 backbone contains multiple convolutional layers and residual connection modules, used to progressively extract local and global features from the image, forming feature maps at different levels. The feature extraction process can be represented as:

[0069]

[0070] Where Bone(·) represents convolution, deconvolution, residual connections, and other operations in the backbone network. This represents the feature map at level i.

[0071] 2-2 The multi-scale feature map in step 2-1 The input is fed into the Neck part of the YOLO v8x model for feature aggregation at different scales. The Neck part contains the following modules:

[0072] The SPPF (Spatial Pyramid Pooling-Fast) module reduces the feature map to different spatial scales through pooling operations at different scales, thereby obtaining multi-scale information to adapt to targets of different sizes. The input feature map... After processing by the SPPF module, multi-scale feature maps are obtained.

[0073] The PAA (Probabilistic Anchor Assignment) module is used for intelligent anchor box assignment. It assigns an adaptive anchor box to each feature location based on prediction confidence to accommodate manhole covers of different sizes and shapes. The PAA module selects the optimal anchor box by calculating the probability distribution of each candidate box.

[0074] The PAN (PathAggregation Network) module further aggregates feature maps at different levels, performing multi-scale feature map aggregation. By integrating multi-level features through upsampling and downsampling operations, the model's ability to detect targets at multiple scales is enhanced. The aggregated feature map is F. neck The specific formula is as follows:

[0075]

[0076] Where PAN(·) represents the multi-level feature aggregation operation in the PAN module.

[0077] 2-3 The aggregated feature map F neckInput into the head network (Head) of YOLO v8x. The Head consists of two main parts: a detection head (for bounding box regression) and a classification head (for target class prediction).

[0078] The detection head performs bounding box regression on anchor boxes through convolution and deconvolution operations, predicting the specific position and size of each anchor box. The output of the detection head is the bounding box parameters B and the target confidence C:

[0079] (B, C) = DH(F neck )

[0080] Where B represents the regression value of the bounding box, C represents the target confidence, which is used to determine whether the candidate box contains a manhole cover target; DH(·) represents a series of convolution and deconvolution operations in the detection head.

[0081] The classification head extracts global features from the feature map through global average pooling operations and performs class prediction for each detected target. The class prediction output is P class .

[0082] The final output of the classification head is the final target detection result of the YOLO v8x model.

[0083] As a preferred, step 3-1 is specifically:

[0084] 3-1-1 The manhole cover image data is sorted into a uniform size (352*352*3) and then input into the encoder module composed of PVTv2-B4. Specifically: the encoder module has four layers of PVTv2-B4 encoders, denoted as e x ,x={1,2,3,4}. The manhole cover image is input into the encoder e1, and after processing, it is input into the next layer of encoder e2, and so on. The feature map output of each encoder is denoted as

[0085] 3-1-2 The outputs of the encoders e2, e3, and e4 in step (3) are input into the FPN for feature aggregation. The aggregated feature map is denoted as , which has the same width, height, and channel number as .

[0086] 3-1-3 The aggregated feature map is subjected to mean-pooling and convolution operations to obtain the feature map , which has the same size as ; at the same time, through bilinear interpolation and convolution, the feature map is obtained, which has the same size as ​The same is true. respectively, and the third layer of the jump connection feature is obtained by splicing The third layer of the jump connection feature is obtained by splicing

[0087] As a preferred, step 3-2 is specifically:

[0088] The decoder module has four layers of Swin-Transformer decoder, denoted as d x , and the output feature map of each layer of decoder is denoted as The fourth layer of the jump connection feature is obtained by splicing Input into the fourth layer decoder d4, and after decoder processing, the fourth layer decoder output feature is obtained. After convolution operation, the size is the same as , and is used as the input of d3 together with . d2 is the same. d1 only accepts the output of d2, and after processing, the output feature map is obtained. The segmentation ground truth S GT is used for supervision, and PPALoss is used for model parameter update. The feature map P ring output by the updated model is the final prediction result of the semantic segmentation part, and the well circle prediction result is output.

[0089] As a preferred, step (4) is specifically:

[0090] 4-1 Through Bootstrap Sampling, three different manhole cover data subsets D1, D2 and D3 are randomly sampled with replacement, which are used to train three independent YOLO v8l-cls subnetworks. After the training of each subnetwork is completed, the manhole cover hidden danger class is independently predicted. The prediction result of each subnetwork is denoted as

[0091] 4-2 When fusing the prediction results of the three YOLO v8l-cls subnetworks and the initial YOLO v8x model, a weighted voting mechanism is used, the prediction result of the YOLO v8x model is set to three votes, the prediction result of the three YOLO v8l-cls subnetworks is set to one vote, and weighted summation is performed to obtain the discrimination result P well of the manhole cover hidden danger class, and the specific formula is:

[0092]

[0093] 4-3 For the well circle part, the well circle prediction result P ringAfter being masked, the input is input into another YOLO v8x model for training, and finally the discrimination result P of the well cover hidden danger category is output circle .

[0094] 4-4 obtains the well cover hidden danger discrimination result P well (well cover intact, well cover missing, well cover damaged, well cover not covered) and the well ring hidden danger discrimination result P circle (well ring intact, well ring problem) are fused according to the following rules to obtain the final hidden danger category P final : if P well is "well cover intact" and P circle is "well ring intact", then P final is "well ring problem"; if P well is "well cover intact" and P circle is "well ring has a problem", then P final is "well ring problem"; if P well is not "well cover intact", then P circle is not considered, and P final is equivalent to P well hidden danger category.

[0095] In a second aspect, the present application provides a well cover hidden danger identification system for implementing the above method, comprising:

[0096] A data processing module is configured to obtain well cover hidden danger image data through crawling, AIGC and the like. The data is divided according to categories, and data augmentation, size normalization and other preprocessing are performed, and cells and anchor boxes are divided;

[0097] A target positioning module simultaneously deploys two independently running neural network architectures, which are respectively composed of an encoder module of a YOLO v8x model and a PVTv2-B4 for a category detection task of a well cover and a segmentation and classification task of a well ring, to obtain well cover prediction results and well ring prediction results. Among them, the target detection network realizes accurate positioning and category identification of various well covers, and the segmentation network is used for pixel-level segmentation of the well ring area and further refines the category label.

[0098] A category determination module adopts an integrated classification network strategy and combines a weighted voting mechanism to optimize the final decision-making process. The module integrates the prediction results of multiple classifiers, reasonably screens and adjusts the weights of the candidate anchor boxes, and finally outputs the judgment category label after sufficient fusion.

[0099] Embodiment:

[0100] In the process of manhole cover image recognition and classification, due to the difficulty in distinguishing the boundary between the manhole cover and the well ring, and the large difference between the same type of manhole cover, the characteristics of the manhole cover and the well ring are difficult to guarantee the recognition accuracy. Therefore, in order to better improve the classification accuracy of the model for manhole cover classification, the problems to be solved are: how to realize multi-target detection and improve the fine classification and generalization ability in the classification task as much as possible.

[0101] Based on this, the embodiment of the application provides a manhole cover hidden danger integrated recognition method based on target detection and segmentation. The application adopts a YOLO v8x model for target detection pre-training, focuses on the recognition of different types of manhole covers, and excludes well rings as training targets. At the same time, a segmentation model based on the cascade U-shaped composite structure of PVTv2-B4-Swin-Transformer is constructed for accurate segmentation of well rings. For each anchor box generated by target detection, three Yolo v8l-cls classification networks are trained using a bootstrap sampling method. The three classification models independently predict and evaluate the results with the original target detection model's own class prediction results. In the final class decision process, the class weight predicted by the target detection model itself is given three votes, and the prediction results of each classification model are given one vote each. In this way, weighted voting is performed, and the manhole cover and well ring classes are finally fused to realize manhole cover hidden danger recognition.

[0102]

[0103] The manhole cover hidden danger recognition method based on target detection and segmentation, such as Figure 1 comprises the following steps:

[0104] Step 1: Two independent neural networks are deployed to be responsible for the class detection of the manhole cover and the segmentation and classification of the well ring, respectively. The target detection network is used to realize the positioning and recognition of the manhole cover, and the segmentation network is focused on the pixel-level segmentation and class refinement of the well ring. The specific steps are as follows:

[0105] Step 1: A Yolo v8x target detection model architecture is used to train a specific image dataset with supervised learning. The dataset in this stage contains multiple different types of manhole cover sample images, aiming to strengthen the model's ability to accurately detect and locate various manhole cover entities and the preliminary class distinction function. Yolo v8x usually uses a combined loss function, which can be represented as:

[0106] L=ω coord L coord +ω conf L conf +ω class L class

[0107] where L coord is the position loss, L​conf is the confidence loss, L class is the classification loss, ω coord , ω conf and ω class are weight coefficients for balancing these losses.

[0108] Step (2): A PVTv2-B4-Swin-Transformer cascaded U-shaped composite structure is proposed by using some elements of PVTv2-B4, SwinTransformer and U-Net architecture. By utilizing the deep feature extraction and fusion capability contained in the model architecture, the accurate delineation of the well circle region boundary and the accurate decision of the region to which the internal pixels belong are ensured. The model utilizes the attention mechanism in the Transformer encoder.

[0109]

[0110] where Q, K and V represent the query vector, key vector and value vector in the attention mechanism, respectively. k is the dimension of the key vector.

[0111] Step (3): After Step 2 is completed, the output well circle segmentation mask marks whether each point pixel belongs to the target well circle structure. The segmentation mask is point multiplied with the well circle image. The pixels of the well circle part in the original image are marked as significant values in the mask, which are retained and strengthened in the point multiplication process, while the pixels of the background and other non-well circle parts are faded or disappeared after being multiplied by the lower or zero values in the corresponding positions of the mask.

[0112] Step (4): The integrated classification network strategy and re-voting mechanism are adopted to integrate the prediction results of multiple classifiers and screen and adjust the weights of the candidate regions, and finally the category judgment of the manhole cover hidden danger is obtained. The specific steps are as follows:

[0113] Step 1: For each anchor box generated by the target detection algorithm, a multi-model ensemble learning strategy is adopted, which is specifically implemented as training three Yolo v8l-cls subnetworks independently, each of which focuses on the prediction of the object class in the anchor box. Bootstrap sampling is used to prevent overfitting, and different data subsets are used to train each classification network.

[0114] Step 2: In the decision fusion stage, four key information sources are considered: the basic prediction class of the target in each candidate box from the original target detection algorithm, and the prediction output of the three Yolo v8l-cls classification networks. Specifically, the initial target detection algorithm is given a higher degree of trust, and its prediction class has a weight of three votes in the voting process, while each independently trained classification network has a weight of one vote.

[0115]

[0116] where {c i = c} is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

[0117] Step 3: A separate Yolo v8x model is used to perform deep secondary analysis on the images processed by the segmentation network, with the goal of accurately identifying whether there is a quality problem with the well circle. Through in-depth learning of the training samples, the model has high sensitivity and accurate discrimination ability for the state of the well circle. Finally, the model can master the characteristics of the well circle in the "circle-good" (good well circle) and "circle-problem" (well circle with problems) states and form accurate identification patterns.

[0118] Step 4: Combining Step 1, Step 2 and Step 3, when the well cover itself remains intact, but the well circle around it has obvious problems, such cases are classified into the "circle" category. Conversely, for samples with damaged well covers, they are classified according to their actual damage degree and type.

[0119] For the training of the above model, the invention gives a greater weight to the class prediction branch (classification branch) and sets it to 5 to emphasize the training intensity of the model in distinguishing different categories. In terms of training strategy, a phased training method is chosen, with each training cycle containing 100 iterations (epochs), and a total of two such cycles, i.e. a total of 200 training rounds. Within each cycle, especially in the learning rate adjustment scheme of the segmentation network, cosine annealing is used with parameters set to GFLOPs = 92.81 MMac, Params = 126.72M to ensure that the learning rate has a reasonable decline curve during training, thereby facilitating the convergence and generalization performance of the model. The initial learning rate is set to the highest value of 5e-5 at the beginning of the first cycle and decreases according to the cosine annealing rule as the training progresses, and decreases to the lowest value of 1e-7 at the end of the second cycle. This dynamic learning rate adjustment mechanism helps the model to learn quickly in the early stage of training, and to fine-tune the model parameters in the later stage, effectively suppressing the overfitting phenomenon. In order to prove the effectiveness of the model, the target positioning module and the category determination module of the model are tested and the P-R curve is drawn, as shown in Figure 3 .

[0120] The computer program product of the readable storage medium provided by the embodiment of the present application comprises a computer readable storage medium storing program codes, the program codes comprise instructions for executing the method described in the foregoing method embodiments, and the specific implementation can be referred to the foregoing method embodiments, which will not be described here. The functions, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of software products, which are stored in a storage medium and comprise instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0121] Finally, it should be noted that: the above-described embodiments are merely specific implementations of the present application, which are used to illustrate the technical solutions of the present application, rather than limit the same, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art within the technical range disclosed by the present application, they can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and all should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An integrated identification method for manhole cover hazards that combines target detection and segmentation, characterized in that, Includes the following steps: Step 1. Preprocess the collected manhole cover image data and divide it into cells and anchor frames; Step 2. Extract multi-scale features from the preprocessed manhole cover image using the YOLO v8x model, perform multi-scale feature fusion and anchor box assignment, and output the manhole cover prediction results; Step 3. Input the preprocessed manhole cover image into the encoder module composed of PVTv2-B4 for manhole ring feature extraction, complete semantic segmentation, and output the manhole ring prediction result. The specific implementation process is as follows: Step 3-1 saves the output features of each layer of the encoder, and concatenates the features of the last three layers into the Feature Pyramid (FPN) for feature aggregation. The aggregated features are then processed and concatenated with the output features of the last three encoder layers to obtain the skip connection features corresponding to the last three layers. The specific implementation process is as follows: Step 3-1-1: After adjusting the manhole cover image data to a fixed size, input it into the encoder module composed of PVTv2-B4 encoders. The encoder module has four layers of PVTv2-B4 encoders, denoted as e. x x = {1, 2, 3, 4}; the manhole cover image is input to encoder e1, processed by it, and then input to the next layer encoder e2, and so on. The feature map output of each encoder is denoted as... Step 3-1-2 converts the outputs of encoders e2, e3, and e4. The input is fed into the FPN for feature aggregation, and the aggregated feature map is denoted as... Its width, height, and number of channels are all related to same; Step 3-1-3 aggregates the feature map The feature map is obtained by performing average pooling and convolution operations. Its size and The same; feature maps are obtained simultaneously through bilinear interpolation and convolution. Its size and Same; will respectively with By splicing the data, we obtain the skip connection features of the last three layers. Step 3-2: In the decoder section, the skip connection features of the fourth layer are used as input to the fourth layer decoder. After processing by the decoder, the output is convolved and used as input to the skip connection features corresponding to the third layer encoder. This process continues until the output of the first layer decoder. Then, the model is supervised using ground truth segmentation data and updated using the pixel position-aware PPALoss loss function. The updated model output is the final prediction result for the semantic segmentation part, outputting the well circle prediction result. The specific implementation process is as follows: The decoder module has four layers of Swin-Transformer decoders, denoted as d. x And the output feature map of each decoder layer is denoted as The skip connection feature corresponding to the fourth layer The input is fed into the fourth-layer decoder d4, and after processing by the decoder, the output features of the fourth-layer decoder are obtained. Later Perform a convolution operation to make its size equal to Same as, and with Both are used as input to d3; d2 is processed similarly; d1 only accepts the output of d2, and the processed output is a feature map. Will Use the segmentation truth value S GT Supervision is performed, and model parameters are updated using PPA Loss; the updated model outputs a feature map P. ring This is the final prediction result of the semantic segmentation part, outputting the well circle prediction result; Step 4. Using a Bagging ensemble learning strategy, based on the manhole cover prediction results, a weighted voting mechanism is used to obtain the classification results of manhole cover hazard categories; Simultaneously, a secondary analysis of the well ring is performed based on the well ring prediction results to obtain the classification results of well ring hidden danger categories; the hidden danger categories are obtained by combining the two classification results.

2. The integrated identification method for manhole cover hazards based on fusion of target detection and segmentation as described in claim 1, characterized in that, The implementation process of step 1 is as follows: Step 1-1: Adjust all manhole cover images I to a fixed size (H, W, 3), and add padding to fill any gaps to maintain the aspect ratio of the manhole cover images; Steps 1-2 normalize the resized manhole cover image I by mapping pixel values ​​to the range [0,1], resulting in the normalized manhole cover image I. pre ; Steps 1-3: Normalize the manhole cover image I pre Divide the data into S×S cells, and assign several anchor boxes to each cell.

3. The integrated identification method for manhole cover hazards based on fusion of target detection and segmentation as described in claim 2, characterized in that, The specific implementation process of step 2 is as follows: 2-1 Normalized manhole cover image I pre The input is fed into the backbone of the YOLO v8x model for feature extraction. The YOLO v8x backbone contains multiple convolutional layers and residual connection modules, which are used to progressively extract local and global features from the image, forming multi-scale feature maps at different levels. 2-2 Multi-scale feature maps The input is fed into the Neck part of the YOLO v8x model, where features are aggregated at different scales to obtain the feature map F. neck ; 2-3 The aggregated feature map F neck The data is input into the YOLO v8x head network; the head consists of a detection head and a classification head. The detection head generates bounding box predictions, confidence predictions, and class predictions for each anchor box, which are then input into the classification head for classification. Finally, the manhole cover prediction result P is output. class .

4. The integrated identification method for manhole cover hazards based on fusion of target detection and segmentation as described in claim 3, characterized in that, The specific implementation process of step 4 is as follows: 4-1 Three distinct subsets of manhole cover data, D1, D2, and D3, were obtained through bootstrapping and random sampling with replacement. These subsets were used to train three independent YOLO v8l-cls subnetworks. After training, each subnetwork independently predicted the category of manhole cover hazards. The prediction result of each subnetwork is denoted as... 4-2 When fusing the prediction results of the three YOLO v8l-cls subnetworks and the initial YOLO v8x model, a weighted voting mechanism is used. The prediction results of the YOLO v8x model are assigned three votes, and the prediction results of the three YOLO v8l-cls subnetworks are assigned one vote. A weighted sum is then performed to obtain the discrimination result P for the manhole cover hazard category. well The specific formula is as follows: 4-3 For the well ring part, the output well ring prediction result P ring After being masked, the input is given to another YOLO v8x model for training, and the final output is the discrimination result P of the well ring hazard category. circle ; 4-4 Obtain the result of manhole cover hazard identification P well Well-hole hazard identification results P circle Then, the final hazard category P is obtained by category fusion according to the following rules. final If P well The manhole cover is intact and P circle If the well ring is intact, then P final This is a wellbore problem; if P well The manhole cover is intact and P circle If there is a problem with the well ring, then P final This is a wellbore problem; if P well If the manhole cover is not intact, then P is not considered. circle P final Equivalent to P well Hazard category.

5. A manhole cover hazard integrated identification system that combines target detection and segmentation, used to implement the manhole cover hazard integrated identification method according to any one of claims 1 to 4, characterized in that, Includes the following modules: The data processing module is used to acquire image data of potential hazards in manhole covers, preprocess the data according to categories, and divide the data into cells and anchor frames. The target localization module uses the preprocessed manhole cover hazard image and the encoder module composed of YOLO v8x model and PVTv2-B4 to obtain the manhole cover prediction result and the manhole ring prediction result, respectively. The category determination module adopts an integrated classification network strategy. Based on the prediction results of manhole covers and manhole rings, and combined with a weighted voting mechanism, it obtains the discrimination results of manhole cover hazard categories and manhole ring hazard categories respectively. Finally, the two discrimination results are combined and fused to obtain the hazard category.

Images