Defect semantic segmentation method and device for unmanned inspection robot

Abstract

The invention discloses a defect semantic segmentation method and device for an unmanned inspection robot, and provides a defect image semantic segmentation model based on weak supervision domain self-adaption for the problems of low efficiency, high cost and high risk of a traditional inspection mode. According to the model, target domain data distribution is optimized through resampling and data enhancement, a high-quality pseudo tag is generated in combination with a teacher-student network, and the training effect is improved. The weight of a student encoder is optimized through low-rank weight decomposition, and the model is lightened so as to adapt to the calculation requirement of unmanned inspection equipment. The method effectively improves the segmentation precision in a defect edge fuzzy scene, and is suitable for complex inspection environments such as dams, electric power, petrifaction, railways and the like.

Description

Technical Field

[0001] The present invention relates to a defect semantic segmentation method and device for an unmanned inspection robot, specifically a defect semantic segmentation method based on weakly supervised domain adaptation. Aiming at problems such as unclear defect edge information and class imbalance, it improves the segmentation accuracy in the unmanned inspection scenario and belongs to the technical field of semantic segmentation. Background Art

[0002] As an important part of China's engineering system, reservoir dams undertake multiple functions such as flood control, water supply, power generation, irrigation, and ecological regulation, and are important infrastructure for regulating water resources, optimizing allocation, preventing water disasters, and ecological protection. Due to the long-term influence of factors such as water pressure, temperature change, weathering erosion, and geological movement, dams are prone to defects such as cracks, leakage, spalling, and steel bar corrosion, which affect the structural safety of the dams. Timely and accurately identifying dam defects is the key to ensuring the safe operation of dams. In the dam safety guarantee system, regular inspections and real-time monitoring are indispensable important links. However, the traditional dam inspection methods mainly rely on manual detection or fixed sensor monitoring, which have problems such as low efficiency, high cost, limited coverage, and high danger, and are difficult to meet the development needs of modern intelligent water conservancy.

[0003] Semantic segmentation, as an important technology in the field of computer vision, can classify each pixel in an image to achieve accurate defect identification and positioning, and has important application value in industrial defect detection. However, in the application of dam inspection, problems such as diverse defect types, difficult data annotation, and blurred target edges limit the detection accuracy of existing semantic segmentation models. Facing the diversity of defect types, existing data sets are often difficult to cover comprehensively. Traditional machine vision algorithms have limitations in establishing a comprehensive model, and the reusability of the model is not high. Although existing large semantic segmentation models have strong generalization ability, there are still problems of decreased segmentation accuracy in scenarios with unclear defect edge information, making it difficult to meet the actual inspection needs. Summary of the Invention

[0004] Object of the Invention: Aiming at the problem of decreased segmentation accuracy of domain adaptation-based semantic segmentation technology in scenarios with unclear defect edge information, the present invention proposes a defect semantic segmentation method and device for unmanned inspection. The present invention introduces weakly supervised prompts and combines a teacher-student network framework to improve the semantic segmentation accuracy of dam defect images, especially having strong adaptability in defect detection with blurred defect edges. At the same time, unmanned inspection is used to solve the limitations of traditional manual defect detection tasks, such as strong subjectivity, low efficiency, many interference factors, high labor costs, poor accuracy, and high safety risks.

[0005] Technical solution: A defect semantic segmentation method for unmanned inspection robots. The object processed by the method is a three-channel optical image, mainly for scenarios where the boundaries of defect areas are unclear. The method includes the following steps:

[0006] Step 1) Data collection and preprocessing: Obtain defect images collected during dam inspection, extract different types of defect samples on the concrete surface from them, construct an initial defect dataset, resample the defect data based on the training consistency principle, and adjust the data distribution;

[0007] Step 2) Network construction and data augmentation: Construct a three-way network framework including an anchor network, a teacher network, and a student network. Perform strong data augmentation and weak data augmentation on the defect images respectively, and input the augmented images into the anchor network, the teacher network, and the student network respectively to extract defect image features;

[0008] Step 3) Generation and filtering of pseudo-labels: The teacher network generates initial pseudo-labels under the guidance of weak supervision signals (such as bounding boxes, point annotations, or rough segmentation masks), designs a filtering mechanism (confidence threshold, consistency check, etc.) to screen the pseudo-labels, and retains high-quality pseudo-labels for subsequent training;

[0009] Step 4) Network training and parameter update: Use the high-quality pseudo-labels obtained in Step 3 to guide the training of the student network, calculate the self-training loss, the anchor loss, and the contrast loss, and optimize the weights of the student network encoder in combination with the low-rank matrix update strategy. The teacher network synchronizes the parameters of the student network through the exponential moving average (EMA) mechanism, and iteratively trains to obtain a defect image semantic segmentation model;

[0010] Step 5) Model lightweighting: Perform full-stage distillation on the trained semantic segmentation model to reduce the model complexity, and combine the hierarchical segmentation strategy to reduce the computational overhead of SAM in the "segment everything" mode;

[0011] Step 6) Model deployment: Deploy the lightweight weak supervision domain adaptation semantic segmentation model to unmanned inspection equipment, collect dam surface images in real time, and input the image data into the semantic segmentation model to achieve online image segmentation and defect area recognition.

[0012] Among them, the data collection and preprocessing in Step 1) include the following processes:

[0013] 1-1) Collect different types of defect images on the dam concrete surface from the data obtained from the network and on-site shooting, and use weak supervision annotations (such as bounding boxes, point annotations, rough masks) to annotate the defect areas to generate preliminary label data;

[0014] 1-2) The defective images are sent to the resampling module based on training consistency. For each category, the online average category score ACS is calculated through the output probability map of the teacher model to dynamically estimate the distribution of each category. At each time step, the ACS is updated using the exponential moving average EMA.

[0015] 1-3) Determine the minority categories according to the obtained online average category scores. The categories with scores lower than the set value are identified as minority categories. For each category k, the sampling rate SR k is determined through normalization operations to ensure a significant increase in the sampling rate of minority categories.

[0016] 1-4) For each sample in the target domain, calculate its prediction consistency in multiple iterations, that is, the reliability of each category According to the reliability of each category Sort all the target domain images in descending order and select the top C samples as the copy-paste candidate set to ensure the high reliability of the candidate samples.

[0017] 1-5) Copy and paste the images in the candidate set to complete the resampling.

[0018] Among them, step 2) constructs a three-way network framework including an anchor network, a teacher network, and a student network. Strong data augmentation and weak data augmentation are performed on the defective images respectively, and the augmented images are input into the anchor network, the teacher network, and the student network respectively, which specifically includes the following steps:

[0019] 2-1) Initialize the anchor network, the teacher network, and the student network. The three networks use the pre-trained SAM image encoder as their basic architecture. Among them, the weights of the anchor network are set to a fixed state, that is, the weights of the anchor network are frozen and do not change following the training; the initial weights of the student network are the same as those of the anchor network, and the weights are updated following the training; the initial weights of the teacher network are the same as those of the student network, and the weights are updated following the student network.

[0020] 2-2) Perform weak data augmentation on the original data x of the target domain i and input the processed image data into the anchor network to extract features.

[0021] 2-3) Perform weak data augmentation on the data resampled in step 1 and input the processed image data into the teacher network to extract features.

[0022] 2-4) Perform strong data augmentation on the data resampled in step 1 and then input it into the student network to extract features.

[0023] Among them, in step 3), on the basis of the original teacher model generating pseudo-labels, weak supervision is introduced as a hint to generate a fixed hint set. The teacher model generates pseudo-labels based on the hint set and filters the pseudo-labels to obtain high-quality pseudo-labels to guide the training of the student network, which specifically includes the following steps:

[0024] 3-1) Prepare the weak supervision hints. Among them, the weak supervision includes: (1) Bounding box: the minimum bounding rectangle of the target defect; (2) Point annotation: randomly sample positive points in the defect area and negative points outside the area; (3) Rough mask: fit the real mask through polygon.

[0025] 3-2) Construct a hint encoder, which converts the weak supervision hints into embedding vectors of a fixed dimension. Among them, for the bounding box, map the upper left and lower right coordinates of the bounding box to position embeddings; for point annotation, the coordinates of positive and negative points are respectively encoded as position embeddings; for the rough mask, use the vertex coordinate sequence of the polygon as the position embedding.

[0026] 3-3) Input the weak supervision into the hint encoder, convert it into an embedding vector, and generate a hint set {e j}.

[0027] 3-4) The teacher model receives the weakly augmented image and the hint set, and generates pseudo-labels corresponding to class j.

[0028] 3-5) Calculate the IoU between each pseudo-label mask and the mask generated by the anchor network. If the IoU is greater than 0.5, then retain the pseudo-label.

[0029] 3-6) Calculate the prediction consistency of each pseudo-label mask under different weak augmentations. If the stability is greater than 0.8, then retain the current pseudo-label mask. The different weak augmentations refer to different forms of weak data augmentation, including setting small-angle rotation, translation, and scaling. Here, it is necessary to calculate the prediction consistency of each pseudo-label under these three different forms. If the stability exceeds 80%, retain the current pseudo-label, otherwise discard the current pseudo-label.

[0030] 3-7) Sort by the stability score, retain the mask with the highest score, and remove the mask whose IoU with it exceeds 0.7 to generate a high-quality pseudo-label set.

[0031] Step 4) Use the high-quality pseudo-labels generated in step 3) to guide the training of the student network, and update the weights of the student network and the teacher network, which specifically includes the following steps:

[0032] 4-1) Use the high-quality pseudo-label set generated by the teacher network to supervise the training of the student network.

[0033] 4-2) The self-training loss calculation consists of two parts: Focal Loss and Dice Loss. Focal Loss addresses the class imbalance problem and focuses on difficult-to-segment samples, as follows:

[0034]

[0035] Among them, is the mask value predicted by the student network, γ is the focusing parameter (default set to 2), Calculate the loss for all pixel points on the image, N p represents all the hint numbers, is the mask value predicted by the teacher network. In the present invention, the base of the log is e. The role of Dice Loss is to optimize the intersection over union (IoU) of the mask, as follows:

[0036]

[0037] Among them, ∈ is a small constant to prevent division by zero (default set to 1), represents the sum over all pixel points (h, w), is the mask value predicted by the student network, is the mask value predicted by the teacher network.

[0038] The total self-training loss is the sum of FocalLoss and Dice Loss, that is:

[0039]

[0040] 4-3) Use the output of the anchor network to constrain the predictions of the student network and the teacher network, preventing the model from deviating from the source domain knowledge:

[0041]

[0042] Among them, m s and m t are the predicted masks of the student network and the teacher network respectively, and λ tea and λ stu are weight coefficients, default set to 0.5.

[0043] 4-4) Extract features F a and F t from the image encoders of the anchor network and the teacher network, calculate the contrast loss between the anchor network and the teacher network, align the intermediate features of the anchor network and the teacher network, and enhance the feature consistency. The feature alignment is as follows:

[0044]

[0045] where τ is the temperature coefficient (set to 0.3 by default).

[0046] 4-5) The student network updates its weights under the guidance of high-quality pseudo labels and adopts a low-rank matrix update strategy. The encoder weights of the student network is decomposed into two low-rank matrices and θ=AB, r is the rank of the low-rank matrix, the default is 4. In the weight update process, only A and B are updated, the original weight θ original Fixed, specifically θ = θ original +A·B, optimizes A and B through back-propagation, reducing memory usage and computational overhead.

[0047] 4-6) The weights of the teacher network are updated through the EMA of the student network as follows:

[0048]

[0049] Among them, α is the smoothing coefficient, set to 0.999, Θ tea and Θ stu The weights of the teacher network and the student network are divided into the following: The teacher network indirectly inherits the low-rank matrix update of the student network through EMA.

[0050] 4-7) Repeat steps 3) and 4). In each iteration, new pseudo labels are generated and the student network and teacher network are updated. After multiple iterations, the performance of the student network is gradually improved. After training, the student network is used as a defect segmentation model for drone inspection tasks.

[0051] Step 5) The semantic segmentation model obtained in step 4) is lightweighted through full-stage distillation, and a hierarchical segmentation strategy is applied to reduce the computational complexity of SAM in the "segment everything" mode. The specific steps are as follows:

[0052] 5-1) Perform full-stage knowledge distillation on the model: knowledge transfer is achieved by aligning the features and outputs of the student network and the teacher network at multiple levels. The first step is image embedding alignment, as follows:

[0053]

[0054] Among them, E tea (I) and E stu (I) are the embedding outputs of the teacher and student networks for image I, respectively. ||·||1 represents the L1 distance, which is used to enforce feature similarity.

[0055] 5-2) Align the output features of the bidirectional Transformer in the mask decoder:

[0056]

[0057] Among them, and are the output of the Transformer module of the teacher and student networks.

[0058] 5-3) Align the segmentation mask outputs of the student and teacher using Dice Loss and Focal Loss:

[0059]

[0060] Step 6) Transfer the semantic segmentation model based on weakly supervised domain adaptation completed in step 5) to the unmanned inspection equipment for inspection. The unmanned inspection system collects the surface images of the dam concrete in real time and realizes online defect recognition; the unmanned inspection system includes:

[0061] 6-1) A data acquisition module, belonging to a camera device, capable of collecting and storing the dam images within the shooting range;

[0062] 6-2) An image recognition module, which receives the defect images taken by the unmanned inspection equipment in real time and uses the constructed weakly supervised domain adaptation semantic segmentation model to analyze the type information of the defect images in real time;

[0063] 6-3) A data transmission module, which transmits the image information and mask information processed by the image segmentation module;

[0064] 6-4) An inspection system, which receives the images taken by the unmanned inspection equipment and the defect mask information after image recognition, optimizes the inspection route according to the identified defect positions, ensures that the inspection equipment can efficiently cover all areas to be inspected, generates a detailed inspection task list, and sends the task to the control system;

[0065] 6-5) A control system, which receives the instructions sent by the inspection system and controls the unmanned inspection equipment to take pictures, recognize, and transmit the areas to be recognized.

[0066] A defect semantic segmentation device for unmanned inspection includes the following:

[0067] A module for constructing a special dataset for defect detection semantic segmentation, which obtains the defect images obtained during the dam inspection, collects different types of defect images on the concrete surface, resamples the defect images of the minority class, adjusts the data distribution, and solves the problem of class imbalance.

[0068] The data augmentation module performs weak data augmentation on the collected original defective image dataset, applies slight rotation, translation, scaling, etc., and inputs the processed images into the anchor network for feature extraction; performs weak data augmentation on the resampled dataset, applies slight rotation, translation, scaling, etc., and inputs the processed images into the teacher network for feature extraction; performs strong data augmentation on the resampled dataset, applies color jitter, noise, large rotation, etc., and inputs the processed images into the student network for feature extraction.

[0069] The self-training module inputs weak supervision (bounding boxes, point annotations, rough masks) into the prompt encoder to generate a fixed set of prompts. The teacher model generates pseudo-labels under the guidance of weak supervision, applies a pseudo-label filtering strategy to obtain high-quality pseudo-labels, calculates the self-training loss based on the high-quality pseudo-labels to guide the update of the student network; calculates the anchor loss and contrast loss to regularize the update and alignment of features of the student network; updates the parameters of the student network through low-rank matrix, and the weights of the teacher network synchronize the parameters of the student network through EMA, indirectly inheriting the update of the low-rank matrix.

[0070] The lightweight module performs knowledge distillation on the trained model in all stages, including the alignment of image embeddings, intermediate feature outputs, and final segmentation mask outputs.

[0071] The application device loads the trained weakly supervised domain adaptation semantic segmentation model. The application device collects the defective images on the surface of the dam concrete in real time, inputs them into the weakly supervised domain adaptation defective image semantic segmentation model, and realizes online defective image segmentation.

[0072] The application device is an unmanned inspection device that loads the trained weakly supervised domain adaptation semantic segmentation model. The inspection device collects defective images in real time and realizes online image segmentation.

[0073] The unmanned inspection system includes:

[0074] The data acquisition module: The camera mounted on the unmanned inspection device collects and temporarily stores the defective images of the dam within the shooting range.

[0075] The image recognition module: Receives the defective images taken in real time and uses the constructed weakly supervised domain adaptation semantic segmentation model to analyze the type information of the defective images in real time.

[0076] The data transmission module transmits the image information and mask information processed by the image segmentation module.

[0077] An inspection system that receives images captured by unmanned inspection devices and defect mask information after image recognition, optimizes the inspection route according to the identified defect locations to ensure that the inspection devices can efficiently cover all areas that need to be inspected, generates a detailed inspection task list, and sends the tasks to the control system;

[0078] A control system that receives instructions sent by the inspection system and controls the inspection devices to take pictures, recognize, and transmit in the areas that need to be recognized.

[0079] The implementation processes and methods of the device modules are the same.

[0080] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the defect semantic segmentation method for unmanned inspection as described above.

[0081] A computer-readable storage medium storing a computer program for executing the defect semantic segmentation method for unmanned inspection as described above.

[0082] Beneficial effects: Facing the diversity of defect types, existing datasets are often difficult to comprehensively cover. Traditional machine vision algorithms have limitations in establishing comprehensive models, and the reusability of the models is not high. The large semantic segmentation model provides new possibilities for solving challenges under variable working conditions with its excellent generalization ability. However, the large model training set is mainly composed of natural images with strong edge information, while industrial defect detection datasets often have unclear defect edge information. The present invention proposes a semantic segmentation method based on weakly supervised domain adaptation, which combines weakly supervised prompts and domain adaptation strategies to solve the domain shift problem caused by data with unclear defect edges and improve the defect detection accuracy in scenarios with unclear defect edges. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is the method flow chart of the embodiment of the present invention;

[0084] Figure 2 is the overall framework diagram of the weakly supervised domain adaptation of the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0085] The following further clarifies the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent modifications made by those skilled in the art to the present invention all fall within the scope defined by the appended claims of this application.

[0086] The environment used in the embodiments of the present invention: Based on the Pytorch deep learning framework, the model is trained and tested on a server with 1 NVIDIA A100 GPU, 1 Jetson AGX development board (16G video memory), and 1 Ubuntu operating system.

[0087] A defect semantic segmentation method for unmanned inspection robots specifically includes the following steps:

[0088] Step 1) Obtain the defect images collected during the dam inspection process, resample the defect data based on the training consistency principle, and adjust the data distribution;

[0089] 1-1) Collect different types of defect images on the surface of the dam concrete from the network and on-site shooting materials, and use weak supervision annotation (such as bounding boxes, point annotations, rough masks) to annotate the defect areas to generate preliminary label data;

[0090] 1-2) Transmit the defect images to the resampling module based on training consistency. For each category k, calculate its average score (ACS) through the output probability map p of the teacher model, and dynamically estimate the distribution of each category: t Calculate its average score (ACS) and dynamically estimate the distribution of each category:

[0091]

[0092] Among them, h·w is the resolution of the probability map, Represents the predicted probability of category k at position (i,j). At each time step t, use the exponential moving average (EMA) to update ACS:

[0093]

[0094] α is the smoothing coefficient (default set to 0.999) to ensure the stability of ACS.

[0095] 1-3) Determine the minority categories according to the online average category scores obtained in step 1-2). The categories with scores lower than the set value are identified as minority categories. For each category k, the sampling rate SR k Is determined through normalization operations to ensure a significant increase in the sampling rate of minority categories:

[0096]

[0097] K is the total number of all categories, j traverses all categories, Calculate the total scores of all categories.

[0098] 1-4) For each sample in the target domain Calculate its prediction consistency in multiple iterations, that is, the reliability of each category

[0099]

[0100] ξ(·) is the label generation function, IoU calculates the intersection over union of two predicted masks, and measures the prediction consistency of class k. is the mask generated by the teacher model at t-I. According to Sort all target domain images in descending order, and select the top C samples as the copy-paste candidate set to ensure the high reliability of the candidate samples.

[0101] 1 - 5) Copy and paste the images in the candidate set to complete resampling.

[0102] Step 2) Construct a three - network framework of the anchor network, teacher network, and student network, and perform data augmentation on the image data as follows:

[0103] 2 - 1) Initialize the anchor network, teacher network, and student network. All three networks use the pre - trained SAM image encoder as their basic architecture. Among them, the weights of the anchor network are frozen and do not change during training; in contrast, the student network is based on the initial weights of the anchor network and continuously adjusts and optimizes its own weights during training; the initial weights of the teacher network are the same as those of the student network, and its weight update strategy is synchronously adjusted according to the learning progress of the student network.

[0104] 2 - 2) Apply weak data augmentation i (Set small - angle rotation, translation, and scaling) to the original data x of the target domain, and input the processed image data into the anchor network to extract features f(·) is the encoder of the anchor network, and Θ is the parameter of the anchor network. a is the parameter of the anchor network.

[0105] 2 - 3) Apply weak data augmentation (Set small - angle rotation, translation, and scaling) to the data resampled in Step 1, and then input the processed image data into the teacher network to extract features f(·) is the encoder of the teacher network, and Θ t is the parameter of the teacher network.

[0106] 2 - 4) Apply strong data augmentation (Color jittering, adding noise, large - angle rotation (more than 90 degrees)) to the data resampled in Step 1, and then input it into the student network to extract features:

[0107]

[0108] f(·) is the encoder of the student network, and Θ s is the parameter of the student network.

[0109] Step 3) Introduce weak supervision as a prompt to generate a fixed set of prompts. The teacher model generates pseudo-labels based on the set of prompts and performs three-layer filtering on the pseudo-labels. First, filter the pseudo-labels using IoU, then filter out low-quality pseudo-labels using the stability score, and finally remove duplicate masks to obtain high-quality pseudo-labels, which specifically includes the following steps:

[0110] 3-1) Prepare the weak supervision prompts, where the weak supervision includes: bounding box: the minimum bounding rectangle of the target defect; point annotation: randomly sample positive points in the defect area and negative points outside the area; rough mask: fit the true mask through polygon.

[0111] 3-2) Construct a prompt encoder. The prompt encoder g(p; Ω) converts the weak supervision prompts into embedding vectors of a fixed dimension. Among them, for the bounding box, map the upper left and lower right coordinates of the bounding box to position embeddings; for point annotation, encode the coordinates of positive and negative points as position embeddings respectively; for the rough mask, use the sequence of vertex coordinates of the polygon as the position embedding.

[0112] 3-3) Input the weak supervision into the prompt encoder, convert it into an embedding vector, and generate a set of prompts {e j}.

[0113] 3-4) The teacher model receives the weakly augmented image x weak and the set of prompts {e j}, and generates a pseudo-label corresponding to j:

[0114]

[0115] f(x weak ; Θ tea ) is the output feature of the teacher network, h(·) is the mask decoder, and Φ represents the set of decoder parameters. Combine the image features and prompt embeddings to generate pseudo-labels.

[0116] 3-5) Calculate the IoU between each pseudo-label mask and the mask generated by the anchor network . If the IoU is greater than 0.5, then retain the pseudo-label

[0117]

[0118] 3-6) Calculate the prediction consistency of each pseudo-label mask under different weak augmentations. If the stability is greater than 0.8, then retain

[0119]

[0120] Among them, K is the number of weak augmentations, is the prediction mask after the k-th augmentation; is the prediction mask of the original image.

[0121] 3-7) Sort by stability score, retain the mask with the highest score, remove masks with an IoU exceeding 0.7 with it, and generate a high-quality pseudo-label set

[0122] Step 4) Use the high-quality pseudo-labels generated in Step 3) to guide the training of the student network, calculate the self-training loss, contrast loss, and anchor loss, and update the weights of the student network and the teacher network as follows:

[0123] 4-1) Use the high-quality pseudo-labels generated by the teacher network to supervise the training of the student network.

[0124] 4-2) Calculate the self-training loss, which includes two parts: Focal Loss and Dice Loss. Focal Loss solves the problem of class imbalance and focuses on difficult-to-segment samples as follows:

[0125]

[0126] Among them, is the mask value predicted by the student network, γ is the focusing parameter (default set to 2), calculate the loss for all pixel points on the image, N p represents all the hint numbers, is the mask value predicted by the teacher network. The role of Dice Loss is to optimize the intersection over union (IoU) of the mask as follows:

[0127]

[0128] Among them, ∈ is a small constant to prevent division by zero (default set to 1), means to sum over all pixel points (h, w), is the mask value predicted by the student network, is the mask value predicted by the teacher network.

[0129] The total self-training loss is the sum of FocalLoss and Dice Loss, that is:

[0130]

[0131] 4-3) Use the output of the anchor network to constrain the predictions of the student network and the teacher network, preventing the model from deviating from the source domain knowledge:

[0132]

[0133] Among them, m s and m t are the prediction masks of the student network and the teacher network respectively, and λ tea and λ stu are weight coefficients, which are default set to 0.5.

[0134] 4-4) Extract features F a and F t from the image encoders of the anchor network and the teacher network, calculate the contrast loss between the anchor network and the teacher network, align the intermediate features of the anchor network and the teacher network, and enhance the feature consistency. The feature alignment is as follows:

[0135]

[0136] Among them, τ is the temperature coefficient (default set to 0.3).

[0137] 4-5) The student network updates the weights under the guidance of high-quality pseudo-labels, adopts a low-rank matrix update strategy, and the encoder weights of the student network are decomposed into two low-rank matrices and θ = AB, where r is the rank of the low-rank matrix, which is default 4. During the weight update process, only A and B are updated, and the original weight θ original is fixed, specifically θ = θ original + A·B, and A and B are optimized through backpropagation to reduce memory occupancy and computational overhead.

[0138] 4-6) The weights of the teacher network are updated by the EMA of the student network, specifically as follows:

[0139] [[ID=·44]]

[0140] Among them, α is the smoothing coefficient, set to 0.999, Θ tea and Θ stu are the weights of the teacher network and the student network respectively. The teacher network indirectly inherits the low-rank matrix update of the student network through EMA.

[0141] 4-7) Repeat steps 3) and 4). In each iteration, new pseudo-labels are generated and the student network and the teacher network are updated. After multiple iterations, the performance of the student network is gradually improved. After training is completed, the student network is used as a defect segmentation model for UAV inspection tasks.

[0142] Step 5) Lightweight the semantic segmentation model obtained in step 4). The specific steps are as follows:

[0143] Knowledge transfer is achieved by aligning the features and outputs of the student network and the teacher network at multiple levels. First is the alignment of image embeddings: Align the output features of the bidirectional Transformer in the alignment mask decoder: Use Dice Loss and Focal Loss to align the segmentation mask outputs of the student and the teacher: The total distillation loss is the sum of the above three comparison losses:

[0144] Step 6) Transfer the trained weakly-supervised domain adaptation semantic segmentation model to the unmanned inspection device, and based on the images collected in real time by the unmanned inspection system, realize the recognition of the defective image area online, specifically:

[0145] Establish a control connection between the inspection system and the control system of the unmanned inspection device;

[0146] The control system of the inspection device receives the instructions from the inspection system to trigger the corresponding functions and control the movement of the inspection device;

[0147] During the movement of the inspection device, collect the images within the shooting range and transmit the collected images to the control system;

[0148] The control system inputs the collected images into the weakly-supervised domain adaptation semantic segmentation model, outputs the recognition results of the defective areas, and transmits the results and the original images to the inspection system;

[0149] The data processing model of the inspection system receives the segmentation result information, parses it and presents it on the user interface.

[0150] A defective semantic segmentation device for unmanned inspection includes the following:

[0151] Build a dedicated dataset module for defect detection semantic segmentation, obtain the defective images obtained during the dam inspection, collect different types of defective images on the concrete surface, resample the defective images of the minority classes, adjust the data distribution, and solve the problem of class imbalance.

[0152] Data augmentation module, perform weak data augmentation on the collected original defective image dataset, apply slight rotation, translation, scaling, etc., and input the processed images into the anchor network for feature extraction; perform weak data augmentation on the resampled dataset, apply slight rotation, translation, scaling, etc., and input the processed images into the teacher network for feature extraction; perform strong data augmentation on the resampled dataset, apply color jitter, noise, large rotation, etc., and input the processed images into the student network for feature extraction.

[0153] Self-training module: Input weak supervision (bounding boxes, point annotations, rough masks) into the prompt encoder to generate a fixed set of prompts. The teacher model generates pseudo-labels under the guidance of weak supervision, applies a pseudo-label filtering strategy to obtain high-quality pseudo-labels, calculates the self-training loss based on the high-quality pseudo-labels to guide the update of the student network; calculates the anchor loss and contrast loss to regularize the update and alignment features of the student network; updates the parameters of the student network through low-rank matrix, and the weights of the teacher network synchronize the parameters of the student network through EMA, indirectly inheriting the update of the low-rank matrix.

[0154] Lightweight module: Perform knowledge distillation on the trained model in all stages, including the alignment of image embeddings, intermediate feature outputs, and final segmentation mask outputs.

[0155] Unmanned inspection system: Load the trained weak supervision domain adaptation semantic segmentation model, and the inspection device collects defect images in real time to achieve online image segmentation.

[0156] Obviously, those skilled in the art should understand that each step of the above-mentioned defect semantic segmentation method for unmanned inspection in the embodiments of the present invention or each unit of the defect semantic segmentation device for unmanned inspection can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in the storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A defect semantic segmentation method for unmanned inspection robots, characterized in that, The method includes the following steps: Step 1) Obtain the defect images collected during the inspection of the dam, extract different types of defect samples on the concrete surface therefrom, construct an initial defect dataset, resample the defect data based on the training consistency principle, and adjust the data distribution; Step 2) Construct an anchor network, a teacher network, and a student network, perform strong data augmentation and weak data augmentation on the defect images respectively, input the augmented images into the anchor network, the teacher network, and the student network respectively, and extract the defect image features; Step 3) The teacher network generates initial pseudo-labels under the guidance of weak supervision signals, designs a filtering mechanism to screen the pseudo-labels, and obtains high-quality pseudo-labels for subsequent training; Step 4) Use the high-quality pseudo-labels obtained in Step 3) to guide the training of the student network, calculate the self-training loss, the anchor loss, and the contrast loss, optimize the weights of the student network encoder in combination with the low-rank matrix update strategy, and the teacher network synchronizes the parameters of the student network through the exponential moving average mechanism, and iteratively trains to obtain a defect image semantic segmentation model; Step 5) Lightweight the trained defect image semantic segmentation model through full-stage distillation; Step 6) Deploy the lightweight weak supervision domain adaptive semantic segmentation model to the unmanned inspection device, the monitoring device collects the dam surface images in real time, and inputs the image data into the weak supervision domain adaptive semantic segmentation model to realize online image segmentation and defect area recognition.

2. The defect semantic segmentation method for an unmanned inspection robot according to claim 1, wherein, The said Step 1) includes the following processes: 1-1) Collect defect images of different types on the dam concrete surface, label the defect areas using weak supervision annotation, and generate preliminary label data; 1-2) Transport the defect images to the resampling module based on training consistency. For each category, calculate its online average category score ACS through the output probability map of the teacher model, and dynamically estimate the distribution of each category; at each time step, update ACS using the exponential moving average EMA; 1-3) Determine the minority categories according to the obtained online average category scores. The categories with scores lower than the set value are identified as minority categories, and the sampling rate of each category is determined through normalization; 1-4) For each sample in the target domain, calculate the reliability of each category; sort all the target domain images in descending order according to the reliability of each category, and select the top C samples as the copy-paste candidate set; 1-5) Copy and paste the images in the candidate set to complete the resampling.

3. The defect semantic segmentation method for unmanned inspection robots according to claim 1, wherein The said Step 2) includes the following steps: 2-1) Initialize the anchor network, the teacher network, and the student network. All three networks use the pre-trained SAM image encoder as their basic architecture; among them, the weights of the anchor network are set to a fixed state; the initial weights of the student network are the same as those of the anchor network, and the weights of the student network are updated following the training; the initial weights of the teacher network are the same as those of the student network, and the weights of the teacher network are updated following the student network; 2-2) Perform weak data augmentation on the original data in the target domain; input the processed image data into the anchor network to extract features; 2-3) Perform weak data augmentation on the resampled data, and input the processed image data into the teacher network to extract features; 2-4) Apply strong data augmentation to the resampled data and then input it into the student network to extract features.

4. The defect semantic segmentation method for an unmanned inspection robot according to claim 1, wherein Step 3) includes the following steps: 3-1) Prepare weak supervision cues. The weak supervision includes: (1) Bounding box: the minimum bounding rectangle of the target defect; (2) Point annotation: randomly sample positive points in the defect area and negative points outside the area; (3) Coarse mask: fit the true mask through polygon fitting. 3-2) Construct a cue encoder that converts the weak supervision cues into embedding vectors of a fixed dimension. For the bounding box, map the upper left and lower right coordinates of the bounding box to position embeddings. For point annotation, the coordinates of positive and negative points are respectively encoded as position embeddings. For the coarse mask, use the sequence of vertex coordinates of the polygon as the position embedding. 3-3) Input the weak supervision into the cue encoder to convert it into embedding vectors and generate a cue set. 3-4) The teacher model receives the weakly augmented image and the cue set and generates pseudo-labels corresponding to the classes. 3-5) Calculate the IoU between each pseudo-label mask and the mask generated by the anchor network. If the IoU is greater than the first set threshold, retain the pseudo-label. 3-6) Calculate the prediction consistency of each pseudo-label mask under different weak augmentations. If the stability is greater than the second set threshold, retain the current pseudo-label mask. 3-7) Sort by the stability score, retain the mask with the highest score, and remove the masks whose IoU exceeds the third set threshold to generate a high-quality pseudo-label set.

5. The defect semantic segmentation method for unmanned inspection robots according to claim 1, characterized in that Step 4) uses the high-quality pseudo-labels generated in step 3) to guide the training of the student network and update the weights of the student network and the teacher network. Specifically, it includes the following steps: 4-1) Use the high-quality pseudo-label set generated by the teacher network to supervise the training of the student network. 4-2) Calculate the self-training loss, which includes two parts: FocalLoss and Dice Loss. FocalLoss is: wherein, is the mask value predicted by the student network, γ is the focusing parameter, calculates the loss of all pixel points on the image, N p represents all the number of prompts, is the mask value predicted by the teacher network, and the base of log is e; the role of Dice Loss is to optimize the intersection over union of the mask, as follows: where ∈ is a small constant to prevent division by zero, denotes the summation over all pixel points (h, w), is the mask value predicted by the student network, is the mask value predicted by the teacher network; The total self-training loss is the sum of Focal Loss and Dice Loss, that is: 4-3) Using the output of the anchor network Constraining the predictions of the student network and the teacher network to prevent the model from deviating from the source domain knowledge: where m s and m t are the prediction masks of the student network and the teacher network respectively, and λ tea and λ stu are weight coefficients; 4-4) Extract features F from the image encoders of the anchor network and the teacher network a and F t , calculate the contrastive loss between the anchor network and the teacher network, and align the intermediate features of the anchor network and the teacher network. The feature alignment is as follows: where τ is the temperature coefficient. (4 - 5) The student network updates the weights under the guidance of high-quality pseudo-labels and adopts a low-rank matrix update strategy. The encoder weights of the student network are decomposed into two low-rank matrices and θ = AB, where r is the rank of the low-rank matrix; during the weight update process, only A and B are updated, and the original weight θ original is fixed, specifically θ = θ original + A·B, and A and are optimized through backpropagation; 4-6) The weights of the teacher network are updated by the EMA of the student network, specifically as follows: where α is the smoothing coefficient, Θ tea and Θ stu are the weights of the teacher network and the student network respectively; the teacher network indirectly inherits the low-rank matrix update of the student network through EMA; 4-7) Repeat steps 3) and 4). In each iteration, generate new pseudo-labels and update the student network and the teacher network. Gradually improve the performance of the student network after multiple iterations. After training is completed, the student network is used as a defect segmentation model for the UAV inspection task.

6. The defect semantic segmentation method for unmanned inspection robots according to claim 1, characterized in that Step 5) lightweight the semantic segmentation model obtained in step 4) through full-stage distillation, apply a hierarchical segmentation strategy, and reduce the computational cost of SAM in the "segment everything" mode. The specific steps are as follows: 5-1) Perform full-stage knowledge distillation on the model: achieve knowledge transfer by aligning the features and outputs of the student network and the teacher network at multiple levels. First is the alignment of image embeddings, specifically as follows: Among them, E tea (I) and E stu (I) are the embedding outputs of the teacher and student networks for the image I respectively; ||·||1 represents the L1 distance, which is used to enforce feature similarity. 5-2) Align the output features of the bidirectional Transformer in the mask decoder. Among them, and are the outputs of the Transformer modules of the teacher and student networks; 5-3) Use Dice Loss and Focal Loss to align the segmentation mask outputs of the student and the teacher.

7. The defect semantic segmentation method for unmanned inspection robots according to claim 1, wherein, Step 6) Transfer the semantic segmentation model based on weakly supervised domain adaptation completed in step 5) to the unmanned inspection device for patrol inspection. The unmanned inspection system collects the images of the dam concrete surface in real time to achieve online defect recognition; The unmanned inspection system includes: 6-1) A data acquisition module, which belongs to a camera device and can collect and store the dam images within the shooting range; 6-2) An image recognition module, which receives the defect images taken by the unmanned inspection device in real time and uses the constructed weakly supervised domain adaptation semantic segmentation model to analyze the type information of the defect images in real time; 6-3) A data transmission module, which transmits the image information and mask information processed by the image segmentation module; 6-4) An inspection system, which receives the images taken by the unmanned inspection device and the defect mask information after image recognition, optimizes the inspection route according to the identified defect positions to ensure that the inspection device can efficiently cover all areas that need to be inspected, generates a detailed inspection task list, and sends the task to the control system; 6-5) A control system, which receives the instructions sent by the inspection system and controls the unmanned inspection device to take pictures, recognize, and transmit the areas that need to be identified.

8. A defect semantic segmentation device for unmanned inspection, characterized in that It includes the following modules: A module for constructing a special dataset for defect detection semantic segmentation, which obtains the defect images obtained during the dam inspection, collects different types of defect images on the concrete surface, resamples the defect images of the minority class, and adjusts the data distribution; A data augmentation module, which performs weak data augmentation on the collected original defect image dataset, applies slight transformations, and inputs the processed images into the anchor network for feature extraction; performs weak data augmentation on the resampled dataset, applies slight transformations, and inputs the processed images into the teacher network for feature extraction; Performs strong data augmentation on the resampled dataset and inputs the processed images into the student network for feature extraction; A self-training module, which generates a fixed prompt set by the weakly supervised input prompt encoder, the teacher model generates pseudo-labels under the guidance of weak supervision, applies a pseudo-label filtering strategy to obtain high-quality pseudo-labels, calculates the self-training loss based on the high-quality pseudo-labels to guide the update of the student network; calculates the anchor loss and contrast loss to standardize the update and alignment features of the student network; Updates the parameters of the student network through low-rank matrix, and the weights of the teacher network synchronize the parameters of the student network through EMA, indirectly inheriting the update of the low-rank matrix; A lightweight module, which performs knowledge distillation on the trained model in all stages, including the alignment of image embedding, intermediate feature output, and final segmentation mask output; An application device, which loads the trained weakly supervised domain adaptation semantic segmentation model, the application device collects the defect images of the dam concrete surface in real time, inputs the weakly supervised domain adaptation defect image semantic segmentation model, and realizes online defect image segmentation; The application device is an unmanned inspection device, which loads the trained weakly supervised domain adaptation semantic segmentation model, and the inspection device collects defect images in real time to realize online image segmentation; The unmanned inspection system includes: Data acquisition module: The camera mounted on the unmanned inspection device collects and temporarily stores the defect images of the dam within the shooting range. Image recognition module: Receives the defect images taken in real time and uses the constructed weakly supervised domain adaptation semantic segmentation model to analyze the type information of the defect images in real time. Data transmission module: Transmits the image information and mask information processed by the image segmentation module. Inspection system: Receives the images taken by the unmanned inspection device and the defect mask information after image recognition, optimizes the inspection route according to the identified defect positions to ensure that the inspection device can efficiently cover all areas that need to be inspected, generates a detailed inspection task list, and sends the tasks to the control system. Control system: Receives the instructions issued by the inspection system and controls the inspection device to take pictures, recognize, and transmit the areas that need to be identified.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the above computer program, it implements the defect semantic segmentation method for unmanned inspection as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the defect semantic segmentation method for unmanned inspection as described in any one of claims 1-7.

Images