Semi-supervised segmentation method for reference hole of robot drilling for aerospace structure

Abstract

The application belongs to the technical field of aviation manufacturing, and particularly relates to a datum hole semi-supervised segmentation method for robot drilling of an aviation structural part, training of a datum hole semi-supervised segmentation network based on robot drilling, design of a total loss function for training of the datum hole semi-supervised segmentation network, obtaining of a trained segmentation network by combining a training model of the datum hole semi-supervised segmentation network and the total loss function, and then robust and accurate segmentation of the datum hole based on the segmentation network to obtain an accurate datum hole feature region; the application can realize robust and accurate segmentation of the datum hole in a visual detection image during robot drilling of the aviation structural part, obtain the accurate datum hole feature region, and provide a direct and important basis for accurate detection and positioning of the datum hole during robot drilling of the aviation structural part.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aviation manufacturing technology, in particular to a reference hole semi-supervised segmentation method for robot drilling of aviation structural parts. BACKGROUND

[0002] Hole drilling of aviation structural parts is a key process link of aircraft assembly manufacturing. Considering the low cost and high flexibility of robots, robot drilling systems are widely used for hole drilling of aviation structural parts.

[0003] During robot drilling, due to the motion error of the robot moving platform, the absolute positioning error of the robot, the positioning error and deformation of the aviation structural part, there is a relative position error between the TCP origin of the robot drilling system end effector and the point to be processed, which leads to the generation of aviation structural part drilling position deviation.

[0004] In order to eliminate the relative position error between the robot drilling system TCP and the point to be processed, a certain number of reference holes are usually pre-processed on the aviation structural part, the actual position deviation of the reference holes is detected by the vision system integrated on the end effector, and then the linear interpolation compensation method is used to calculate and correct the position deviation of each point to be processed in the reference hole surrounding area based on the position deviation of each reference hole. In the case of determining the internal and external parameters of the vision system, the position deviation visual detection accuracy of the reference hole will directly affect the position correction result of the point to be processed and the drilling position accuracy, and the segmentation result of the reference hole in the image collected by the vision system directly affects the position deviation visual detection accuracy of the reference hole. Poor segmentation result of the reference hole may deteriorate the drilling accuracy and the safety of aviation part processing. SUMMARY

[0005] The purpose of the present application is to provide a reference hole semi-supervised segmentation method for robot drilling of aviation structural parts to solve the problems in the above background.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] The reference hole semi-supervised segmentation method for robot drilling of aviation structural parts is based on the training of the reference hole semi-supervised segmentation network for robot drilling. The total loss function of the reference hole semi-supervised segmentation network training is designed. By combining the training model of the reference hole semi-supervised segmentation network and the total loss function, the trained segmentation network is obtained, and then the robust and accurate segmentation of the reference hole is carried out based on the segmentation network to obtain the accurate reference hole feature region.

[0008] Further, the method comprises the following steps:

[0009] S1, the reference hole semi-supervised segmentation model is used to separate the reference hole from the background area of the visual detection image and label the image data set ;

[0010] S2, constructing a training model of a benchmark hole semi-supervised segmentation network based on S1, specifically comprising the following steps:

[0011] S21, using the labeled image dataset Training the segmentation network to obtain the loss value of the fully supervised part ;

[0012] S22, using the unlabeled image dataset Training the segmentation network;

[0013] S3, designing a perturbation consistency loss function combining Dice loss and CrossEntropy loss for training the benchmark hole semi-supervised segmentation network, wherein Dice loss and CrossEntropy loss are used to calculate the error between the predicted value and the true value or pseudo label, and the total loss function of the benchmark hole semi-supervised segmentation network training is

[0014] ;

[0015] ;

[0016] wherein:

[0017] are the weights of the loss values corresponding to the training of the labeled and unlabeled data, respectively;

[0018] are the loss values trained with labeled and unlabeled data, respectively;

[0019] , the weights of the loss values corresponding to the first image layer strong perturbation and the second feature layer strong perturbation; are the loss values corresponding to the first image layer strong perturbation and the second feature layer strong perturbation, respectively;

[0020]

[0021] S4, using the training model of the benchmark hole semi-supervised segmentation network constructed in S2, considering strong and weak consistency, based on multiple reliability, and taking U-Net as the segmentation network;

[0022] the total loss function of the benchmark hole semi-supervised segmentation network trained by S3 ​​​​, the training, testing and verification of the segmentation network are performed, and a trained segmentation network is obtained, and then the robust and accurate segmentation of the benchmark hole can be performed based on the segmentation network;

[0023] ;

[0024] Among them:

[0025] is a trained segmentation network function;

[0026] are the original benchmark hole image and the segmented benchmark hole feature region, respectively.

[0027] Further, S22 includes, for a single unlabeled image :

[0028] S221, a single unlabeled image is obtained by random weak perturbation , and the random weak perturbation method includes image cropping, image flipping, image scaling, etc.

[0029] S222, random strong perturbation is performed times on the image layer to obtain image , and the random strong perturbation method on the image layer includes noise, color enhancement, blur processing, shear fusion, multi-angle rotation, etc.

[0030] S223, random strong perturbation is performed times on the feature layer to obtain feature map , and the random strong perturbation method on the feature layer is random discarding in the feature space.

[0031] S224, the segmentation network is used to predict , and , to obtain , and ;

[0032] S225, to enhance the accuracy of the segmentation model, a weak disturbance branch pseudo label reliability enhancement module based on targeted post-processing is designed, after the model outputs the weak disturbance prediction image , the percentage of each prediction segmentation region in the entire image region is calculated, and when the segmentation region percentage is between 0.02%-0.5%, the segmentation region is retained, otherwise it is deleted.

[0033] S226, takes as and The pseudo-labels are used to train the segmentation network, and the loss values ​​of the two strongly perturbated images from the unlabeled data are obtained. and;

[0034] in:

[0035] A single unlabeled image was obtained using random weak perturbation;

[0036] In order to conduct A single unlabeled image is obtained by random strong perturbation at the sub-image level. ;

[0037] In order to conduct Random strong perturbations at the sub-feature level yield feature maps, where... ;

[0038] For image with random weak perturbation The predicted image;

[0039] for Sub-image level random strong perturbation image The predicted image;

[0040] for Sub-feature level random strong perturbation feature map The predicted image.

[0041] Furthermore, in step S3, to enhance the robustness of the segmentation model, a strong perturbation branch contribution adjustment strategy based on unreliable features of unlabeled data is adopted, specifically including the following steps:

[0042] S31. Calculate the predicted images for strong perturbations at the image level and strong perturbations at the feature level, respectively. and Entropy map composed of the information entropy of pixels and ;

[0043] S32. Set the corresponding dynamic threshold for information entropy. and Predicted images of statistical image-level strong perturbation branches and feature-level strong perturbation branches and Number of unreliable pixels and ;

[0044] ;

[0045] S33, based on the number of unreliable pixel points and , calculate the loss value weight corresponding to the first image layer strong disturbance and the first feature layer strong disturbance ;

[0046] ;

[0047] wherein:

[0048] respectively, the information entropy of any pixel point of the predicted image and ;

[0049] , respectively, the entropy map composed of the information entropy of the pixel points of the predicted image and ;

[0050] , respectively, the dynamic threshold of the information entropy of the pixel points of the predicted image and ;

[0051] , respectively, the number of unreliable pixel points of the predicted image and of the image layer strong disturbance branch and the feature layer strong disturbance branch.

[0052] By adopting the foregoing design scheme, the beneficial effects of the present application are: the present application can realize robust and accurate segmentation of the reference hole in the visual detection image during robot hole drilling of the aviation structural part, and accurate reference hole feature areas can be obtained, which can provide direct and important basis for accurate detection and positioning of the reference hole during robot hole drilling of the aviation structural part. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a training model schematic diagram of the reference hole semi-supervised segmentation network of the present application;

[0054] Figure 2 is a progressive training model schematic diagram of the connection hole damage area semi-supervised segmentation network of the present application. DETAILED DESCRIPTION

[0055] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0056] With reference to Figure 1 : A reference hole semi-supervised segmentation method for robot hole drilling of an aviation structure, based on training of a reference hole semi-supervised segmentation network for robot hole drilling, a total loss function for training of the reference hole semi-supervised segmentation network is designed, a trained segmentation network is obtained by combining the training model of the reference hole semi-supervised segmentation network and the total loss function, and then the segmentation network is used to perform robust and accurate segmentation of the reference hole to obtain an accurate reference hole feature region. The reference hole semi-supervised segmentation method aims to realize robust and accurate segmentation of the reference hole through a training mode of a small amount of labeled data and a large amount of unlabeled data, and to provide reliable feature region support for subsequent visual detection.

[0057] Specifically, the following steps are included:

[0058] S1, a reference hole semi-supervised segmentation model is used to segment the reference hole from the background region of a visual detection image, and an image data set is labeled . The labeled image data set is used as a sample to train the encoder and the decoder of the network through a supervised loss, and the goal of this stage is to let the network preliminarily learn the difference features of the reference hole and the background.

[0059] S2, a training model of a reference hole semi-supervised segmentation network with a U-Net segmentation network is constructed based on S1, and the following steps are specifically included:

[0060] S21, the labeled image data set is used to train the segmentation network to obtain a loss value of the fully supervised part .

[0061] S22, the unlabeled image data set is used to train the segmentation network.

[0062] The consistency information of the labeled data and the unlabeled data is used to gradually optimize the model parameters and improve the segmentation robustness of the reference hole under a complex background.

[0063] Further, to enhance the robustness of the segmentation model, an unreliable feature strong disturbance branch contribution degree adjustment strategy of unlabeled data is adopted, and the following steps are specifically included:

[0064] S221, a single unlabeled image is obtained by randomly weakly perturbing it The random weak perturbation mode includes image cropping, image flipping, image scaling, etc.

[0065] S222, random strong perturbation is performed times on the image layer, and an image is obtained The random strong perturbation mode on the image layer includes noise, color enhancement, blur processing, shear fusion, multi-angle rotation, etc.

[0066] S223, random strong perturbation is performed times on the feature layer, and a feature map is obtained The random strong perturbation mode on the feature layer is random discarding in the feature space.

[0067] S224, the segmentation network is used to predict , and , and , and are obtained.

[0068] S225, in order to enhance the accuracy of the segmentation model, a weak perturbation branch pseudo label based on targeted post-processing is designed The reliability enhancement module calculates the percentage of each predicted segmentation region in the entire image region after the model outputs the weak perturbation prediction image When the segmentation region percentage is between 0.02% and 0.5%, the segmentation region is retained, otherwise it is deleted.

[0069] S226, taking as the pseudo label of and , the segmentation network is trained, and the loss values of the two strong perturbation images of the unlabeled data are obtained and.

[0070] By performing multiple different random data augmentations on the same unlabeled image, the augmented multiple images are input into the network, and the network is required to output multiple segmentation masks as much as possible. The essential features of the benchmark hole will not change due to slight data augmentation. If the network can output consistent segmentation results for the augmented images, it means that it has learned the stable features of the benchmark hole, rather than background noise. At the same time, the segmentation mask generated by the supervised pre-trained network on the unlabeled image is used as the pseudo label, and the pseudo label pair is used as the new training sample to train the network together with the original labeled sample. The high-confidence pseudo label can be approximately regarded as a quasi-labeled data, which can expand the training sample size and avoid the error interference caused by low-confidence pseudo labels.

[0071] wherein:

[0072] randomly weakly perturbing the image to obtain a single unlabeled image;

[0073] for performing randomly strongly perturbing the image at the image level, to obtain a single unlabeled image, wherein,

[0074] for performing randomly strongly perturbing the feature map at the feature level, to obtain a feature map, wherein,

[0075] predicting the image for the randomly weakly perturbed image;

[0076] predicting the image for the randomly strongly perturbed image at the image level;

[0077] predicting the feature map for the randomly strongly perturbed feature map at the feature level. S3, designing a perturbation consistency loss function combining Dice loss and CrossEntropy loss, for training the benchmark hole semi-supervised segmentation network, wherein the error between the predicted value and the true value or pseudo label is calculated using Dice loss and CrossEntropy loss, and the total loss function of the benchmark hole semi-supervised segmentation network training

[0078]

[0079]

[0080]

[0081] Further, to enhance the robustness of the segmentation model, a strong perturbation branch contribution degree adjustment strategy for unreliable features of unlabeled data is adopted, which specifically includes the following steps:

[0082] S31, respectively calculating the information entropy of the pixel points of the predicted image and of the image level strong perturbation and the feature level strong perturbation and ;

[0083] S32, setting the corresponding dynamic threshold value of the information entropy and​​​​​​​ , the number of unreliable pixel points of the predicted image of the image-level strong disturbance branch and the feature-level strong disturbance branch and ; and ;

[0084] ;

[0085] S33, based on the number of unreliable pixel points and , the weight of the loss value corresponding to the first image-level strong disturbance and the first feature-level strong disturbance is calculated ; ;

[0086] ;

[0087] Wherein:

[0088] respectively, the information entropy of any pixel point of the predicted image and ;

[0089] , respectively, the entropy map composed of the information entropy of the pixel points of the predicted image and ;

[0090] , respectively, the dynamic threshold of the information entropy of the pixel points of the predicted image and ;

[0091] , respectively, the number of unreliable pixel points of the predicted image and of the image-level strong disturbance branch and the feature-level strong disturbance branch;

[0092] , respectively, the weight of the loss value corresponding to the first image-level strong disturbance and the first feature-level strong disturbance. S4, the training model of the benchmark hole semi-supervised segmentation network constructed by S2, considering strong and weak consistency, based on multi-reliability, using U-Net as segmentation network;

[0093] S4, the training model of the benchmark hole semi-supervised segmentation network constructed by S2, considering strong and weak consistency, based on multi-reliability, using U-Net as segmentation network;

[0094] ​​Total loss function for training the benchmark hole semi-supervised segmentation network using S3 , training, testing and verifying the segmentation network to obtain the trained segmentation network, and then the robust and accurate segmentation of the benchmark hole can be performed based on the segmentation network;

[0095] ;

[0096] Among them:

[0097] are the weights of the loss values corresponding to the labeled and unlabeled data training, respectively;

[0098] are the loss values corresponding to the first image layer strong disturbance and the first feature layer strong disturbance, respectively;

[0099] is the trained segmentation network function;

[0100] are the original benchmark hole image and the segmented benchmark hole feature region, respectively.

[0101] The final value of the benchmark hole semi-supervised segmentation method for robot hole machining of the aviation structure in this embodiment is to provide a high-reliability feature region for visual detection of the benchmark hole for robot hole machining of the aviation structure. The specific application scenarios include:

[0102] Benchmark hole center positioning: the segmented benchmark hole mask can be directly used for circle center detection algorithm, and since the mask has removed background interference, the circle center positioning accuracy can be improved by 15%-20%, avoiding the positioning deviation caused by background noise when positioning directly on the original image.

[0103] Benchmark hole diameter measurement: the contour of the benchmark hole can be accurately extracted through the segmentation mask, and the diameter of the contour can be calculated combined with the image pixel resolution, and the measurement error can be controlled within ±0.02mm, meeting the precision requirements of aviation hole machining.

[0104] Robot hole machining path correction: if the segmentation result shows that there is a deviation between the actual position of the benchmark hole and the theoretical position, the deviation information can be fed back to the robot control system to correct the path of subsequent hole machining in real time, ensuring that the hole machining position is consistent with the design drawing.

[0105] Compared with the traditional full-supervised segmentation method and the manual segmentation method, the benchmark hole semi-supervised segmentation method has the following significant advantages:

[0106] Low data dependency: only a small amount of labeled data is required, which greatly reduces the cost of manual labeling and solves the problem of scarcity of labeled data in aviation scenarios.

[0107] Strong generalization ability: By learning the common features of benchmark holes from unlabeled data, the system can adapt to different structural materials, different lighting conditions, and different background interference scenes, significantly improving the robustness of segmentation.

[0108] Real-time requirements are met: The optimized segmentation network inference speed can meet the real-time visual detection requirements during the robot drilling process.

[0109] Accuracy meets standards: In the aviation structural part visual image test set, the segmentation accuracy can reach 0.95 or above, and the benchmark hole miss detection rate is less than 0.5%, fully meeting the accuracy requirements of subsequent visual detection and robot drilling.

[0110] Reference Figure 2 The embodiment of the present application also provides a semi-supervised segmentation method for the damaged area of a connecting hole for drilling an aviation structural part. Based on the progressive training of the semi-supervised segmentation network for the damaged area of a connecting hole for robot drilling, a total loss function for the training of the semi-supervised segmentation network for the damaged area of a connecting hole is designed. By combining the progressive training model of the semi-supervised segmentation network for the damaged area of a connecting hole and the total loss function, a trained segmentation network is obtained, and then based on the segmentation network, a robust and accurate segmentation of the damaged area of a connecting hole is performed to obtain an accurate damaged area of a connecting hole. The semi-supervised segmentation method for the damaged area of a connecting hole aims to become a key requirement based on a small number of labeled samples and a large number of unlabeled samples. The progressive training of the semi-supervised segmentation network for the damaged area of a connecting hole for robot drilling and the design of the total loss function are two core pillars for implementing this method. The ultimate goal is to obtain a model with strong robustness and high segmentation accuracy through efficient training, accurately locate the damaged area of a connecting hole, and provide data support for subsequent damage assessment and repair.

[0111] Specifically includes the following steps:

[0112] SA1, a semi-supervised segmentation model for the damaged area of a connecting hole is used to segment the damaged area of a connecting hole from the background area of a visual detection image and label the image data set . Taking the labeled image data set as a sample, the encoder and decoder of the network are trained through a supervised loss. The goal of this stage is to let the network preliminarily learn the difference features of benchmark holes and backgrounds.

[0113] SA2, based on SA1, a progressive training model of a semi-supervised segmentation network for the damaged area of a connecting hole is constructed, which uses a lightweight TransUNet as a segmentation network and combines consistency regularization and pseudo labeling. Specifically includes the following steps:

[0114] SA21, the TransUNet is lightened by reducing the number of stacked Transformer modules;

[0115] SA22, defining the to-be-detected class of the segmentation model and labeling the image data , combining the unlabeled image data to establish a semi-supervised segmentation training data set;

[0116] SA23, training the connection hole damage area segmentation network using the labeled image data set and the unlabeled image data set . Only a small amount of labeled connection hole images are used, and the images need to be preprocessed to ensure the consistency of the input features. Through supervised learning, the model can preliminarily master the connection hole area positioning and damage type distinguishing ability. At this time, only the segmentation error of the model on the labeled samples is optimized, laying the foundation for subsequent use of unlabeled samples.

[0117] Further, the training of the connection hole damage area segmentation network in SA23 is as follows:

[0118] SA231, in the early stage of training, a full supervision method is used based on the labeled data set to perform full supervision training of the lightweight TransUNet segmentation network, so that the segmentation network has segmentation capability;

[0119] SA232, in the middle stage of training, the labeled data and the unlabeled data are used for training;

[0120] The labeled data training part calculates the loss value using the manually labeled label and the prediction result for the original image of the labeled data ;

[0121] The unlabeled data training part inputs the unlabeled image into the segmentation network after one weak disturbance and two strong disturbances, respectively, to obtain segmentation results , and ;

[0122] As a preferred embodiment of the present application, the strong disturbance method in the middle stage of training is a random combination of adding noise, color enhancement, blurring processing and shear fusion, mainly to gradually enhance the robustness of the model;

[0123] SA233, in the later stage of training, as the segmentation accuracy of the segmentation network gradually improves with the advancement of training, the accuracy of the pseudo label will also gradually improve, and the pixel class confidence predicted in is used as a judgment index, and a dynamic threshold is used as a judgment standard;

[0124] When the weak disturbance segmentation result is taken as a pseudo label , the model is trained using the output of the strong disturbance and , and the loss value of the unlabeled part is obtained and ;

[0125] The calculation formula of the dynamic threshold value is as follows:

[0126] ;

[0127] Among them:

[0128] is the loss value calculated using the artificial labeled label and the prediction result for the original image of the labeled data in the training part of the labeled data ;

[0129] is the current training round number

[0130] is the total number of network training rounds.

[0131] As a preferred embodiment of the present application, in the strong disturbance mode in the later training period, in addition to the random combination of adding noise, color enhancement, blur processing, shear fusion and random pixel interference to simulate light noise points. By introducing a large number of unlabeled connection hole images, a semi-supervised training set is constructed by mixing with labeled samples. In order to improve the generalization of the model, data enhancement is performed on all samples to simulate the features of the connection hole image under different detection angles and light conditions.

[0132] SA3, a progressive strong-weak consistency loss function is designed for the training of the connection hole damage area semi-supervised segmentation network, wherein the Dice and CrossEntropy loss are used to calculate the error between the predicted value and the true value or the pseudo label, and the total loss function of the base hole semi-supervised segmentation network training is

[0133] .

[0134] As a preferred embodiment of the present application, in SA3, for the training part based on the labeled data, the supervision item loss value is the loss between the predicted image obtained based on the prediction model and the artificial labeled true label ;

[0135] ​ ;

[0136] ;

[0137] For the training part of the unlabeled data, the supervision item loss value is the loss between the prediction image generated by the strong disturbance branch and the weak disturbance pseudo label The loss between the prediction image generated by the strong disturbance branch and the pseudo label and the loss between the prediction image generated by another strong disturbance branch and the pseudo label The weighted sum of ;

[0138] ;

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] Wherein:

[0144] is the serial number of the unlabeled data sample;

[0145] is the number of unlabeled data samples;

[0146] In the loss calculation process, , respectively represent the weights of the loss and ;

[0147] is the th unlabeled original image data sample used for disturbance branch 1;

[0148] is the th unlabeled original image data sample used for disturbance branch 2;

[0149] is an indicator function;

[0150] is the image level strong disturbance, including adding noise, color enhancement, blur processing and shear fusion;

[0151] For random pixel interference;

[0152] For the number of rounds between the middle and the end of training.

[0153] Predict the unlabeled samples using the pre-trained model to generate pseudo labels. To avoid unreliable pseudo labels interfering with training, pseudo label samples are screened by confidence to ensure the reliability of the labels of the input unlabeled samples; samples with a confidence lower than the threshold are not included in this round of training, and are predicted again after the model is optimized.

[0154] Input the labeled samples and screened pseudo label samples into the network for iterative training. After a certain number of iterations, use the current optimized model to generate new pseudo labels for the remaining unlabeled samples, and update the confidence screening threshold to include more high-reliability unlabeled samples. During this process, the model gradually transitions from relying on labeled samples to using unlabeled samples to supplement features, gradually improving its ability to recognize complex damage scenarios.

[0155] In the later stages of iteration, difficult-to-segment samples in labeled samples and high-confidence but large-segmentation-error samples in unlabeled samples are sampled for emphasis to strengthen the model's learning of complex scenarios.

[0156] SA4, a progressive training model of a connected hole damage semi-supervised segmentation network using SA2 with a lightweight TransUNet as a segmentation network, combined with consistency regularization and pseudo labeling;

[0157] Progressive strong and weak consistency loss function for training of connected hole damage semi-supervised segmentation network using SA3 , train, test and verify the segmentation network, obtain the trained segmentation network, and then perform robust and accurate segmentation of the connected hole damage region based on the segmentation network. When the segmentation accuracy of the model on the labeled validation set does not improve for several consecutive rounds, and the consistency of the pseudo labels of the unlabeled samples reaches a threshold, stop training. At this point, the model has fully learned the features of labeled and unlabeled samples and has strong robustness.

[0158] ;

[0159] wherein:

[0160] , are the labeled and unlabeled image datasets, respectively;

[0161] is the loss value of the supervision term;

[0162] are the weights of the loss values corresponding to the labeled and unlabeled data training;

[0163] is the current training round number;

[0164] is the round number node between the initial and intermediate training;

[0165] is the segmented network function after training;

[0166] are the original connection hole image and the segmented connection hole damage feature region, respectively.

[0167] After the training of the segmentation network, it can be directly applied to the connection hole damage detection scene of aviation structures. The specific steps are as follows:

[0168] Image input: pre-process the connection hole image to be detected, and output a standardized image.

[0169] Model inference: input the standardized image into the trained segmentation network. The model extracts multi-scale features through the encoder, and realizes feature mapping and pixel classification through the decoder, and outputs a pixel-level damage segmentation map.

[0170] Post-processing of results: remove small areas and smooth edges from the segmentation map to obtain the final damage area result, and output quantitative information such as the area and position coordinates of the damage area, providing data support for subsequent damage level assessment.

[0171] Compared with traditional full-supervised segmentation methods and manual segmentation methods, the connection hole damage region semi-supervised segmentation method has the following significant advantages:

[0172] High data efficiency: only a small amount of labeled samples are needed to train a high-performance model, greatly reducing the labeling cost of aviation connection hole damage samples and solving the industry pain point of few labels and much data.

[0173] Strong segmentation robustness: through progressive training and multi-constraint loss function, the model can adapt to different damage types and different detection environments, avoiding performance fluctuations in a single scene.

[0174] Accuracy meets industrial demand: in view of the high safety requirements of aviation structures, the model's segmentation IoU of the damage area can reach more than 0.85, and the recall rate of small-size damage can reach 0.9, which can accurately support subsequent damage repair and structure safety evaluation.

[0175] The reference hole semi-supervised segmentation in the embodiment is mainly applied to the positioning stage of robot hole making. By accurately identifying the reference hole, accurate position reference is provided for subsequent hole making operation, ensuring the accuracy and consistency of hole making. The accurate segmentation and positioning of the reference hole helps to reduce the processing error of the connecting hole, thereby reducing the possibility of damage to the connecting hole. The semi-supervised segmentation of the connecting hole damage area is mainly applied to the detection of the quality of the connecting hole after hole making, and the damage condition of the hole is evaluated to ensure the connecting strength and reliability of the aviation structure. The segmentation result of the connecting hole damage area can be fed back to the hole making process to provide a reference for optimizing the positioning and hole making parameters of the reference hole, forming a closed loop of quality control. For example, if the connecting hole damage area is concentrated in certain specific positions, it may be that the positioning of the reference hole is not accurate or the parameter setting in the hole making process is unreasonable. By adjusting the segmentation and positioning method of the reference hole or optimizing the hole making parameters, the quality of hole making can be improved. The semi-supervised segmentation of the reference hole and the semi-supervised segmentation of the connecting hole damage area jointly serve the whole process of hole making of the aviation structure, and guarantee the quality of hole making.

[0176] It should be noted that, in this document, the terms such as first and second are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Although embodiments of the present application have been shown and described, it is understood that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A semi-supervised segmentation method for reference holes in robotic hole making of aerospace structural components, characterized in that: Training a semi-supervised segmentation network for reference holes based on robotic hole making is conducted. A total loss function for training the semi-supervised segmentation network for reference holes is designed. By combining the training model of the semi-supervised segmentation network for reference holes with the total loss function, the trained segmentation network is obtained. Based on this segmentation network, robust and accurate segmentation of reference holes is performed, and accurate feature regions of reference holes are obtained. Specifically, the following steps are included: S1. A semi-supervised segmentation model using the reference aperture is used to segment the reference aperture from the background region of the visually detected image, and the image dataset is labeled. ; S2. Based on S1, construct a training model for a semi-supervised segmentation network with U-Net as the segmentation network, specifically including the following steps: S21. Utilizing labeled image datasets The segmentation network is trained to obtain the loss value of the fully supervised part. ; S22. Utilizing unlabeled image datasets Train the segmentation network; S221. Perform random weak perturbation on the image to obtain a single unlabeled image. Random weak perturbation methods include image cropping, image flipping, and image scaling; S222, proceed Random strong perturbations at the sub-image level yield the image. Image-level random perturbation methods include noise, color enhancement, blurring, cropping and fusion, and multi-angle rotation; S223, proceed Random strong perturbations at the sub-feature level yield feature maps. The random strong perturbation method at the feature level is to randomly discard features in the feature space; S224. Using a segmentation network to... , and Prediction is performed, and corresponding predicted images are obtained respectively. , and ; S225, Design of weakly perturbation branch pseudo-labels based on targeted post-processing The reliability enhancement module predicts images with weak perturbations in the model output. Then, calculate the percentage of each predicted segmentation region in the entire image region. If the percentage of the segmentation region is between 0.02% and 0.5%, the segmentation region is retained; otherwise, it is deleted. S226, will As and The pseudo-labels are used to train the segmentation network, and the loss values ​​of the two strongly perturbated images from the unlabeled data are obtained. sum; S3. Design a perturbation consistency loss function that combines Dice loss and CrossEntropy loss for training the benchmark aperture semi-supervised segmentation network. Dice loss and CrossEntropy loss are used to calculate the error between the predicted value and the true value or pseudo-label. The total loss function for training the benchmark aperture semi-supervised segmentation network is... for ； ； The strategy for adjusting the contribution of strong perturbations in unreliable features of unlabeled data includes the following steps: S31. Calculate the predicted images for strong perturbations at the image level and strong perturbations at the feature level, respectively. and Entropy map composed of the information entropy of pixels and ; S32. Set the corresponding dynamic threshold for information entropy. and Predicted images of statistical image-level strong perturbation branches and feature-level strong perturbation branches and Number of unreliable pixels and ; ； S33, Based on the number of unreliable pixels and Calculate the first Strong perturbations at the first image level and the first The weights of the loss values ​​corresponding to strong perturbations at each feature level ; ; ； S4. A training model of a semi-supervised segmentation network based on a reference aperture, constructed using S2 with U-Net as the segmentation network, considering strong and weak consistency, and based on multiple reliability. The total loss function of the benchmark aperture semi-supervised segmentation network trained using S3 is... The segmentation network is trained, tested, and validated to obtain the trained segmentation network, which can then be used for robust and accurate segmentation of reference holes. ； in: These are the weights corresponding to the loss values ​​trained on labeled and unlabeled data, respectively. These are the loss values ​​for training with labeled and unlabeled data, respectively. , No. Strong perturbations at the first image level and the first The weights of the loss values ​​corresponding to strong perturbations at each feature level; The first Strong perturbations at the first image level and the first The loss value corresponding to strong perturbations at each feature level; Predicted images and The information entropy of any pixel; This refers to the segmentation network function after training. These are the original reference hole image and the segmented reference hole feature region, respectively.

Images