Aero-engine semi-supervised semantic segmentation method for damage class imbalance

Abstract

The application discloses a semi-supervised semantic segmentation method for aero-engine damage category imbalance, which uses a student-teacher model with a category balancing module and a category-aware loss function, and the whole method is divided into two parts of source model and target model training, first, a labeled image is used to train the source model in a fully supervised manner, the model loss is calculated according to the prediction result of the network and the real label, the network weight is updated through back propagation, and the network weight is saved after the training is completed; then, the network weight of the source model is used to initialize the student and teacher models in the target model, and the input is an unlabeled image; after weak and strong data enhancement is performed on all the unlabeled images, the images are input into the teacher model and the student model for training; then, the pixel features obtained by the teacher model are input into the category balancing module for retraining to obtain category-aware weights; finally, the unsupervised consistency loss is calculated by using the category-aware weights, and the network weight is updated through back propagation.

Description

A semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories Technical Field

[0001] This invention relates to a semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories, belonging to the field of computer vision. Background Technology

[0002] With the rapid development of my country's aviation industry, ensuring the stable flight of aircraft has become a key concern. The aircraft engine is the core of an aircraft, and its core components typically operate in harsh environments with high temperature, high pressure, and high speed. Prolonged operation can lead to a decline in the performance of critical components. Therefore, timely observation of the engine's internal condition and assessment of its compliance with flight standards have become crucial tasks in aircraft maintenance and are essential guarantees for safe aircraft flight.

[0003] Currently, the most advanced detection technology is borescope testing, which is a commonly used technique for non-destructive testing of aero-engines. However, in practice, manual analysis in the later stages can lead to misdiagnosis or missed diagnosis, making it difficult to guarantee the diagnostic results.

[0004] In recent years, deep learning has achieved great success in semantic segmentation. Deep learning has fundamentally changed the way we handle some traditionally difficult or challenging visual tasks and has been successfully applied in many aspects of practical work with excellent results. However, commonly used fully supervised semantic segmentation models require a large amount of accurately labeled data, and pixel-by-pixel labeling is time-consuming and costly, taking more than an hour to label all pixels. Secondly, labeling aero-engine damage requires relevant professional knowledge, which undoubtedly requires more labeling time and places higher demands on the labelers. Finally, the background class in aero-engine crack damage datasets is much larger than the foreground class, leading to class imbalance. All of these factors are detrimental to the application and promotion of semantic segmentation methods in the field of aero-engine damage detection. Summary of the Invention

[0005] The technical challenges of this invention are: how to improve the efficiency and accuracy of aero-engine damage detection; at the same time, it is also necessary to solve the time-consuming and costly problem of manually annotating a large number of images in fully supervised semantic segmentation; to mine the potential information of unlabeled images to obtain new and additional supervision signals to improve the model's learning and performance; and finally, it is also necessary to alleviate the class imbalance problem in the aero-engine crack damage dataset.

[0006] This invention proposes a semi-supervised semantic segmentation method for damage identification in aero-engines.

[0007] The design concept of this method is as follows: the entire method is divided into a source model and a target model;

[0008] The input image in the source model is a labeled image; the fully supervised loss of the labeled image is calculated by using the labels of the labeled image as supervision. The network then performs gradient descent; updates the network weights based on the gradient descent results; and finally saves the trained network weights. ;

[0009] Then, use the network weights saved in the source model. Initialize the student and teacher networks in the target model; the input graph in the target model is an unlabeled image. The consistency loss between the teacher network predictions and the student network predictions is calculated by using the predictions generated by the teacher network as supervision for the student network. ;

[0010] Next, in each round of training, the pixel feature values ​​are... Input into the auxiliary model for retraining; use pixel predictions As a supervisory function, calculate the total supervisory loss. And its gradient descent; based on the gradient descent results, the network weights of the auxiliary model are updated to obtain more accurate pixel prediction values. Using pixel prediction values Calculate the category-aware weights ;

[0011] Finally, weights based on category awareness. And the loss of consistency between teacher networks and student networks Calculate the total unsupervised consistency loss. The gradient descent method is used to update the weights of the student network based on the gradient descent results, and the exponential moving average method is used to update the weights of the teacher network.

[0012] This invention can process labeled and unlabeled images simultaneously, thereby reducing the need for manual annotation. By deeply mining the hidden information in unlabeled images, additional supervision signals are obtained, which improves the performance of the model. In addition, by adding a class-aware weight to the loss function, the class imbalance problem in the dataset is effectively alleviated, which also improves the performance of the model.

[0013] Specifically, the present invention provides a semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories. This method uses a fully supervised trained source model and a target model with a student-teacher network having a category balancing module and a category-aware loss function to detect aero-engine damage images; the student and teacher models use the network weights obtained by training the source model.

[0014] The detection method includes the following steps:

[0015] Step 1: Process the image dataset of crack damage in aero-engines and initialize the weights of the source model network;

[0016] Step 2: Train the source model: The input is a labeled image. First, perform strong data augmentation on the image to obtain the strongly augmented image in the source model. ; Enhance images with strong data The input is fed into the source model for training;

[0017] In the source model's network, the fully supervised loss for labeled images is calculated by using the labels of the labeled images as supervision. The network then performs gradient descent; updates the network weights based on the gradient descent results; and finally saves the trained network weights. ;

[0018] Step 3: Train the target model: The input is an unlabeled image. First, use the network weights saved in the previous step. The student and teacher networks are initialized; then, weak data augmentation is performed on all unlabeled images to obtain the weakly augmented images in the target model. Then, perform strong data augmentation on all unlabeled images to obtain the strongly augmented images in the target model. Enhance images from weak data. and strong data augmentation images The data is input into both the teacher network and the student network for training.

[0019] In both the student and teacher networks, the consistency loss between teacher network predictions and student network predictions is calculated by using the predictions generated by the teacher network as supervision for the student network. ;

[0020] Step 4: For the class balancing module: First, the pixel feature values ​​of the teacher network in each round of training are... All are stored in the feature bank, pixel predicted values The data is stored in the prediction bank, and the feature bank and prediction bank are updated during each round of model training.

[0021] Next, the pixel feature values Input into the auxiliary model for retraining; use pixel predictions As a supervisory function, calculate the total supervisory loss. And its gradient descent; based on the gradient descent results, the network weights of the auxiliary model are updated to obtain more accurate pixel prediction values. ;

[0022] After the auxiliary model is trained, pixel prediction values ​​are used. Calculate the category-aware weights ;

[0023] Step 5: Calculate the category-aware weights using the category balancing module. And the loss of consistency between teacher networks and student networks Calculate the total unsupervised consistency loss. and its gradient descent;

[0024] The weights of the student network are updated based on the gradient descent results, and the weights of the teacher network are updated using the exponential moving average method. Finally, the performance is validated on the validation set.

[0025] Specifically, step one includes the following steps:

[0026] Step 1.1: Obtain image dataset of crack damage in aero-engines. First, a small portion of the damaged images were manually annotated at the pixel level to obtain labeled images; then the dataset was divided into labeled image sets. and unlabeled image sets Next, the labeled image sets will be... and unlabeled image sets Both sets are divided into training and testing sets in an 8:2 ratio; ultimately, the entire image dataset of aero-engine crack damage consists of a labeled image set containing M image-label pairs. and an unlabeled image set containing N images constitute;

[0027] Step 1.2: Construct the MobileNet architecture; the network in the source model adopts the MobileNet architecture;

[0028] Step 1.3: Initialize the network using the pre-trained MobileNet network weights to obtain the network in the source model. , These are the initial weights of the network.

[0029] Specifically, step two includes the following steps:

[0030] Step 2.1: For labeled images , and , Representing images and Indicates a label;

[0031] For images and tags Perform the same strong enhancement, the strong enhancement method is as follows:

[0032] For images and tags The same random scale cropping is performed; this method first randomly selects the image scale based on a certain probability, and then performs corresponding scaling operations on the image. Next, random cropping is used to further process the image to ensure that the image size matches specific requirements.

[0033] The cropped image and tags The same random rotation is performed. The method first generates a random rotation angle within a specified range of 90 degrees to 360 degrees. Then, under a given probability threshold, it is determined whether to rotate the image. If the condition is met, the image and label are rotated using the specified rotation angle.

[0034] For the rotated image and tags The same random flipping is performed. First, the method generates a random probability value to determine whether to perform a horizontal flip. If the probability condition is met, the image and label are flipped horizontally. Then, another random probability value is generated to determine whether to perform an vertical flip. If the probability condition is met, the image and label are flipped vertically.

[0035] For the flipped image and tags The same elastic transformation is performed; this method applies minute nonlinear transformations to the image based on set parameters, including random stretching, compression, and distortion.

[0036] Image after elastic transformation and tags Add the same random salt and pepper noise; this method increases the diversity of data by adding random salt and pepper noise to the image; specifically, the method randomly adds white (representing salt) and black (representing pepper) noise to the pixel values ​​of the image to simulate the noise interference that images may suffer in the real world;

[0037] Images with added random salt and pepper noise and tags Perform the same random brightness adjustment; in this method, a random number is first generated as a seed, and if the random number is greater than 0.5, then the brightness is adjusted.

[0038] Image after random brightness adjustment and tags The same random erasure process is performed; this method randomly erases a rectangular region in the image. First, it checks if a random number is greater than 0.5. If not, it directly returns the original sample; otherwise, it performs the erasure operation, ultimately obtaining the strongly enhanced image from the source model. and its corresponding tags ;

[0039] Step 2.2: Enhance the image Input to the network in the source model Training to obtain predictions ;

[0040] Step 2.3: Using prediction and tags Calculate the fully supervised loss for labeled images. And its gradient descent, and update the network weights based on the gradient descent results, where It is a standard binary cross-entropy loss function;

[0041] Step 2.4: After training is complete, save the final network weights of the source model. ;

[0042] Specifically: Step three includes the following steps:

[0043] Step 3.1: Use the final network weights from the source model. The student network and teacher network of the target model are initialized separately to obtain two networks with the same structure and weights. and , This refers to the initial weights of the teacher network and the student network;

[0044] Step 3.2, for unlabeled images , and Since unlabeled images contain only images and no labels, only the images are processed. Weak data augmentation is performed, and the methods for weak data augmentation are as follows:

[0045] For all images Adjust the size to 512x512 to ensure all images have the same dimensions;

[0046] For the adjusted image The same random flipping process is performed. First, the method generates a random probability value to determine whether to perform a horizontal flip. If the probability condition is met, the image and labels are flipped horizontally. Next, another random probability value is generated to determine whether to perform a vertical flip. If the probability condition is met, the image and labels are flipped vertically, ultimately resulting in the weakly augmented image in the target model. ;

[0047] Step 3.3: For unlabeled images , and Since unlabeled images contain only images and no labels, only the images are processed. Strong data augmentation is performed, and the methods for strong data augmentation are as follows:

[0048] For all images Adjust the size to 512x512, ensuring all images have the same dimensions;

[0049] The adjusted image Add random salt and pepper noise; this method increases the diversity of data by adding random salt and pepper noise to the image; specifically, the method randomly adds white (representing salt) and black (representing pepper) noise to the pixel values ​​of the image to simulate the noise interference that images may suffer in the real world;

[0050] Images with added random salt and pepper noise Perform random brightness adjustment; in this method, a random number is first generated as a seed, and if the random number is greater than 0.5, then brightness adjustment is performed;

[0051] Image after random brightness adjustment This method performs random erasure; its purpose is to randomly erase a rectangular region in the image. First, it checks if a random number is greater than 0.5. If it is not greater than 0.5, it directly returns the original sample; otherwise, it performs the erasure operation, ultimately obtaining the strongly augmented image of the target model. ;

[0052] Step 3.4: Enhance the image with strong data. Input to student network Training to obtain predictions ;

[0053] Step 3.5: Enhance the image with weak data. Input to teacher network Training to obtain pseudo-labels And save the weights of the teacher network in each epoch. ;

[0054] Step 3.6: Use prediction and pseudo-tags Calculate the consistency loss between teacher network predictions and student network predictions. ;

[0055] Specifically, step four includes the following steps:

[0056] Step 4.1: After each round of training of the target model, the pixel feature values ​​of the teacher network are... All are stored in the feature bank, pixel predicted values Stored in the prediction bank for use in training auxiliary models;

[0057] Step 4.2: Use the weights of the teacher network saved during this training round. The auxiliary model is initialized to obtain an auxiliary model. , These are the initial weights of the auxiliary model;

[0058] Step 4.3: Convert pixel feature values Input to auxiliary model The pixel prediction values ​​of the auxiliary model are obtained through training. ;

[0059] Step 4.4: Use the pixel prediction values ​​from the auxiliary model and pixel prediction values ​​in the prediction library Calculate the loss under full supervision and its gradient descent; update the network weights of the auxiliary model based on the gradient descent results, where It is a standard binary cross-entropy loss function;

[0060] Step 4.5, Final Auxiliary Model After training, use pixel prediction values Calculate the category-aware weights ;

[0061] Specifically, step five includes the following steps:

[0062] Step 5.1: Use the category-aware weights calculated by the auxiliary model. Consistency loss between teacher network predictions and student network predictions Calculate the total unsupervised consistency loss. ;

[0063] Step 5.2: Calculate the total unsupervised consistency loss. Gradient descent is used to update the weights of the student network based on the gradient descent results, and the corresponding teacher network weights are updated using the exponential moving average method. The update formula for the teacher network weights is as follows:

[0064]

[0065] in and These are the weights of the teacher network and the student network, respectively. It is the smoothing coefficient hyperparameter. The value range is 0.99 to 0.999. Determines the student's network weight The contribution to the current exponential moving average is relatively small. This will cause the exponential moving average to place more emphasis on the weight of the teacher network itself, while a larger... This will cause the exponential moving average to place more emphasis on the weight of student networks;

[0066] Step 5.3: Verify the performance of the teacher network on the validation set, save the weights of the best-performing teacher network, and repeat steps two through five until training ends.

[0067] The above-mentioned loss is calculated as follows:

[0068] Total loss The calculation formula is:

[0069]

[0070] in It is a standard binary cross-entropy loss function;

[0071] Category-aware weights The calculation formula is:

[0072]

[0073] in It is the number of pixels in each category. It is the number of pixels for the background category. It is the hyperparameter of the exponential term, set according to previous experiments. .

[0074] Compared with the prior art, the present invention, employing the above technical solution, has the following advantages:

[0075] 1. By simultaneously utilizing labeled and unlabeled images, the potential of unlabeled images can be explored in depth with only a small number of labeled images, thereby obtaining additional supervision signals to improve the performance of semantic segmentation models;

[0076] 2. Apply deep learning technology to the field of aero-engine damage detection, and use semantic segmentation networks to assist or replace traditional manual detection methods, aiming to improve the efficiency and accuracy of aero-engine damage detection;

[0077] 3. By adding class-aware weights to the loss function, the class imbalance problem in the aero-engine damage dataset is effectively alleviated, thereby improving the model's performance. Attached Figure Description

[0078] Figure 1 is a flowchart of the framework of the source model of this invention.

[0079] Figure 2 is a flowchart of the framework of the target model of the present invention. Detailed Implementation

[0080] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0081] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0082] Under the above conditions, the steps of the semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories are shown in Figures 1 and 2. Specifically, it includes the following steps:

[0083] Step 1: Process the image dataset of crack damage in aero-engines and initialize the weights of the source model network;

[0084] Step 1.1: Obtain image dataset of crack damage in aero-engines. ,

[0085] First, a small portion of the damaged images were manually annotated at the pixel level to obtain labeled images;

[0086] Then the dataset is divided into labeled image sets. and unlabeled image sets ;

[0087] Next, the labeled image sets will be... and unlabeled image sets Both sets are divided into training and testing sets in an 8:2 ratio; ultimately, the entire image dataset of aero-engine crack damage consists of a labeled image set containing M image-label pairs. and an unlabeled image set containing N images constitute;

[0088] Step 1.2: Construct the MobileNet architecture; the network in the source model adopts the MobileNet architecture;

[0089] Step 1.3: Initialize the network using the pre-trained MobileNet network weights to obtain the network in the source model. , These are the initial weights of the network.

[0090] Step 2: Train the source model: The input is a labeled image. First, perform strong data augmentation on the image to obtain the strongly augmented image in the source model. ; Enhance images with strong data The input is fed into the source model for training;

[0091] In the source model's network, the fully supervised loss for labeled images is calculated by using the labels of the labeled images as supervision. The network then performs gradient descent; updates the network weights based on the gradient descent results; and finally saves the trained network weights. ;

[0092] Step 2.1: For labeled images , and , Representing images and Indicates a label;

[0093] For images and tags Perform the same strong enhancement, the strong enhancement method is as follows:

[0094] For images and tags The same random scale cropping is performed; this method first randomly selects the image scale based on a certain probability, and then performs corresponding scaling operations on the image. Next, random cropping is used to further process the image to ensure that the image size matches specific requirements.

[0095] The cropped image and tags The same random rotation is performed. The method first generates a random rotation angle within a specified range of 90 degrees to 360 degrees. Then, under a given probability threshold, it is determined whether to rotate the image. If the condition is met, the image and label are rotated using the specified rotation angle.

[0096] For the rotated image and tags The same random flipping is performed. First, the method generates a random probability value to determine whether to perform a horizontal flip. If the probability condition is met, the image and label are flipped horizontally. Then, another random probability value is generated to determine whether to perform an vertical flip. If the probability condition is met, the image and label are flipped vertically.

[0097] For the flipped image and tags The same elastic transformation is performed; this method applies minute nonlinear transformations to the image based on set parameters, including random stretching, compression, and distortion.

[0098] Image after elastic transformation and tags Add the same random salt and pepper noise; this method increases the diversity of data by adding random salt and pepper noise to the image; specifically, the method randomly adds white (representing salt) and black (representing pepper) noise to the pixel values ​​of the image to simulate the noise interference that images may suffer in the real world;

[0099] Images with added random salt and pepper noise and tags Perform the same random brightness adjustment; in this method, a random number is first generated as a seed, and if the random number is greater than 0.5, then the brightness is adjusted.

[0100] Image after random brightness adjustment and tags The same random erasure process is performed; this method randomly erases a rectangular region in the image. First, it checks if a random number is greater than 0.5. If not, it directly returns the original sample; otherwise, it performs the erasure operation, ultimately obtaining the strongly enhanced image from the source model. and its corresponding tags ;

[0101] Step 2.2: Enhance the image Input to the network in the source model Training to obtain predictions ;

[0102] Step 2.3: Using prediction and tags Calculate the fully supervised loss for labeled images. And its gradient descent, and update the network weights based on the gradient descent results, where It is a standard binary cross-entropy loss function;

[0103] Step 2.4: After training is complete, save the final network weights of the source model. ;

[0104] Step 3: Train the target model: The input is an unlabeled image. First, use the network weights saved in the previous step. The student and teacher networks are initialized; then, weak data augmentation is performed on all unlabeled images to obtain the weakly augmented images in the target model. Then, perform strong data augmentation on all unlabeled images to obtain the strongly augmented images in the target model. Enhance images from weak data. and strong data augmentation images The data is input into both the teacher network and the student network for training.

[0105] In both the student and teacher networks, the consistency loss between teacher network predictions and student network predictions is calculated by using the predictions generated by the teacher network as supervision for the student network. ;

[0106] Step 3.1: Use the final network weights from the source model. The student network and teacher network of the target model are initialized separately to obtain two networks with the same structure and weights. and , This refers to the initial weights of the teacher network and the student network;

[0107] Step 3.2, for unlabeled images , and Since unlabeled images contain only images and no labels, only the images are processed. Weak data augmentation is performed, and the methods for weak data augmentation are as follows:

[0108] For all images Adjust the size to 512x512 to ensure all images have the same dimensions;

[0109] For the adjusted image The same random flipping process is performed. First, the method generates a random probability value to determine whether to perform a horizontal flip. If the probability condition is met, the image and labels are flipped horizontally. Next, another random probability value is generated to determine whether to perform a vertical flip. If the probability condition is met, the image and labels are flipped vertically, ultimately resulting in the weakly augmented image in the target model. ;

[0110] Step 3.3: For unlabeled images , and Since unlabeled images contain only images and no labels, only the images are processed. Strong data augmentation is performed, and the methods for strong data augmentation are as follows:

[0111] For all images Adjust the size to 512x512, ensuring all images have the same dimensions;

[0112] The adjusted image Add random salt and pepper noise; this method increases the diversity of data by adding random salt and pepper noise to the image; specifically, the method randomly adds white (representing salt) and black (representing pepper) noise to the pixel values ​​of the image to simulate the noise interference that images may suffer in the real world;

[0113] Images with added random salt and pepper noise Perform random brightness adjustment; in this method, a random number is first generated as a seed, and if the random number is greater than 0.5, then brightness adjustment is performed;

[0114] Image after random brightness adjustment This method performs random erasure; its purpose is to randomly erase a rectangular region in the image. First, it checks if a random number is greater than 0.5. If it is not greater than 0.5, it directly returns the original sample; otherwise, it performs the erasure operation, ultimately obtaining the strongly augmented image of the target model. ;

[0115] Step 3.4: Enhance the image with strong data. Input to student network Training to obtain predictions ;

[0116] Step 3.5: Enhance the image with weak data. Input to teacher network Training to obtain pseudo-labels And save the weights of the teacher network in each epoch. ;

[0117] Step 3.6: Use prediction and pseudo-tags Calculate the consistency loss between teacher network predictions and student network predictions. ,in It is a standard binary cross-entropy loss function;

[0118] Step 4: For the class balancing module: First, the pixel feature values ​​of the teacher network in each round of training are... All are stored in the feature bank, pixel predicted values The data is stored in the prediction bank, and the feature bank and prediction bank are updated during each round of model training.

[0119] Next, the pixel feature values Input into the auxiliary model for retraining; use pixel predictions As a supervisory function, calculate the total supervisory loss. And its gradient descent; based on the gradient descent results, the network weights of the auxiliary model are updated to obtain more accurate pixel prediction values. ;

[0120] After the auxiliary model is trained, pixel prediction values ​​are used. Calculate the category-aware weights ;

[0121] Step 4.1: After each round of training of the target model, the pixel feature values ​​of the teacher network are... All are stored in the feature bank, pixel predicted values Stored in the prediction bank for use in training auxiliary models;

[0122] Step 4.2: Use the weights of the teacher network saved during this training round. The auxiliary model is initialized to obtain an auxiliary model. , These are the initial weights of the auxiliary model;

[0123] Step 4.3: Convert pixel feature values Input to auxiliary model The pixel prediction values ​​of the auxiliary model are obtained through training. ;

[0124] Step 4.4: Use the pixel prediction values ​​from the auxiliary model and pixel prediction values ​​in the prediction library Calculate the loss under full supervision and its gradient descent; update the network weights of the auxiliary model based on the gradient descent results, where It is a standard binary cross-entropy loss function;

[0125] Step 4.5, Final Auxiliary Model After training, use pixel prediction values Calculate the category-aware weights The calculation formula is:

[0126]

[0127] in It is the number of pixels in each category. It is the number of pixels for the background category. It is the hyperparameter of the exponential term, set according to previous experiments. ;

[0128] Step 5: Calculate the category-aware weights using the category balancing module. And the loss of consistency between teacher networks and student networks Calculate the total unsupervised consistency loss. and its gradient descent;

[0129] The weights of the student network are updated based on the gradient descent results, and the weights of the teacher network are updated using the exponential moving average of the student network weights. Finally, the performance is validated on the validation set.

[0130] Step 5.1: Use the category-aware weights calculated by the auxiliary model. Consistency loss between teacher network predictions and student network predictions Calculate the total unsupervised consistency loss. The calculation formula is:

[0131]

[0132] in It is a standard binary cross-entropy loss function. These are the category-aware weights calculated earlier;

[0133] Step 5.2: Calculate the total unsupervised consistency loss. Gradient descent is used to update the weights of the student network based on the gradient descent results, and the corresponding teacher network weights are updated using the exponential moving average method. The update formula for the teacher network weights is as follows:

[0134]

[0135] in and These are the weights of the teacher network and the student network, respectively. It is the smoothing coefficient hyperparameter. The value range is 0.99 to 0.999. Determines the student's network weight The contribution to the current exponential moving average is relatively small. This will cause the exponential moving average to place more emphasis on the weight of the teacher network itself, while a larger... This will cause the exponential moving average to place more emphasis on the weight of student networks;

[0136] Step 5.3: Verify the performance of the teacher network on the validation set, save the weights of the best-performing teacher network, and repeat steps two through five until training ends.

[0137] In the application scenario of this invention, taking aero-engine damage detection as an example, after obtaining an aero-engine crack damage image dataset, the dataset is first divided into labeled image sets by manually annotating a portion of the images at the pixel level. and unlabeled image sets The entire method consists of a source model and a target model. The input image in the source model is a labeled image. By using the labels of the labeled images as supervision, the fully supervised loss of the labeled images is calculated. The network then performs gradient descent; updates the network weights based on the gradient descent results; and finally saves the trained network weights. Use the network weights saved in the source model. Initialize the student and teacher networks in the target model. The input graph in the target model is an unlabeled image. By using the predictions generated by the teacher network as supervision for the student network, calculate the consistency loss between the teacher network predictions and the student network predictions. In each round of training, pixel feature values ​​are... Input into the auxiliary model for retraining; use pixel predictions As a supervisory function, calculate the total supervisory loss. And its gradient descent; update the network weights based on the gradient descent results to obtain more accurate pixel prediction values. Using pixel prediction values Calculate the category-aware weights Weights based on category And the loss of consistency between teacher networks and student networks Calculate the total unsupervised consistency loss. The system employs gradient descent to update the weights of the student network based on the gradient descent results, and then updates the weights of the teacher network using the exponential moving average of the student network weights. Performance validation is then performed on the validation set. Finally, the above steps are repeated until training is complete.

[0138] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories, characterized in that: Damage images of aero-engines are detected using a source model and a target model; the source model is a fully supervised trained deep learning network. The target model is a student-teacher network with a class balancing module and a class-aware loss function; the target model uses the network weights obtained by training the source model. Finally, a student-teacher network is deployed to achieve semi-supervised semantic segmentation of aero-engines with imbalanced damage categories. The detection method includes the following steps: Step 1: Process the image dataset of aero-engine crack damage, perform pixel-level manual annotation on a portion of the damage images to obtain labeled images, and initialize the network weights of the source model; Step 2: Train the source model: The input is labeled images. First, perform strong data augmentation on the labeled images to obtain the strongly augmented images in the source model. Then, the strong data is used to augment the image. The input is fed into the source model for training; within the source model's network, the fully supervised loss of the labeled images is calculated by using the labels of the labeled images as supervision. The process involves gradient descent; updating the network weights based on the gradient descent results; and finally, saving the network weights of the trained source model. Step 3: Training the target model: The input is an unlabeled image; First, use network weights. The student and teacher networks are initialized; then, weak data augmentation is performed on all unlabeled images to obtain the weakly augmented images in the target model. Strong data augmentation is performed on all unlabeled images to obtain the strongly augmented images in the target model. Then, the weak data augmentation image will be obtained. and strong data augmentation images The predictions are fed into the teacher network and student network respectively for training. Within the teacher and student networks, the predictions generated by the teacher network are used as supervision for the student network, and the consistency loss between the teacher network predictions and the student network predictions is calculated. Step 4: For the class balancing module: First, the pixel feature values ​​of the teacher network in each round of training are... All are stored in the feature bank, pixel predicted values The feature bank and prediction bank are stored in the prediction bank. Each training iteration of the target model updates both the feature bank and the prediction bank. Then, the pixel feature values ​​are... The input is fed into an auxiliary model for retraining; in the auxiliary model, pixel predictions are used. As a supervisory function, calculate the total supervisory loss. And its gradient descent; based on the gradient descent results, update the network weights of the auxiliary model to obtain more accurate pixel predictions from the auxiliary model. After the auxiliary model is trained, pixel prediction values ​​are used. Calculate the category-aware weights Category-aware weights The calculation formula is: ;in, It is the number of pixels in each category. It is the number of pixels for the background category. These are exponential hyperparameters; p and q represent the damage category in the image, p=0 indicates the background category, p= These correspond to different damage categories; C is the total number of damage categories. and This represents the proportion of pixels belonging to the corresponding damage category to the total number of pixels; Step 5: Weighting by category. and consistency loss Calculate the total unsupervised consistency loss. The algorithm employs gradient descent; then updates the weights of the student network based on the gradient descent results; next, it uses exponential moving average to update the weights of the teacher network; finally, it performs performance validation on the validation set; the total loss... The calculation formula is: 。 2. The semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories according to claim 1, characterized in that: The steps in step one include: Step 1.1, acquiring an image dataset of crack damage in aero-engines. First, a small subset of crack damage images were manually annotated at the pixel level to obtain labeled images; then, the image dataset was... Divided into labeled image sets and unlabeled image sets Next, the labeled image sets will be... and unlabeled image sets The dataset is divided into training and testing sets; ultimately, the entire image dataset of aero-engine crack damage consists of a labeled image set containing M image-label pairs. and an unlabeled image set containing N images Composition; Step 1.2: Construct the MobileNet architecture; The network in the source model adopts the MobileNet architecture; Step 1.3: Initialize the network using the pre-trained MobileNet network weights to obtain the network in the source model. ， These are the initial weights of the network.

3. The semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories according to claim 2, characterized in that: The steps in step two include: Step 2.1, for labeled images ， and ， Representing images and Represents labels; for images and tags Perform the same strong data augmentation, the steps of which include: 2.1.1) For the image and tags Perform the same random scale cropping: First, randomly select the image scale size according to a certain probability; then perform corresponding scaling operations on the image; next, further process the image using random cropping to ensure that the image size matches the requirements; 2.1.2) For the cropped image and tags Perform the same random rotation: First, generate a random rotation angle within a specified range of 90 to 360 degrees; then, under a given probability threshold, determine whether to rotate the image. If the probability threshold is met, rotate the image and label using the specified rotation angle; 2.1.3) For the rotated image... and tags Perform the same random flipping: First, generate a random probability value to determine whether to perform a horizontal flip. If the probability value condition is met, then perform a horizontal flip operation on the image and label. Next, generate another random probability value to determine whether to perform an vertical flip. If the probability value condition is met, then perform a vertical flip operation on the image and label. 2.1.4) For the flipped image... and tags Perform the same elastic transformation: apply a small nonlinear transformation to the image according to the set parameters. The nonlinear transformation includes random stretching, compression, and / or distortion; 2.1.5) apply the elastic transformation to the image. and tags Adding the same random salt-and-pepper noise increases data diversity by adding random salt-and-pepper noise to the image. Specifically, white noise representing salt and black noise representing pepper are randomly added to the pixel values ​​of the image to simulate noise interference that images may suffer in the real world; 2.1.6) For images with added random salt-and-pepper noise and tags Perform the same random brightness adjustment: First, generate a random number as a seed. If the random number is greater than 0.5, then perform brightness adjustment; 2.1.7) Apply the same random brightness adjustment to the image. and tags The same random erasure process is performed, which randomly erases a rectangular region in the image: First, it checks if a random number is greater than 0.

5. If it is not greater than 0.5, the original sample is returned directly; otherwise, the erasure operation is performed. The final result is the strongly enhanced image from the source model. and its corresponding tags Step 2.2: Enhance the image Input to the network in the source model Training to obtain predictions Step 2.3: Use prediction and tags Calculate labeled images Full supervision loss The gradient descent is performed, and the network weights are updated based on the gradient descent results; where It is a standard binary cross-entropy loss function; Step 2.4: After training, save the final network weights of the source model. 。 4. The semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories according to claim 2, characterized in that: The steps in step three include: Step 3.1, using the final network weights of the source model. The student network and teacher network of the target model are initialized separately, resulting in two networks with identical structures and weights. and ， This involves the initial weights of the teacher and student networks; Step 3.2: For unlabeled images... ， and Since unlabeled images contain only images and no labels, only the images are processed. Weak data augmentation is performed. The weak data augmentation method is as follows: 3.2.1) For all images... Adjust the size to 512x512, ensuring all images have the same dimensions; 3.2.2) For the images adjusted in step 3.2.1), The same random flipping process is performed: First, a random probability value is generated to determine whether to perform a horizontal flip. If the probability condition is met, the image and labels are flipped horizontally. Next, another random probability value is generated to determine whether to perform a vertical flip. If the probability condition is met, the image and labels are flipped vertically. This ultimately yields the weakly augmented image in the target model. Step 3.3: For unlabeled images ， and Since unlabeled images contain only images and no labels, only the images are processed. Perform strong data augmentation. The strong data augmentation method is as follows: 3.3.1) Apply strong data augmentation to all images. Adjust the size to 512x512, ensuring all images have the same dimensions; 3.3.2) Adjust the images... Adding random salt-and-pepper noise increases data diversity by adding random salt-and-pepper noise to the image; specifically, white noise representing salt and black noise representing pepper are randomly added to the pixel values ​​of the image to simulate noise interference that images may suffer in the real world; 3.3.3) For images with added random salt-and-pepper noise Perform random brightness adjustment: First, generate a random number as a seed. If the random number is greater than 0.5, then perform brightness adjustment; 3.3.4) Apply random brightness adjustment to the image. Random erasure is performed to randomly erase a rectangular region in the image. First, a random number is checked to see if it is greater than 0.

5. If it is not greater than 0.5, the original sample is returned directly; otherwise, the erasure operation is performed, ultimately resulting in a strongly data-augmented image for the target model. Step 3.4: Enhance the image with strong data. Input to student network Training to obtain predictions Step 3.5: Enhance the image with weak data. Input to teacher network Training to obtain pseudo-labels And save the weights of the teacher network in each epoch. Step 3.6: Use prediction and pseudo-tags Calculate the consistency loss between teacher network predictions and student network predictions. 。 5. The semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories according to claim 1, characterized in that: The steps in step four include: Step 4.1, after each round of training of the target model, the pixel feature values ​​of the teacher network are... All are stored in the feature bank, pixel predicted values Store the weights in the prediction bank for use in training the auxiliary model; Step 4.2: Use the weights of the teacher network saved in this round of training. The auxiliary model is initialized to obtain an auxiliary model. ， These are the initial weights for the auxiliary model; Step 4.3: Set the pixel feature values... Input to auxiliary model The pixel prediction values ​​of the auxiliary model are obtained through training. Step 4.4: Use the pixel prediction values ​​from the auxiliary model. and pixel prediction values ​​in the prediction library Calculate the loss under full supervision and its gradient descent; update the network weights of the auxiliary model based on the gradient descent results, where It is a standard binary cross-entropy loss function; Step 4.5, final auxiliary model After training, use pixel prediction values Calculate the category-aware weights 。 6. The semi-supervised semantic segmentation method for aero-engines with imbalanced damage categories according to claim 1, characterized in that: Step five includes: Step 5.1, using the category-aware weights calculated by the auxiliary model. Consistency loss between teacher network predictions and student network predictions Calculate the total unsupervised consistency loss. Step 5.2: Calculate the total unsupervised consistency loss. Gradient descent is used to update the weights of the student network based on the gradient descent results, and the corresponding teacher network weights are updated using the exponential moving average method. The update formula for the teacher network weights is as follows: ;in and These are the weights of the teacher network and the student network, respectively. It is the smoothing coefficient hyperparameter. The value range is 0.99 to 0.

999. Determines the student's network weight The contribution to the current exponential moving average is relatively small. This will cause the exponential moving average to place more emphasis on the weights of the teacher network itself, while a larger... This will make the exponential moving average focus more on the weights of the student network; Step 5.3: Validate the performance of the teacher network on the validation set, save the weights of the best performing teacher network, and repeat steps two through five until training ends.

7. The semi-supervised semantic segmentation method for aero-engines oriented towards damage category imbalance according to any one of claims 1 to 6, characterized in that: Using a fully supervised trained source model and a target model with a student-teacher network having a class-balanced module and a class-aware loss function, the total loss is... This includes a consistency loss and a category-aware weight, which are the consistency losses for teacher network predictions and student network predictions, respectively. and category-aware weights ； Set as 。

Images