A Hole Exploration Damage Counting Method Based on Multi-Task Learning and Background Decoupling

By employing multi-task learning and background decoupling methods, combined with damage and background feature similarity determination, the problem of duplicate counting in borehole damage counting of aero-engines was solved, achieving high-precision damage video counting.

CN118365902BActive Publication Date: 2026-11-17NANJING TECH UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410388302.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-29
Publication Date
2026-11-17
Estimated Expiration
2044-03-29

AI Technical Summary

Technical Problem

In existing technologies for borehole damage counting of aero-engines, the same damage may be counted repeatedly due to differences in shape, size, and location in different scenarios. This makes it difficult to accurately identify and track damage, especially when switching between complex scenarios, which affects flight safety and reliability.

Method used

A multi-task learning approach is adopted, which uses the YOLOv5 network to detect damage features and combines it with a pre-trained CNN model to extract damage features. The background decoupling module (BDM) and background module (BGM) are combined to extract background features. The similarity between damage and background is calculated, and a joint determination is made on whether to count the damage.

Benefits of technology

It improves the accuracy and robustness of damage counting, solves the problem of repeated counting of the same damage in complex scene switching, and achieves high-precision damage video counting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118365902B_ABST
    Figure CN118365902B_ABST
Patent Text Reader

Abstract

This invention discloses a hole-detection damage counting method based on multi-task learning and background decoupling. The first task branch of this method detects damage in the current video frame using YOLOv5, extracts damage features using a pre-trained CNN model, and calculates the similarity to damage features in the previous frame. The second task branch extracts background features from the original feature map of YOLOv5 using a background decoupling module (BDM) and calculates the similarity to background features in the previous frame. A background generation module (BGM) is further introduced, using a pre-trained FCN segmentation model to generate background region labels, and calculating the Manhattan distance between the labels and the background region images obtained from the BGM. Finally, the damage feature similarity obtained from multi-task learning is combined with the background feature similarity to calculate a comprehensive similarity, which is then compared with a preset threshold. This method improves the accuracy and robustness of damage counting, achieving high-precision damage counting for damaged videos.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a borehole damage counting method based on multi-task learning and background decoupling (especially applicable to aero-engines).

[0002] Through multi-task learning, the feature similarity between the background and foreground of the same damage in different video frames was jointly determined, which solved the problem of repeated counting of the same damage during the switching of complex borehole inspection scenes, and achieved high-precision damage counting of aero-engine borehole inspection damage videos. Background Technology

[0003] Damage count is a key indicator for assessing the internal health of aero engines. Through damage counting, inspectors can not only continuously track and monitor engine damage and promptly identify trends and patterns in damage changes, but it is also crucial for discovering potential faults and problems, taking preventative measures, and ultimately improving flight safety and reliability. Furthermore, damage counting technology provides a more scientific and effective basis for the maintenance and repair of aero engines. However, in practical applications, due to the complexity of the internal environment of the engine and the frequent switching of scenes captured by borescope cameras, the shape, size, and location of the same damage can vary in different scenarios, leading to instances where the same damage is counted repeatedly.

[0004] Currently, damage counting techniques rely solely on the external features of damage, using simple damage feature similarity metrics to determine if they belong to the same individual, thus enabling counting. However, this method fails to adequately address the varying characteristics of damage under different environments. Furthermore, it is susceptible to interference from factors such as lighting and viewing angle when dealing with complex scenes requiring frequent switching, making damage identification and counting difficult.

[0005] In the prior art, patent application CN115147385A, entitled "An Intelligent Detection and Judgment Method for Repeated Damage in Airborne Borehole Video," includes the following steps: First, target detection is performed on internal damage, and the position of the target detection box and the depth features of each detection box image block in the current frame are read; then, the detection boxes are filtered according to confidence; maximum suppression is performed on the detection boxes to eliminate the phenomenon of multiple detection boxes for the same damage; the position of the target in the current frame is predicted using a Kalman filter tracker; then, the trajectory information and the cost matrix of the detection information are calculated based on the Mahalanobis distance of the appearance information, and cascaded matching and IOU matching are performed successively; finally, the parameters and feature set of the Kalman filter tracker are updated, and the disappearance of the target and the appearance of a new target are judged simultaneously.

[0006] This technology, which focuses solely on the appearance of damage, offers the ability to track dynamic targets. However, it may face challenges in identification and tracking when dealing with scenarios where the target's appearance or background changes significantly across different video frames. This is especially true in complex and frequently changing background conditions, such as borehole inspection of aero-engines, where it may lead to inaccurate damage counting. Summary of the Invention

[0007] To address the challenges mentioned in the background section, this invention proposes a damage counting method based on multi-task learning and background decoupling. This method is particularly suitable for borehole damage counting in aero-engines and similar applications.

[0008] This method processes damage features and background features of images simultaneously through two task branches. It determines whether to perform a count by combining the "damage feature similarity" and "background feature similarity" obtained through multi-task learning. Through multi-task learning, this method fully considers the differences in shape, size, and position of the same damage in different video frames across both the foreground and background. Compared to current technologies, this method not only relies on damage appearance similarity but also better adapts to changes in the damage background of the same damage in different shooting scenarios. This significantly improves the accuracy and robustness of damage counting, effectively solving the problem of repeated counting of the same damage caused by complex scene transitions. It provides an efficient and reliable solution for damage counting in aero-engines and other borehole drilling fields.

[0009] The method of this invention uses multi-task learning to jointly determine the feature similarity between the background and foreground of the same damage in different video frames, which greatly improves the accuracy and robustness of damage counting. It focuses on solving the problem of repeated counting of the same damage during the switching of complex scenes in borehole exploration, and realizes high-precision damage counting of damaged videos.

[0010] The idea behind this method is as follows:

[0011] This algorithm processes damage features and background features simultaneously through two task branches, and determines whether to count by combining the "damage feature similarity" and "background feature similarity" obtained from multi-task learning.

[0012] In the first task branch of this algorithm, damage in the current video frame is detected by the YOLOv5 network, and "damage features" are extracted by combining the pre-trained CNN model and the "damage feature similarity" with the previous frame is calculated.

[0013] The second task branch of this method extracts "background features" from the original feature map of YOLOv5 through the background decoupling module (BDM) and calculates the "background feature similarity" with the previous frame. To better extract background features, a background generation module (BGM) is further introduced during the training phase. This module uses a pre-trained FCN segmentation model to generate background region labels and calculates the Manhattan distance between the labels and the background region images obtained by BGM.

[0014] Finally, the combined similarity between the "damage feature similarity" obtained from multi-task learning and the "background feature similarity" is calculated and compared with a preset threshold. If the similarity is greater than the threshold, the damage is considered the same and is not counted; otherwise, the damage is considered different and is counted.

[0015] The specific technical solution of the present invention is as follows:

[0016] Step 1: First, collect and label the images and video data of borehole damage of aero-engines. Then, divide the data into training, validation and test sets in an 8:1:1 ratio to form a borehole damage dataset. Set the training parameters: initial learning rate, impulse size, input image size, batch size, number of iterations, etc. Finally, train the YOLOv5 network and obtain the optimal detection model file through parameter tuning.

[0017] Step 2: Perform multi-task learning from both damage features and background features;

[0018] Step 2.1: Obtain the damage target prediction box of the current frame through the YOLOv5 damage detection network, crop the corresponding damage region of the current video frame according to the prediction box parameters, and feed the cropped region into the pre-trained CNN model to obtain the "damage feature" vector.

[0019] Step 2.2: Extract the original feature map from the first convolutional layer of the YOLOv5 damage detection network head, and obtain the "background feature" vector through the background decoupling module (BDM).

[0020] Step 3: Calculate the cosine similarity between the "damage feature" vector and the "background feature" vector between the current frame and the previous frame, respectively, to obtain the damage feature similarity Sim. damage and background feature similarity Sim background ;

[0021] Step 4: Select appropriate weights based on the specific requirements of the task scenario and calculate the comprehensive similarity Sim. composite ;

[0022] Step 5: Calculate the similarity score and compare it with a preset threshold. If the score is greater than the threshold, it is determined to be the same lesion and is not counted; otherwise, it is determined to be different lesions and is counted.

[0023] Furthermore, the CNN model in step 2.1 uses a Wide Residual Network, consisting of two convolutional layers and six residual blocks. A global feature map of dimension 128 is computed in the fully connected layers, and finally, batch normalization and L2 regularization are used to project the features onto a unit hypersphere. This CNN network model has been pre-trained on the large-scale dataset of aero-engine borehole damage described in step 1.

[0024] Further, the implementation steps of the BDM module in step 2.2 include: passing the original feature map sequentially through four dilated convolutions with kernel sizes of 3×3 and dilation rates of 1, 2, 4, and 8, respectively. During this process, shallow feature information is progressively fused to deeper layers, and finally, the feature maps from each stage are concatenated into an output feature map. Next, a spatial encoder is used, whose overall structure consists of three parallel branches, each corresponding to max pooling, regular convolution, and upsampling operations with pooling coefficients of 1, 2, and 3, respectively. The purpose of this module is to extract background features more comprehensively and effectively.

[0025] Furthermore, in step 2.2, to enable the multi-task network to better extract background features through the BDM module during the prediction phase, a background generation module (BGM) is introduced during the training phase of the second task branch. This module generates a "background region image" for training. The BGM module consists of a ResNet-based feature encoder, reflection padding, instance normalization, and transposed convolution. Its purpose is to reconstruct an image containing only the background from the feature map generated by the BDM module.

[0026] The "background region labels" used for training are obtained by masking the borehole damage dataset using the FCN segmentation model. This process includes: calculating the position information of the damaged region and its bounding box through the mask; averaging the pixels of the non-damaged region within the bounding box; and replacing the pixels of the damaged region with this average value to generate the "background region labels".

[0027] During the training phase, the loss is measured by calculating the Manhattan distance between the pixels of the “background region image” and the “background region label”.

[0028] Further, in step 3, the cosine similarity between the "damage feature" vector and the "background feature" vector of the current nth frame and the corresponding feature vectors of frames 1 to (n-1) is calculated, and the maximum cosine similarity is taken as Sim. damage and Sim background .

[0029] Furthermore, in step 4, the comprehensive similarity Sim...composite Based on damage feature similarity Sim damage Sim similarity with background features background It is composed of a weighted combination. Among them, the contribution of damage feature similarity and background feature similarity to the comprehensive similarity is adjusted by the weight parameter.

[0030] Furthermore, in step 5, the sigmoid function is used to synthesize the similarity Sim. composite Mapping to the (0,1) interval yields a similarity score, which facilitates interpretation and comparison.

[0031] The beneficial effects of this invention are as follows:

[0032] This technical solution focuses on a damage counting method based on multi-task learning and background decoupling. It jointly determines the feature similarity between the background and foreground of the same damage in different video frames, which greatly improves the accuracy and robustness of damage counting. It also solves the problem of repeated counting of the same damage during the switching of complex scenes in borehole exploration, and achieves high-precision damage counting of damaged videos.

[0033] The technical reasons for achieving the above results in this technical solution are as follows:

[0034] 1) By jointly judging the damage feature similarity and the background feature similarity, the differences in shape, size, position and other aspects of the same damage in different video frames are fully considered, which better adapts to the changes in damage in different borehole shooting scenarios.

[0035] 2) By using the background decoupling module BDM and the background generation module BGM, the model's ability to extract background features is enhanced, thereby improving the accuracy of damage counting. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of the damage counting algorithm based on multi-task learning and background decoupling in this method.

[0038] Figure 2 This is a schematic diagram of a CNN-based wide residual network structure in an embodiment of this method.

[0039] Figure 3 This is a schematic diagram of the structure of the BDM module in this embodiment of the method.

[0040] Figure 4 This is a schematic diagram of the structure of the BGM module in this embodiment of the method.

[0041] Figure 5 This is a diagram showing the damage similarity discrimination effect in the counting algorithm based on multi-task learning and background decoupling of this method. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments:

[0043] The present invention provides a hole drilling damage counting method based on multi-task learning and background decoupling. This method improves the accuracy and robustness of damage counting by jointly determining the feature similarity between the background and foreground of the same damage in different video frames. It focuses on solving the problem of repeated counting of the same damage during the switching of complex scene in hole drilling, and realizes high-precision damage counting of damaged videos.

[0044] The specific embodiments of the present invention are as follows:

[0045] like Figure 1 As shown, a borehole damage counting method based on multi-task learning and background decoupling includes the following steps:

[0046] Step 1: First, collect and label the image and video data of borehole damage. Then, divide the data into training, validation, and test sets in an 8:1:1 ratio to form the borehole damage dataset. Set the initial training parameters: initial learning rate of 0.01, impulse size of 0.937, input image size of 1024×1024, batch size of 32, and number of iterations of 200. Finally, train the YOLOv5 network and obtain the optimal detection model file through parameter tuning.

[0047] Step 2: Perform multi-task learning from both damage features and background features;

[0048] Specifically, step 2 includes the following sub-steps:

[0049] Step 2.1: Use the trained YOLOv5 network to detect and locate damaged targets in the current video frame, thereby obtaining the predicted bounding boxes of the damaged targets. Based on the parameters of the predicted bounding boxes, crop the corresponding damaged regions in the original video frame. Finally, feed the cropped damaged regions into the pre-trained CNN model to obtain the "damage feature" vector.

[0050] By cropping the target region, the CNN model can focus more on the features of the damaged area and reduce the interference of background noise. Simultaneously, processing smaller cropped regions reduces computational burden and improves processing speed, especially when processing high-resolution videos. The reason for choosing to feed the cropped region into a pre-trained CNN model to obtain damage feature vectors, rather than directly extracting damage feature vectors from YOLOv5, is that these two models each have their strengths in damage recognition tasks. YOLOv5 excels at quickly and accurately locating objects in images, while CNNs (especially wide residual networks) focus more on extracting and analyzing the deep features of these located regions. This division of labor allows the entire system to handle damage detection and feature extraction tasks more accurately. By combining the advantages of both, the overall model's performance in damage counting can be improved.

[0051] The CNN model described uses a wide residual network. Please refer to the network structure diagram. Figure 2 The network comprises two convolutional layers and six residual blocks, and computes a 128-dimensional global feature map in the fully connected layers. To improve the model's performance in subsequent tasks, we project these features onto a unit hypersphere using batch normalization and L2 regularization. This ensures that each feature vector has a uniform unit length, so that the cosine similarity calculation reflects only the directional differences between vectors, unaffected by length. This approach makes the similarity comparison between feature vectors more accurate, improving the precision and robustness of subsequent tasks. This CNN network model has been pre-trained on the large-scale dataset of aero-engine borehole damage described in step 1.

[0052] Step 2.2: Extract the original feature map from the first convolutional layer of the YOLOv5 damage detection network head, and obtain the "background feature" vector through the background decoupling module (BDM).

[0053] The reason for choosing to extract the raw feature map from the first convolutional layer of the YOLOv5 damage detection network's detection head is that this layer typically contains high-level features crucial for subsequent detection tasks. In the YOLOv5 network, the detection head is a key part of object detection, and its first convolutional layer, after multiple deep learning iterations, captures complex features particularly critical for damage detection. These features not only include basic visual information but also incorporate deeper abstract information learned by the network from previous layers, thus providing a rich data source for background decoupling. This allows the BDM module to more effectively separate damage features from complex backgrounds.

[0054] The algorithm flowchart of the BDM module mentioned above is as follows: Figure 3As shown, this module first processes the original feature map through four 3×3 dilated convolutional layers with dilation rates of 1, 2, 4, and 8. The outputs of each layer pass information to each other in a progressive, deep-layer fusion manner to effectively integrate features at different depths, allowing subsequent layers to have a larger receptive field. Finally, the feature maps from each stage are concatenated to form the final output feature map. This design enhances the module's ability to capture multi-scale spatial information, ensuring a comprehensive understanding from details to the global picture.

[0055] Next, a spatial encoder is used, with an overall structure of three parallel branches. Each branch corresponds to max pooling, regular convolution, and upsampling operations with pooling coefficients of 1, 2, and 3, respectively, to further enrich and refine the feature representation. This multi-branch spatial encoder enables the model to more comprehensively describe the background and improves its ability to process background features at different scales.

[0056] To enable the multi-task network to better extract background features through the BDM module during the prediction phase, a background generation module (BGM) is further introduced during the training phase of the second task branch. The purpose of this is to obtain a "background region image" for training through the BGM.

[0057] The algorithm flowchart for the BGM module is as follows: Figure 4 As shown, this module consists of a ResNet-based feature encoder, reflection padding, instance normalization, and transposed convolution, designed to reconstruct an image containing only the background from the feature maps generated by the BDM module. The ResNet feature encoder is composed of multiple stacked ResNet blocks. Each ResNet block contains skip connections, allowing the network to learn features at multiple levels while avoiding the vanishing gradient problem common in deep neural networks. The ResNet blocks employ a progressive design, meaning the output of each block serves not only as input to the next block but also as input to subsequent blocks, aiming to improve the learning ability of deep networks while retaining sensitivity to primary features. This design enables the ResNet feature encoder to effectively extract deep and complex features while maintaining the stability of network training.

[0058] ReflectionPad is used to process edges before convolution operations to reduce boundary artifacts. This padding method can extend the image boundaries while maintaining the naturalness of the image, avoiding boundary artifacts and making the generated background image more natural and continuous. Instance Normalization, on the other hand, normalizes independently on each instance and is widely used in generative models. Using instance normalization in background generation can not only accelerate model convergence but also maintain the independence between each image instance. Transposed convolution is used to reconstruct the image from encoded features, increasing the spatial dimension of the feature map to obtain a higher resolution output.

[0059] Obtaining the "background region label" involves the following sub-steps: First, segment and label the small-scale borehole damage data to create a segmentation training dataset, and pre-train the dataset using an FCN network. Next, use the trained segmentation model to perform mask prediction on the large-scale aero-engine borehole damage dataset described in step 1 to ensure accurate differentiation between damaged and non-damaged regions. Then, calculate the location information of the damaged region and its bounding box using the mask, average the pixels within the damaged bounding box (excluding the damaged region), and replace the damaged region pixels with this average value to obtain the "background region label." These steps effectively generate labels representing the background region, providing an accurate data foundation for subsequent processing.

[0060] Figure 4 In the diagram, G7 represents a 7×7 convolution kernel, and G3 represents a 3×3 convolution kernel.

[0061] During the training phase, the loss is measured by calculating the Manhattan distance between the pixels of the "background region image" and the "background region label". The Manhattan distance is expressed as:

[0062]

[0063] Wherein, the length and width of the image are H and W respectively, and the pixel value of each point in the "background region label" and "background region image" is represented by I. ij and I i ′ j By utilizing a loss metric mechanism, the network can learn during training to extract background features more accurately through the BDM module, ensuring effective differentiation between background and damaged areas in practical applications.

[0064] Step 3: Calculate the cosine similarity between the "damage feature" vector and the "background feature" vector between the current frame and the previous frame, respectively, to obtain the damage feature similarity Sim. damage and background feature similarity Sim background ;

[0065] Specifically, to facilitate the calculation of the cosine similarity between the damage features and background features between the current frame and previous frames, we extracted the 128-dimensional feature vector of the "damage features" in step 2.1 using a fully connected layer of a Wide Residual Network. Similarly, the 128-dimensional feature vector of the "background features" in step 2.2 was obtained using a similar fully connected layer after the image was processed by the Background Decoupling Module (BDM).

[0066] The specific method for calculating feature similarity is to calculate the cosine similarity between the "damage feature vector" and "background feature vector" of the current nth frame and the corresponding feature vectors from the 1st to the (n-1th)th frames, and then take the maximum cosine similarity as Sim. damage and Sim background Let the damage feature vector and background feature vector, consisting of the vectors from frame 1 to the current nth frame, be respectively: The calculation expression for "damage feature similarity" can then be more intuitively represented as:

[0067]

[0068] in, This represents the j-th element in the current n-th frame of the damage feature vector. This represents the j-th element of the i-th frame in the damage feature vector. Similarly, the expression for calculating the cosine similarity of the background features is:

[0069]

[0070] Step 4: Select appropriate weights based on the specific requirements of the task scenario and calculate the comprehensive similarity Sim. composite Overall similarity Sim composite The expression is:

[0071] Sim composite =α·Sim damage +(1-α)·Sim background

[0072] Here, α is a weighting parameter used to adjust the contribution of damage feature similarity and background feature similarity to the overall similarity. The selection of this parameter needs to be determined experimentally, by observing the model's performance on the test set and choosing the most suitable weight to meet the task requirements.

[0073] Step 5: Calculate the similarity score and compare it with a preset threshold. If the score is greater than the threshold, the damage is considered the same and not counted; otherwise, the damage is considered different and counted. The expression for the similarity score is:

[0074]

[0075] score = sigmoid(sim composite )

[0076] The similarity score is mapped to the interval (0,1), which facilitates interpretation and application in practical applications. At the same time, the key to setting the threshold is to balance sensitivity and specificity to ensure that the model works effectively in different scenarios.

[0077] This study uses Total Counting Quantity (TCQ), Counting Error Ratio (CER), Repeat Counting (RC), and Missing Counting (MC) to comprehensively evaluate the counting method. The total counting quantity represents the total number of counts achieved by the counting method, while the counting error ratio reflects the ratio of the counted quantity to the actual number. The expression for the counting error ratio is:

[0078]

[0079] Considering the phenomenon of low CER caused by the mutual cancellation of duplicate counts and undercounts—for example, if a method has a high number of duplicate counts but also a high number of undercounts—its total count may be close to the actual target number, but this does not mean that the counting method is accurate. Therefore, this study proposes Comprehensive Counting Discrepancy (CCD) to more comprehensively reflect the actual performance of the counting method. Its calculation formula is as follows:

[0080]

[0081] In the two equations above, S represents the actual quantity of the target. By introducing the evaluation index CCD, the performance of the counting method can be evaluated more intuitively, especially considering the impact of duplicate counting and undercounting on counting accuracy, providing a more objective evaluation method.

[0082] The damage similarity discrimination performance in counting algorithms based on multi-task learning and background decoupling is as follows: Figure 5As shown in the figure, (a) and (c) represent the similarity scores between the anchored sample and the positive sample, while (b) and (d) represent the similarity scores between the anchored sample and the negative sample. The similarity discrimination results demonstrate that the proposed similarity discrimination method can effectively identify the feature differences of the same damage in different scenes caused by frequent changes in the scene captured by the borehole camera. This result not only proves that the damage similarity judgment method based on foreground and background features proposed in this study has high accuracy and reliability, but also lays a good foundation for subsequent counting functions based on similarity scores.

[0083] Given that YOLOv5 combined with the Deepsort algorithm and the counting line algorithm has been widely applied in counting research across multiple fields and has demonstrated good performance, this study aims to compare the counting performance of the proposed counting algorithm based on multi-task learning and background decoupling with these existing methods through comparative experiments, thereby verifying the effectiveness of the proposed method in this field. As shown in Table 1, this study experimentally investigated the counting performance of different methods in 40 segments of aero-engine borehole videos and conducted a detailed comparative analysis to ensure the accuracy and reliability of the research results.

[0084] Table 1 Counting results of different algorithms

[0085] Table 1 Counting performance of different algorithms

[0086]

[0087] To objectively evaluate the performance of different counting algorithms, this study analyzed the three sets of data in the table in detail. The results show that, while keeping the target detection algorithm unchanged, the counting method based on multi-task learning and background decoupling proposed in this study has significant performance advantages compared to the traditional counting line algorithm and the Deepsort algorithm. Especially when dealing with complex scene transitions in aero-engine borehole damage videos, this algorithm significantly reduces repeated counting of the same damage, reducing the Comprehensive Counting Deviation (CCD) to 4.9%, far lower than the 17.65% of the Deepsort algorithm and the 49.02% of the counting line algorithm, achieving higher accuracy in aero-engine borehole damage counting.

Claims

1. A borehole damage counting method based on multi-task learning and background decoupling, characterized in that, Includes the following steps: Step 1) Create a borehole damage dataset and train a damage detection network based on YOLOv5; Step 2) Perform multi-task learning from both the damage features and background features of the borehole damage video frame; Step 2.1) Obtain the damage target prediction box of the current frame through the damage detection network, crop the corresponding damage region of the current video frame according to the prediction box parameters, and feed the cropped region into the pre-trained CNN model to obtain the "damage feature" vector. Step 2.2) Extract the original feature map from the first convolutional layer of the damage detection network head, and obtain the "background feature" vector through the background decoupling module (BDM). Step 3) Calculate the cosine similarity between the "damage feature" vector and the "background feature" vector between the current frame and the previous frame, respectively, to obtain the damage feature similarity and the background feature similarity; Step 4) Select appropriate weights based on the task scenario requirements and calculate the overall similarity; Step 5) Calculate the similarity score and compare it with a preset threshold; If the similarity score is greater than the threshold, it is determined to be the same damage and is not counted; otherwise, it is determined to be different damage and is counted. The sub-steps of step 2.2) include the following: 2.2.1) Extract the original feature map from the first convolutional layer of the YOLOv5 damage detection network head, and generate a background feature map through the background decoupling module (BDM): First, the original feature map is processed through multiple dilated convolutional layers. The output of each layer passes information to each other in a progressively deeper fusion manner to fuse features at different depths. Then, the feature maps output by each dilated convolutional layer are concatenated to form the final output feature map; next, a spatial encoder with a multi-parallel branch structure is used to process the background feature map. 2.2.2) The background generation module (BGM) reconstructs an image containing only the background from the background feature map, i.e., the "background region image": The BGM module's structure revolves around a ResNet-based feature encoder, reflection padding, instance normalization, and transposed convolution. Reflection padding handles edges before convolution operations. The ResNet feature encoder is composed of stacked ResNet blocks, each with a progressive structure. The output of each ResNet block serves not only as input to the next block but also as input to subsequent blocks. Transposed convolution reconstructs the image from the encoded features, increasing the spatial dimension of the feature map to achieve higher resolution output. Instance normalization is performed independently on each instance. 2.2.3) Background region labels obtained by mask prediction of the borehole damage dataset using a pre-trained FCN segmentation model: The location information of the damaged area and its bounding box is calculated by using a mask. The average value of the pixels in the non-damaged area within the bounding box is calculated and the pixels in the damaged area are replaced with this average value to generate a "background area label". 2.2.4) During the training phase, the loss is measured by calculating the Manhattan distance between the pixels of the "background region image" and the "background region label".

2. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, In step 1), when creating the borehole damage dataset, the images and video data of borehole damage are first collected and labeled; then the training set, validation set and test set are divided proportionally to form the borehole damage dataset; then the training parameters are set; and finally the damage detection network is trained.

3. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, In step 2.1), the CNN network model uses a wide residual network, which includes two convolutional layers and six residual blocks. A global feature map with a dimension of 128 is computed in the fully connected layer. Finally, the features are projected onto the unit hypersphere through batch normalization and L2 regularization. The CNN network model has been pre-trained on the borehole damage dataset described in step 1.

4. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, Steps 2.2.2) to 2.2.4) are implemented only during the training phase, which involves extracting background features from the original feature map of YOLOv5 using the background decoupling module BDM and calculating the similarity to the background features of the previous frame.

5. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, In step 2.2.3): First, segmentation and labeling are performed on small-scale borehole damage data to create a segmentation training dataset, and the FCN segmentation model is pre-trained on the dataset. Then, the trained segmentation model is used to perform mask prediction on the borehole damage dataset obtained in step 1 to distinguish between damaged and non-damaged regions. The position information of the damaged region and its bounding box is calculated through the mask. The average value of the pixels within the damaged bounding box that do not include the damaged region is calculated, and the pixels in the damaged region are replaced with this average value to obtain the "background region label".

6. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, In step 2.2.4), the Manhattan distance expression is: Wherein, the length and width of the image are H and W respectively, and the pixel value of each point in the "background region label" and "background region image" is represented by I. ij and I′ ij .

7. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, In step 3), the cosine similarity between the "damage feature" vector and the "background feature" vector of the current nth frame and the corresponding feature vectors of frames 1 to (n-1) is calculated, and the maximum cosine similarity is taken as Sim. damage and Sim background .

8. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, The comprehensive similarity Sim in step 4) composite Based on damage feature similarity Sim damage Sim similarity with background features background The weighted combination is used to adjust the contribution of damage feature similarity and background feature similarity to the overall similarity.

9. The borehole damage counting method based on multi-task learning and background decoupling according to claim 1, characterized in that, In step 5), the sigmoid function is used to calculate the comprehensive similarity Sim. composite Mapping to the (0,1) interval yields a similarity score for interpretation and comparison.

Citation Information

Patent Citations

  • Intelligent detection and judgment method for repeated damage in aviation hole exploration video

    CN115147385A