Nut loosening detection method and device based on mask guided segmentation network and medium

Abstract

The application discloses a nut loosening detection method and device based on a mask guided segmentation network and a medium, wherein the nut loosening detection method based on the mask guided segmentation network comprises the following steps: image acquisition and diversified data enhancement; a mask guided boundary enhanced multi-path segmentation network; ring and segmented division based on a fine segmentation mask; constructing an optical flow estimation model; obtaining a displacement time sequence and a time sequence loosening feature; and determining the nut loosening state based on an adaptive threshold. Compared with the prior art, the nut loosening detection method based on the mask guided segmentation network can significantly improve the segmentation boundary definition and the perception ability of small amplitude rotation / radial displacement under strong light reflection and noise background, can distinguish short-term random disturbance from persistent loosening, does not need physical markers, and is suitable for in-service monitoring of wind power equipment and visual state detection of other fastener structures.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to a method, device, and medium for detecting loose nuts based on a mask-guided segmentation network. Background Technology

[0002] The flange section of a wind turbine tower is connected and fixed using multiple turns of metal nuts and high-strength bolts. Due to long-term exposure to wind loads, yaw, start-up and shutdown impacts, and environmental changes, the nuts are prone to gradual loosening and accumulation of minute displacements, threatening the structural safety of the tower.

[0003] Existing contact-based or vibration-response methods rely on sensor deployment, which is costly and struggles to accurately pinpoint the state of individual nuts. Traditional image processing and classical machine learning methods are extremely sensitive to changes in lighting, strong reflections, and shadows, resulting in significant false positives and false negatives under complex lighting and corrosion conditions in tower structures. Even with deep segmentation networks, it remains difficult to obtain accurate and reliable nut profiles under highly reflective and complex backgrounds. Furthermore, due to the lack of full utilization of the spatial prior of the annular arrangement of flange nuts, they are insensitive to minute rotational loosening along the circumference. Simultaneously, existing methods lack displacement time-series modeling based on optical flow and adaptive thresholding mechanisms, making it difficult to stably identify nut loosening states under label-free conditions.

[0004] Solving these problems is now a top priority. Summary of the Invention

[0005] To address the technical problems of existing methods, such as difficulty in fine segmentation under highly reflective and complex backgrounds, insufficient utilization of the prior knowledge of the nut's ring structure, inadequate perception of minute optical flow displacement, and lack of temporal loosening modeling and adaptive threshold determination mechanisms, this invention provides a nut loosening detection method, device, and medium based on a mask-guided segmentation network.

[0006] The technical solution is as follows:

[0007] The first aspect of this application relates to a nut loosening detection method based on a mask-guided segmentation network, which is performed according to the following steps:

[0008] S1. Collect an image dataset containing the nut and its surrounding background, and perform various data augmentation processes on the image dataset to obtain a preprocessed image dataset;

[0009] S2. Construct a mask-guided boundary enhancement multipath segmentation network to perform feature extraction and multipath segmentation on the preprocessed image dataset, obtaining a fine-grained segmentation mask for the nut against a highly reflective and complex background. ;

[0010] S3, Nut-based fine segmentation mask The nut group area is divided into angular partitions under the circular coordinate system to determine the spatial location area of ​​each nut;

[0011] S4. Construct an optical flow estimation model that integrates the amorphous search operator and the attention feature localization mechanism, perform optical flow estimation on adjacent frame nut images, and obtain the optical flow displacement vector of each nut under the unmarked condition;

[0012] S5. Perform time-series trajectory modeling on the optical flow displacement vector of each nut, analyze its displacement accumulation trend and direction consistency, and form a nut loosening characteristic sequence;

[0013] S6. Under unmarked conditions, based on the optical flow distribution of historical normal working conditions, the loosening judgment threshold is adaptively set and compared with the nut loosening feature sequence to output nut loosening status information.

[0014] This paper proposes an integrated technical approach for detecting loose nuts using a mask-guided segmentation network based on the above-mentioned mask-guided segmentation network. Addressing the online monitoring needs of wind turbine tower flanges under complex conditions such as strong reflection, obstruction, corrosion, and imaging noise, the approach integrates fine segmentation, ring structure modeling, label-free optical flow estimation, temporal feature analysis, and adaptive threshold determination. Compared with existing methods, this method significantly improves the clarity of segmentation boundaries and the ability to detect small rotational / radial displacements in strong reflective and noisy environments. It can distinguish between short-term random disturbances and persistent loosening, requires no physical markers, and is suitable for in-service monitoring of wind power equipment and visual condition detection of other fastener-like structures.

[0015] The second aspect of this application relates to an electronic device, including at least one processor and a memory communicatively connected to the at least one processor, the memory storing a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the above-described nut loosening detection method.

[0016] Using the above electronic equipment, all the advantages of the aforementioned nut loosening detection method are achieved.

[0017] A third aspect of this application relates to a computer-readable medium, characterized in that the computer-readable medium stores computer instructions for causing at least one processor of the aforementioned electronic device to execute the aforementioned nut loosening detection method.

[0018] Using the above computer-readable medium, all the advantages of the above-mentioned nut loosening detection method are achieved. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a nut loosening detection method based on a mask-guided segmentation network.

[0020] Figure 2This is a time series graph showing the nut displacement under normal operating conditions.

[0021] Figure 3 This is a time series graph showing the displacement of the nut under loosening conditions;

[0022] Figure 4 The diagram shows the actual performance of the adaptive threshold determination mechanism based on Mahalanobis distance.

[0023] Figure 5 This is a performance comparison chart of different methods on four datasets. Detailed Implementation

[0024] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0025] Example 1:

[0026] A nut loosening detection method based on mask-guided segmentation network is performed according to the following steps:

[0027] S1. Collect an image dataset containing the nut and its surrounding background, and perform various data augmentation processes on the image dataset to obtain a preprocessed image dataset.

[0028] Among these, the diverse data augmentation processing of image datasets includes at least the following:

[0029] Simulates brightness and contrast perturbations and local bright spots under complex lighting and reflection conditions;

[0030] Simulate random rotation and scaling from different shooting angles and nut installation postures;

[0031] Noise injection simulating imaging interference is used to generate training images containing occlusion and surface defects through sample mixing, thereby improving the robustness and generalization ability of the segmentation model under conditions with few samples and complex working conditions; among which, simulated imaging interference includes dust, rust and water mist, etc.

[0032] Specifically, industrial cameras are deployed inside and outside the wind turbine tower in the nut area to capture images of the metal nuts at the tower flange. A single original image is denoted as:

[0033] ;

[0034] In the above formula, Represents the set of real numbers; and These represent the image height and width, respectively; 3 represents the number of image channels, which correspond to the RGB values ​​of a color image. These are pixel coordinates.

[0035] The original image undergoes basic preprocessing to obtain the preprocessed image. including but not limited to:

[0036] Geometric correction and distortion correction: Using the calibrated intrinsic parameter matrix and distortion coefficients, the geometric correction is performed on... Perform distortion correction transformation;

[0037] Unified scaling and cropping: scaling images to a fixed resolution uniformly. And make appropriate cuts based on the prior position of the nut area;

[0038] Brightness normalization: Histogram equalization correction is used to adjust the brightness distribution to obtain the basic preprocessed image. .

[0039] To improve the generalization ability of the segmentation model under conditions of few samples and complex scenarios, the basic preprocessed images are... Perform diverse data augmentation. Let one augmentation transformation be denoted as an operator. The enhanced image for:

[0040] .

[0041] Data augmentation includes the following types of operations:

[0042] (1) Simulation of complex lighting and reflection:

[0043] The global brightness and contrast are perturbed to obtain the perturbed image. :

[0044] ;

[0045] In the above formula, This is the contrast scaling factor. This is for brightness bias.

[0046] Simultaneously, a localized high-brightness spot mask is superimposed in certain areas. This creates images that simulate metallic reflections, light spots, and overexposure. :

[0047] ;

[0048] In the above formula, This is the highlight intensity vector. This indicates pixel-by-pixel multiplication.

[0049] (2) Simulation of viewing angle and installation posture:

[0050] Apply random rotation and scaling transformation to the image to obtain the transformed image. :

[0051] ;

[0052] In the above formula, It is an affine transformation consisting of random rotation angles and scaling factors, used to simulate changes in camera viewpoint and differences in nut installation posture.

[0053] (3) Imaging interference noise injection:

[0054] Random noise simulating imaging interference such as dust, rust, and water mist is superimposed on the image to obtain the image with superimposed random noise. :

[0055] ;

[0056] In the above formula, This represents Gaussian noise, salt-and-pepper noise, or texture-fitting noise based on real rust textures, which is used to simulate sensor noise and surface contamination.

[0057] (4) Sample blending, occlusion, and defect scene generation:

[0058] For two images and Perform sample mixing, for example:

[0059] ;

[0060] In the above formula, This represents the blended image. The mixing coefficients are used to paste local nut blocks, occlusion blocks, or defect blocks from another image into the current image to generate training samples that include scenes with local occlusion and surface defects.

[0061] In the actual data augmentation process, the following procedure is adopted to better adapt the model to the actual working conditions of wind turbine towers:

[0062] First, a portion of the original images was retained as the foundation of the dataset. The remaining portion of the original images was used for data augmentation. Instead of applying all augmentation operations simultaneously to every image, each image was randomly selected from multiple augmentation methods and combined, then processed by the aforementioned operators. The effect of this process ultimately yields an enhanced image set, which constitutes a preprocessed image dataset used for fine segmentation of the nut and subsequent optical flow loosening detection.

[0063] S2. Construct a mask-guided boundary enhancement multipath segmentation network to perform feature extraction and multipath segmentation on the preprocessed image dataset, obtaining a fine-grained segmentation mask for the nut against a highly reflective and complex background. .

[0064] Specifically, step S2 is performed according to the following steps:

[0065] S21. Use a unified feature extraction network to perform preliminary feature extraction on the preprocessed image dataset to obtain a basic feature map that can be used by multiple segmentation paths. .

[0066] Feature extraction network It consists of multiple convolutional layers, nonlinear activation, and downsampling operations, forming a basic feature map. Represented as:

[0067] ;

[0068] In the above formula, Indicates the first Layered convolutional activation units. The network internally fuses features at different scales through a feature pyramid, so that the basic feature map simultaneously contains the local texture and overall semantic information of the nut.

[0069] S22, Transfer the basic feature map Input to the main semantic path respectively High-frequency path at the edge and reflection suppression path This yields three types of feature outputs:

[0070] Feature map output by the main semantic path Its expression is , used to characterize the semantic features of the entire nut region;

[0071] Feature map of high-frequency path output at the edge Its expression is , used to characterize the boundary and fine structural features of the nut;

[0072] Feature map of reflection suppression path output Its expression is , used to characterize the weight distribution of highly reflective areas; Its characterizing pixels The confidence level that the area belongs to a highly reflective region.

[0073] S23. Generate an initial nut mask based on the intermediate prediction results output by the main semantic path during the decoding stage. mask the initial nut The information is input into the high-frequency edge path and the reflection suppression path as masking guidance information to construct the boundary attention features. Weighted graph with reflection suppression .

[0074] Specifically, firstly, by the decoder Feature map output from the main semantic path Upsampling yields the initial probability map :

[0075] ;

[0076] In the above formula, Represents pixels This represents the predicted probability of the nut.

[0077] Activated by Sigmoid function Obtain the initial nut mask :

[0078] .

[0079] Then, the initial nut mask Feature map of edge high-frequency path output Pixel-wise multiplication yields boundary attention features:

[0080] ;

[0081] Therefore, it can enhance the high-frequency response in the neighborhood of the mask boundary, suppress background noise outside the mask, and form a boundary attention enhancement feature.

[0082] Simultaneously, the feature map output by the reflection suppression path will be... With the initial nut mask The resulting weighted map is obtained after reflection suppression. :

[0083] ;

[0084] Weighted map after reflection suppression In the subsequent decoding process, the feature response at the corresponding position is suppressed, thereby reducing the interference of the highly reflective area near the nut boundary on the segmentation result.

[0085] S24. Initial nut mask Iterative refinement processing is performed to obtain the precision nut mask. .

[0086] Specifically, for the initial nut mask An iterative refinement method is used to perform multiple updates. During the update process, the mask boundaries and details are gradually corrected using the current mask and multi-path fusion features as input. When the number of iterations reaches a preset value, the refined nut mask is output. .

[0087] The iterative refinement method involves setting up iterative mask optimization units to perform multiple refinements on the nut mask. Each iterative mask optimization unit uses the current mask... Using the fused multi-path features as input, predict the mask residual. Update mask After several iterations, the fine-tuning nut mask is output. .

[0088] Specifically, this involves setting an iterative mask optimization unit. With the current mask Using the fused multi-path features as input, predict the mask residual. :

[0089] ;

[0090] In the above formula, for The fusion result obtained by concatenating the three features along the channel dimension and performing a linear transformation is as follows:

[0091] ;

[0092] In the above formula, Represents the tensor concatenation operator;

[0093] Use residual update rules The mask is iterated and corrected multiple times until the number of iterations reaches a preset value. Finally, the precision nut mask was obtained. :

[0094] .

[0095] S25. Output the feature map of the main semantic path. Boundary attention features Weighted graph with reflection suppression Attention-weighted fusion is performed to obtain attention-weighted fusion features. :

[0096] ;

[0097] Then through the attention fusion module Introducing a precision nut mask As a guide, weighted fusion features are obtained. :

[0098] ;

[0099] Finally, Input final decoder Obtain the predicted probability map And through the threshold function Obtain the fine segmentation mask for the nut :

[0100] , ;

[0101] To ensure that the segmentation network effectively learns clear nut boundaries and suppresses missegmentation of highly reflective areas during the training phase, this embodiment... Compared with the true labeled mask Design the following loss function:

[0102] First, construct a method based on cross-entropy loss. and Dice loss Basic segmentation loss of weighted combination :

[0103] ;

[0104] In the above formula, and These are all weighting coefficients, used to balance the effects of cross-entropy loss and Dice loss, respectively.

[0105] Based on this, boundary contrast loss is introduced. and reflection suppression loss As an additional constraint, it is used to enhance the boundary quality of the nut and reduce the interference of highly reflective areas on the segmentation results:

[0106] ;

[0107] In the above formula, Indicates auxiliary loss, and These are all weighting coefficients, which are used to balance the effects of boundary contrast loss and reflection suppression loss, respectively.

[0108] Finally, the total loss function for:

[0109] ;

[0110] By minimizing Training the network significantly improves the clarity of the nut boundary while maintaining overall segmentation accuracy and suppresses missegmentation caused by strong reflections and complex backgrounds.

[0111] S3, Nut-based fine segmentation mask In a single-circular coordinate system, the nuts within the current field of view are segmented by angle to determine the spatial location regions of each nut, providing structural constraints along the circumferential direction for subsequent optical flow estimation and displacement analysis. Here, the fine segmentation mask for the local arc-shaped nut image of the current frame is denoted as... ,and .

[0112] Specifically, step S3 is performed according to the following steps:

[0113] S31. Based on the fine segmentation mask of the local arc-shaped nut image in the current frame, Get the nut region set :

[0114] ;

[0115] Nut-based fine segmentation mask Extract the connected regions of each nut (for) (Perform connected component analysis) to obtain the target region for each nut. And calculate its centroid coordinates. :

[0116] ;

[0117] .

[0118] S32. Combining equipment calibration with geometric relationships, the position of the geometric center of the tower flange in the current image coordinate system is represented as follows: , as the origin of the polar coordinate system. Based on the centroid coordinates of each nut. Calculate its polar coordinates :

[0119] ;

[0120] .

[0121] And Normalization to The normalized angle is obtained. :

[0122] ;

[0123] In this embodiment, the tower flange nuts are arranged in a single loop, and the radial direction can be regarded as an effective ring, mainly utilizing the normalized angle. Describe the position of the nut on the circumference.

[0124] S33. Divide the entire circle into several equally wide sectors according to the angle, and map the angular position of each nut to the corresponding sector to obtain the nut's angular segment identifier, which is used to constrain subsequent optical flow estimation and displacement analysis.

[0125] Specifically, the angle range Divided into Each sector has an equal width angle, and the sector angle width is... for:

[0126] ;

[0127] No. Each sector Defined as:

[0128] ;

[0129] In the above formula, Indicates the first The starting boundary angle of each sector;

[0130] For the A nut, its normalized angle Sector number for:

[0131] ;

[0132] Therefore, each nut can be identified by its sector number. A unique angular segmentation identifier is obtained on the global single-circle ring. This identifier will be used in subsequent optical flow estimation and temporal analysis to constrain the optical flow search range of the nut and measure the angular drift along the circumferential direction, thereby improving the ability to detect nut rotational loosening.

[0133] S4. Construct an optical flow estimation model that integrates an amorphous search operator and an attention feature localization mechanism. Under the constraint of a single-loop ring structure, perform optical flow estimation on adjacent frame nut images to obtain the optical flow displacement vector of each nut under unmarked conditions. Let the preprocessed images of two adjacent frames be denoted as... and The corresponding fine-grained nut segmentation mask is The nut sector number obtained in step S3 is .

[0134] Specifically, step S4 is performed according to the following steps:

[0135] S41. For adjacent frame nut images, respectively, use a feature extraction network with shared weights. Extract multi-scale features and construct a multi-scale feature pyramid for optical flow estimation:

[0136] ;

[0137] In the above formula, Indicates scale level. and Representing time respectively and scale Feature map on.

[0138] S42. On each scale feature map, the amorphous search operator is used. Within a certain search neighborhood, correlation matching is performed between the nut region of the current frame and the candidate region of the next frame to obtain matching similarity features and optical flow candidates:

[0139] scale Feature map on and For each position Define the distribution of relevant matching similarities. for:

[0140] ;

[0141] In the above formula, Represents the dot product within the channel. This represents the set of offsets that are allowed to be searched. Indicates in At all times within the scale On the feature map, location The feature vector at that location, Indicates in At all times on the same scale On the feature map, position The feature vector at that location, This represents the initial set of candidate optical flow offsets.

[0142] Amorphous search operators allow Its shape and range adapt to scale and local structure to accommodate the tiny rotations and translations of the nut.

[0143] S43. By performing correlation or similarity calculations within multi-scale and multi-morphological search domains, optical flow candidate matching results adapted to the minute rotations and translations of the nut are obtained. That is, for the same position... According to the similarity distribution An initial set of candidate optical flow offsets can be obtained. And its confidence characteristics.

[0144] S44. Construct a global spatial enhanced attention feature localizer, integrating multi-scale feature pyramids. and Input Attention Module This approach models the long-range dependencies between each nut and the background region using global attention, suppressing repetitive textures and background interference. The output is a globally attention-enhanced feature. and :

[0145] .

[0146] Global attention suppresses repetitive textures and background interference that are less relevant to the nut region by weighted aggregation of responses across the entire feature map, making optical flow estimation more focused on the structural region where the nut is located.

[0147] S45. The matching similarity features obtained by the amorphous search operator... With global attention enhancement features and To integrate and form a scale Optical flow estimation input features :

[0148] ;

[0149] In the above formula, This represents feature concatenation and subsequent linear transformation, which is used to compress and reconstruct feature channels.

[0150] Input each optical flow estimate into the features Input the optical flow regression network sequentially. Through progressive upsampling and refinement, the optical flow vector field of each pixel is finally output at the original resolution. :

[0151] ;

[0152] For the The mask area corresponding to each nut The displacement vector of the nut can be obtained by averaging over a region.

[0153] ;

[0154] In the above formula, Indicates the first A nut at all times The planar displacement is expressed as follows: , Indicates the first A nut at all times exist Average optical flow in the direction, Indicates the first A nut at all times exist Average optical flow in the direction.

[0155] S5. Perform time-series trajectory modeling on the optical flow displacement vector of each nut, analyze its displacement accumulation trend and direction consistency, and form a nut loosening characteristic sequence.

[0156] Specifically, step S5 is performed according to the following steps:

[0157] S51. Within a given time window, accumulate and filter the optical flow displacement vectors of multiple consecutive frames to construct the displacement time series of each nut. Specifically, based on the geometric center of the tower flange... Using the polar coordinate origin, this displacement is uniformly converted into a polar coordinate displacement time series. Record the time points. Time polar coordinates of a nut Then in the inter-frame polar coordinate displacement Defined as:

[0158] ;

[0159] Select length as Sliding time window , for the The polar coordinate displacement time series of the nuts is as follows:

[0160] .

[0161] S52. Perform trend analysis and feature extraction on the displacement time series to obtain a time-series loosening feature vector. This time-series loosening feature vector includes cumulative displacement, displacement direction consistency, and oscillation characteristics. Among them, cumulative displacement is defined as the sum of the displacement magnitudes of each frame within a preset time window, which is used to characterize the overall displacement degree of the nut. Displacement direction consistency is defined as a measure of the consistency of the angle between the displacement directions of each frame within a preset time window, which is used to characterize whether the displacement continues to shift along the main direction. Oscillation characteristics are defined as a measure of the amplitude of the reciprocating change of displacement over time, which is used to characterize whether the nut has back-and-forth shaking or vibration behavior.

[0162] Overall, for each nut within a given time window, features are extracted from the following three aspects: the cumulative amount of angular and radial displacement. and Consistency of angular displacement direction Oscillation intensity of angular displacement This allows us to obtain a feature vector that comprehensively reflects displacement intensity, continuous offset trend, and reciprocating sway characteristics. Constructing the first... The temporal loosening feature vector of a nut in the current time window for:

[0163] .

[0164] Specifically, step S52 is performed according to the following steps:

[0165] S521, within the time window within, no. The cumulative angular displacement and cumulative radial displacement of each nut are defined as follows:

[0166] ;

[0167] In the above formula, Used to characterize the overall rotational displacement of the nut during the window period. Used to characterize the overall displacement along the radial direction.

[0168] S522. To characterize whether the angular displacement of the nut continues to shift in a certain main direction within the window period, the angular displacement of each frame is uniformly mapped to a unit direction scalar and averaged to obtain the angular displacement direction consistency feature.

[0169] The angle displacement symbol for each frame Standardized as:

[0170] ;

[0171] Directional Consistency Index for:

[0172] ;

[0173] when When it is close to 1, it indicates that the angular displacement direction of the nut is highly consistent throughout the entire time window; when When the value is close to 0, it indicates that the angular displacement direction changes frequently or cancels each other out within the window.

[0174] S523, To characterize the intensity of the reciprocating oscillation of the nut's angular displacement over time, based on Construct oscillation intensity characteristics. Calculate the average absolute value of the angular displacement within the window. :

[0175] ;

[0176] Angular displacement oscillation variance for:

[0177] ;

[0178] This indicator is used to measure the degree of fluctuation in angular displacement over time. A larger value indicates that the nut is exhibiting significant back-and-forth shaking or vibration.

[0179] S53. Use the temporal loosening feature vector to distinguish between short-term random disturbances and persistent loosening behavior. That is: the first... The temporal loosening feature vector of a nut in the current time window for:

[0180]

[0181] The timing loosening feature vector can be used to distinguish between short-term random disturbances and continuous loosening behavior of nuts.

[0182] To visually demonstrate the ability of the proposed time series analysis method to identify loosened states, Figure 2 and Figure 3 A comparison of the nut displacement time series under normal and loosening conditions is presented. From... Figure 2 As can be seen, under normal operating conditions, the nut displacement exhibits random, small fluctuations, with low directional consistency and minimal cumulative displacement; while from... Figure 3 As can be seen, under the loosening condition, the angular displacement exhibits a continuous unidirectional shift, while the radial displacement accumulates synchronously, showing high directional consistency and significantly enhanced oscillation intensity. This comparative result verifies that the displacement accumulation, directional consistency, and oscillation intensity characteristics constructed in step S5 can effectively distinguish between random disturbances and continuous loosening.

[0183] S6. Under unmarked conditions, based on the optical flow distribution of historical normal working conditions, the loosening judgment threshold is adaptively set and compared with the nut loosening feature sequence to output nut loosening status information.

[0184] Step S6 is performed according to the following steps:

[0185] S61. During the initial normal operation phase of the equipment, collect the temporal loosening feature vectors of each nut within a set time period. ,in The sample is considered to be under "normal operating conditions". Its characteristic distribution parameters are statistically analyzed to construct a baseline model for normal operating conditions; these parameters include the mean, covariance, quantile range, and extreme values. The mean is calculated. With covariance This is used to construct a baseline model under normal operating conditions. The mean vector and covariance matrix are calculated from the eigenvectors of all normal samples:

[0186] ;

[0187] In the above formula, Represents the mathematical expectation. The time-series loosening feature vector of the nut under normal operating conditions;

[0188] Mahalanobis distance Mahalanobis distance measures the deviation of each normal sample from the baseline mean. Represented as:

[0189] ;

[0190] And at a preset significance level of α=0.05, from Select from the distribution quantiles as the initial threshold :

[0191] ;

[0192] In the above formula, This represents the quantile function.

[0193] S62. During the actual operation of the equipment, calculate the nut timing loosening feature vector at the current moment in real time, and measure the deviation with the normal working condition reference model.

[0194] Specifically, during the actual operation of the equipment, for the first The feature vector of the nuts obtained in the current time window is denoted as: Its deviation distance Defined as:

[0195] ;

[0196] To adapt to slow changes in long-term operating conditions, an exponential moving average method is used to update the threshold. The median distance of all nuts during the current operating phase is denoted as... Then the threshold is updated as follows:

[0197] ;

[0198] In the above formula, Update the step size for the threshold. The threshold value at the previous time step. This is the updated threshold. System initial command. .

[0199] S63. Adaptively update the looseness judgment threshold based on the deviation measurement result, and use it to determine whether the corresponding nut is in a loose state, and output the looseness judgment result.

[0200] Specifically, at a given time, the updated threshold is used. The threshold value for the current operating condition. The state of each nut is determined, i.e.: at this time Record the first The distance between the nuts is ,but:

[0201]

[0202] The system uses a statistical model of normal working conditions and an adaptive threshold to quantitatively evaluate the temporal loosening characteristics of each nut within the current time window and provide a stable and clear judgment result on the loosening state.

[0203] Figure 4 The practical performance of the adaptive threshold determination mechanism based on Mahalanobis distance is demonstrated. In the initial normal phase (0-50 frames), the system establishes a baseline model using normal operating condition samples, with an initial threshold... The system covers a 95% confidence interval. During the slow-change phase (50-150 frames), the threshold is adaptively updated using an exponential moving average to track the drift in operating conditions. When entering the abnormal phase (after 150 frames), the Mahalanobis distance of a nut's feature vector continuously exceeds the threshold, and the system successfully detects the loose state. This figure verifies that the adaptive threshold mechanism proposed in step S6 can maintain detection stability under gradually changing operating conditions and promptly identify abnormal looseness.

[0204] The proposed method was tested on nut detection datasets from different sources (Northwestern Polytechnical University Bolt Defect Detection Dataset (NPU-BOLT), Roboflow Bolt Defect Dataset (Roboflow), MVTec Industrial Visual Anomaly Detection Dataset (MVTec), and a self-built field dataset) and compared with other models (Deep Separable Convolutional Neural Network (Xception), Face Forgery Detection Model (Face X-ray), and Convolutional Neural Network-Gated Recurrent Unit Hybrid Model (CNN-GRU)). The results are shown in Table 1.

[0205]

[0206] As can be seen from Table 1, the nut loosening detection method of the present invention is superior to the traditional detection method in all core performance indicators, as detailed below:

[0207] 1. For example Figure 5 As shown, this invention achieves the highest accuracy on all four datasets (NPU-BOLT: 95.6%, Roboflow: 93.2%, MVTec: 96.3%, self-built dataset: 95.0%), comprehensively outperforming comparative methods such as Xception, Face X-ray, and CNN-GRU, demonstrating stronger domain generalization ability.

[0208] 2. Significant improvements in natural scenes and open platform data: On the NPU-BOLT and Roboflow datasets containing complex backgrounds and imaging differences, the improvement of the present invention relative to the suboptimal model is approximately +4.1% and +4.5% respectively (for CNN-GRU and Face X-ray), indicating that it is more robust to changes in real scenes and acquisition differences.

[0209] 3. More robust under industrial anomalies and defects: In the MVTec industrial vision scenario, this invention leads with an accuracy of 96.3% (higher than Face X-ray's 94.0% and CNN-GRU's 92.8%), indicating that mask-guided segmentation and reflection suppression path can effectively improve the consistency between the boundary and the target, thereby reducing misjudgment.

[0210] 4. Better consistency of self-built on-site dataset: On the self-built on-site dataset of wind turbine towers, this invention outperforms other models with an accuracy of 95.0% (1.1% higher than CNN-GRU and 4.7% higher than Xception), and maintains stable performance under changes in day and night, weather and perspective.

[0211] 5. Advantages and Mechanisms: The overall scheme of this invention, which combines "mask-guided segmentation + ring structure constraint + attention-enhanced optical flow + temporal features + Mahalanobis distance adaptive threshold", provides better robustness and transferability for conditions such as strong reflection, occlusion, low illumination, and complex backgrounds / corrosion. At the same time, it can distinguish between short-term random disturbances and continuous loosening, thereby improving the reliability of online judgment.

[0212] In summary, the nut loosening detection method in this embodiment addresses the online monitoring needs of wind turbine tower flanges under complex operating conditions such as strong reflection, obstruction, corrosion, and imaging noise. It proposes an integrated technical route from "fine segmentation - ring structure modeling - labelless optical flow estimation - temporal features - adaptive threshold determination".

[0213] First, diverse data augmentation and preprocessing are performed during the image acquisition stage to construct a dataset covering high reflectivity and noisy scenes. Then, a mask-guided multi-path segmentation network is designed, with the main semantic path, edge high-frequency path and reflectivity suppression path working together. Boundary attention and reflectivity weight modulation mechanisms are used to suppress high-light interference, and the mask is refined through iterative mask optimization. In the decoding stage with fused attention, a fine-grained segmentation mask is output to provide a stable target region for subsequent geometric modeling and optical flow estimation.

[0214] In structural modeling, the nut's centroid is mapped to a single-ring coordinate system with the flange's geometric center as the origin, and angular segmentation is performed to form structural constraints and indices along the circumferential direction. The optical flow estimation module employs a fusion scheme of multi-scale feature pyramids, amorphous search operators, and a global spatial enhanced attention locator to accurately regress the nut's minute rotation and planar displacement vectors under label-free conditions. The time-series analysis module constructs a polar coordinate displacement time series within a window period, extracting statistical features such as cumulative displacement, directional consistency, and oscillation intensity to form the nut's time-series loosening feature vector. The judgment module establishes a Mahalanobis distance benchmark model using normal operating condition samples and uses exponential sliding to update the threshold to adapt to slow drift under long-term operating conditions. Finally, it adaptively judges the state of each nut within the current time window and outputs state information such as "normal / loose".

[0215] Example 2:

[0216] An electronic device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the aforementioned nut loosening detection method.

[0217] It should be noted that electronic devices are intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0218] A processor can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processors include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor performs the various methods and processes described above, such as the nut loosening detection method.

[0219] In some embodiments, the nut loosening detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded into and / or installed on an electronic device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the nut loosening detection method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the nut loosening detection method by any other suitable means (e.g., by means of firmware).

[0220] Various implementations of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0221] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0222] Example 3:

[0223] Computer-readable storage media can be tangible media that may contain or store computer programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer-readable storage media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0225] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0226] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0227] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. Those skilled in the art, under the guidance of the present invention, can make various similar representations without departing from the spirit and claims of the present invention, and such modifications all fall within the protection scope of the present invention.

Claims

1. A method for detecting loose nuts based on mask-guided segmentation networks, characterized in that, Follow these steps: S1. Collect an image dataset containing the nut and its surrounding background, and perform various data augmentation processes on the image dataset to obtain a preprocessed image dataset; S2. Construct a mask-guided boundary enhancement multipath segmentation network to perform feature extraction and multipath segmentation on the preprocessed image dataset, obtaining a fine-grained segmentation mask for the nut against a highly reflective and complex background. ; Step S2 is performed according to the following steps: S21. Use a unified feature extraction network to perform preliminary feature extraction on the preprocessed image dataset to obtain a basic feature map that can be used by multiple segmentation paths. S22. Input the basic feature map into the main semantic path, the edge high-frequency path, and the reflection suppression path respectively to obtain the feature map output by the main semantic path, which is used to characterize the semantic features of the entire nut region. Feature maps of edge high-frequency paths used to characterize the boundaries and fine structural features of nuts. and the feature map of the reflection suppression path output used to characterize the weight distribution of highly reflective regions. ; S23. Generate an initial nut mask based on the intermediate prediction results output by the main semantic path during the decoding stage. mask the initial nut The information is input into the high-frequency edge path and the reflection suppression path as masking guidance information to construct the boundary attention features. Weighted graph with reflection suppression ; S24. Initial nut mask Iterative refinement processing is performed to obtain the precision nut mask. ; S25. Output the feature map of the main semantic path. Boundary attention features Weighted graph with reflection suppression Perform attention-weighted fusion and combine it with a fine-tuned nut mask. The nut fine segmentation mask is output through the decoder and upsampling module. ; S3, Nut-based fine segmentation mask The nut group area is divided into angular partitions under the circular coordinate system to determine the spatial location area of ​​each nut; Step S3 is performed according to the following steps: S31, Nut-based fine segmentation mask Extract the connected regions of each nut to obtain the target region of each nut and its centroid position in the current image coordinate system; S32. Based on the equipment calibration, map the geometric center of the tower flange to the origin of polar coordinates, and calculate the polar coordinates of each nut according to the centroid coordinates of each nut. S33. Divide the entire circle into several equally wide sectors according to the angle, and map the angular position of each nut to the corresponding sector to obtain the nut's angular segmentation mark; S4. Construct an optical flow estimation model that integrates the amorphous search operator and the attention feature localization mechanism, perform optical flow estimation on adjacent frame nut images, and obtain the optical flow displacement vector of each nut under the unmarked condition; Step S4 is performed according to the following steps: S41. Perform multi-scale feature extraction on adjacent frame nut images and construct a multi-scale feature pyramid for optical flow estimation. S42. On each scale feature map, the amorphous search operator is used to perform correlation matching between the nut region of the current frame and the neighborhood region of the candidate frame to obtain the matching similarity feature. S43. By performing correlation or similarity calculations in multi-scale and multi-morphological search domains, optical flow candidate matching results adapted to the minute rotation and translation of the nut are obtained. S44. Construct a global spatial enhanced attention feature locator, input the multi-scale feature pyramid into the attention module, and model the long-range dependency between each nut and the background region through global attention; S45. The matching similarity features obtained by the amorphous search operator are fused with the weighted features output by the attention feature locator, and the fusion is input into the optical flow regression network to output the displacement vector of each nut between adjacent frames. S5. Perform time-series trajectory modeling on the optical flow displacement vector of each nut, analyze its displacement accumulation trend and direction consistency, and form a nut loosening characteristic sequence; S6. Under unmarked conditions, based on the optical flow distribution of historical normal working conditions, the loosening judgment threshold is adaptively set and compared with the nut loosening feature sequence to output nut loosening status information.

2. The method for detecting nut loosening according to claim 1, characterized in that, In step S1, performing diverse data augmentation processing on the image dataset includes at least the following: Simulates brightness and contrast perturbations and local bright spots under complex lighting and reflection conditions; Simulate random rotation and scaling from different shooting angles and nut installation postures; Noise injection to simulate imaging interference is used, and training images containing occlusion and surface defect scenes are generated through sample mixing.

3. The method for detecting nut loosening according to claim 1, characterized in that, In step S23, firstly, the decoder... Feature map output from the main semantic path Upsampling yields the initial probability map The initial nut mask is obtained after Sigmoid activation. ; Then, the initial nut mask Feature map of edge high-frequency path output Pixel-wise multiplication yields boundary attention features: ； In the above formula, Indicates pixel-by-pixel multiplication; Simultaneously, the feature map output by the reflection suppression path will be... With the initial nut mask The resulting weighted map, after reflection suppression, is obtained through fusion: 。 4. The method for detecting nut loosening according to claim 1, characterized in that, Step S5 is performed according to the following steps: S51. Within a given time window, accumulate and filter the optical flow displacement vectors of multiple consecutive frames to construct the displacement time series of each nut. S52. Perform trend analysis and feature extraction on the displacement time series to obtain the time-series loosening feature vector. The time-series loosening feature vector includes the cumulative displacement amount used to characterize the overall displacement of the nut, the consistency of displacement direction used to characterize whether the displacement continues to shift along the main direction, and the oscillation feature used to characterize whether the nut has back-and-forth shaking or vibration behavior. S53. Use the temporal loosening feature vector to distinguish between short-term random disturbances and persistent loosening behavior.

5. The method for detecting nut loosening according to claim 1, characterized in that, Step S6 is performed according to the following steps: S61. During the initial normal operation phase of the equipment, the optical flow displacement characteristics of each nut are collected within a set time period, and its characteristic distribution parameters are statistically analyzed to construct a normal operating condition benchmark model; wherein, the characteristic distribution parameters include mean, covariance, quantile range and extreme values. S62. During the actual operation of the equipment, calculate the nut timing loosening feature vector at the current moment in real time, and measure the deviation with the normal working condition reference model. S63. Adaptively update the looseness judgment threshold based on the deviation measurement result, and use it to determine whether the corresponding nut is in a loose state, and output the looseness judgment result.

6. An electronic device, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor, the memory storing a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the nut loosening detection method according to any one of claims 1 to 5.

7. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions for causing the at least one processor of the electronic device of claim 6 to perform the nut loosening detection method of any one of claims 1 to 5.

Images