Crane beam bolt detection dynamic blurred image removal recovery method based on prompt learning

Abstract

The invention discloses a crane beam bolt detection dynamic blurred image removal recovery method based on prompt learning. The method comprises the following steps: S1, acquiring crane beam bolt images under different working conditions by using an industrial camera, covering various conditions such as vibration and movement during operation of a crane, and acquiring dynamic fuzzy images of different degrees and corresponding clear images; s2, designing and training a prompt learning module according to the characteristics of the various degraded images, so that the prompt learning module can generate effective prompts containing various degradation related information of the images; s3, embedding the prompt learning module trained in the step S2, and constructing a dynamic fuzzy removal model based on prompt learning; and S4, testing the dynamic fuzzy removal model trained in the S3, and judging whether the model meets the actual requirements of crane beam bolt detection or not. The problem of dynamic blurred image restoration in crane beam bolt detection is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial equipment detection and image processing, and in particular to a method for de-dynamic blurring image restoration for crane beam bolt detection based on prompt learning. Background Art

[0002] De-blurring crane beam bolts is a key technology in crane beam bolt inspection, possessing significant engineering value in target detection, intelligent industrial monitoring, and other fields. Because crane beams are subject to frequent mechanical vibrations during operation in steel structure factories, the camera used for bolt inspection cannot maintain stability, resulting in periodic or random blurring. When the camera captures images while the crane is moving or shaking, the transient nature of the vibration can cause streaking of bolt features in the image, creating unavoidable motion blur in the captured image. This motion blur poses a significant challenge to crane beam bolt inspection. It obscures subtle surface features such as thread wear and cracks, making it difficult for inspection technology to accurately determine whether the bolts are loose or falling off, potentially causing safety hazards. Ultimately, bolts may fall off or break, disrupting the normal operation of the crane beam and causing significant economic losses or even casualties.

[0003] However, the current image deblurring technology is mainly divided into two categories:

[0004] Traditional algorithms and deep learning-based methods. Traditional algorithms are mainly divided into non-blind restoration algorithms and blind restoration algorithms. However, traditional algorithms are often based on blur models under ideal conditions and cannot be implemented in actual industrial environments. Due to network structure limitations, deep learning-based algorithms are unable to handle multiple blur conditions (such as low illumination and noise in images) in complex industrial environments. In addition, due to the large number of parameters and high computational complexity, image restoration algorithms deployed on industrial camera equipment consume a long time during the inference phase, resulting in poor real-time performance of the entire system. Summary of the Invention

[0005] In order to overcome the above technical problems, the purpose of the present invention is to provide a method for de-dynamic blurring image restoration for crane beam bolt detection based on prompt learning. This method can not only solve the complex dynamic blurring situations in actual industrial environments, but also can be efficiently deployed on the device side due to its unique network structure, effectively solving the problem of dynamic blurring image restoration in crane beam bolt detection.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for de-dynamic blurring image restoration for crane beam bolt detection based on prompt learning comprises the following steps;

[0008] S1: Collect and construct a dynamic fuzzy dataset of crane beam bolts in actual industrial plants;

[0009] An industrial camera is used to capture images of crane beam bolts under different working conditions to form a crane beam bolt image dataset. Images with different degrees of dynamic blur and corresponding clear images are obtained.

[0010] S2: Design and train a cue learning module based on the characteristics of various image degradations (including motion blur, low illumination, noise, etc.), so that it can generate effective cue information containing various image degradation information;

[0011] S3: Build and train a motion blur removal model;

[0012] The prompt learning module trained by S2 is embedded into it, and a de-motion blurring model based on prompt learning is constructed. The de-motion blurring model is trained using the crane beam bolt image dataset constructed by S1; the de-motion blurring model is allowed to learn to use prompt information to remove image motion blur, continuously optimize model performance, and obtain the restored image.

[0013] S4: Test the deblurring model trained in S3. Use the mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) metrics to quantitatively evaluate the deblurred image. Based on the evaluation results, analyze the performance of the deblurring model to determine whether the model meets the actual requirements of crane beam bolt detection.

[0014] The specific steps of S1 are:

[0015] S1.1: Use an industrial camera to capture the bolts of the crane beam from different angles under various operating conditions, including vibration and movement. Obtain dynamic blurred images with varying degrees of blur and corresponding clear images. Store the raw video data, filter valid frames of the raw video, and save them as image files to obtain a dataset.

[0016] S1.2: Determine the annotation tools and develop annotation specifications to annotate the location, status, fuzziness, and type of the bolts in the dataset. Select a portion of the dataset as a training set, a portion as a validation set, and another portion as a test set.

[0017] S1.3: To further expand the dataset, perform data augmentation on the images in the training set and some validation sets using geometric transformations such as rotation, flipping, and scaling to generate more diverse image samples and update the corresponding annotation information.

[0018] In S1.1, when the crane is stationary and the camera is stable, the bolts at the same position are photographed to obtain corresponding clear images;

[0019] After the shooting is completed, the original video is frame extracted and the extracted valid frames are saved as high-quality image files.

[0020] The S2 is specifically:

[0021] S2.1: Generation of training dataset for the cue learning module: The ImageNet-C method is used to generate motion blur and low-light effects, and the Real-ESRGAN two-stage degradation processing method is used to generate degraded images with Gaussian blur, noise, and JPEG artifacts. In each batch of images, one part is randomly motion blurred and the contrast is adjusted, while the other part is added with Gaussian blur, noise, and JPEG artifacts. This setting ensures that the generated pre-training dataset covers a wealth of image degradation features, which helps the cue learning module to fully learn the feature representations of different degraded images during the pre-training stage.

[0022] S2.2: Design the prompt learning module: The prompt learning module contains an encoder and a momentum encoder. A classifier and a softmax layer are added after the encoder to complete the specific task.

[0023] The encoder is used to obtain feature information from the degraded image, and the momentum encoder is used to obtain feature information from the clean image to generate a purer feature representation that is closer to the true distribution as the learning goal of the encoder (student);

[0024] The classifier and Softmax layer are used to perform classification tasks and assist the model in learning semantic information;

[0025] In order to enable the hint learning module to learn more comprehensive feature representations, we use the following three methods to conduct phased and progressive training:

[0026] S2.3: The unsupervised pre-training method uses the dataset from S2.1, which has been processed by ImageNet-C and Real-ESRGAN. The encoder extracts features from the degraded input image to obtain a feature representation. The InfoNCE loss function is used to calculate the loss, as follows:

[0027]

[0028] where z i and z j is the feature representation of the positive sample pair, z k is the feature representation of negative samples, sim(.;.) represents the similarity vector between features, τ is the temperature parameter, and N is the number of negative samples. By minimizing this loss, the model is encouraged to bring the feature representations of positive sample pairs closer and push the feature representations of negative sample pairs further apart.

[0029] The training settings are batch size 32, crop resize 224, and total iterations 200K.

[0030] S2.4: Introduction of the pre-task (classification): After the encoder extracts the degraded image features from S2.1, the classifier and the Softmax layer map the features to a specific category probability distribution. In this method, the cross-entropy loss function (Cross-Entropy Loss) is used to calculate the loss, which is:

[0031]

[0032] Where C is the number of categories, y c is the probability corresponding to the cth category of the true label (usually 0 or 1), p c is the probability corresponding to the cth category predicted by the model; for a batch of samples, the cross entropy loss is the average of the cross entropy losses of all samples; by minimizing this loss, the model can learn the semantic information of the image and improve the accuracy of classification.

[0033] S2.5: Knowledge distillation method: Momentum encoder is used as the teacher network, and ordinary encoder is used as the student network. The momentum encoder inputs a clean image, and the ordinary encoder inputs a degraded image. A classifier and a softmax layer are added after the momentum encoder to obtain the output probability distribution of the teacher network. The ordinary encoder passes through the classifier and softmax layer to obtain the output probability distribution of the student network. In this method, KL divergence (Kullback-Leibler Divergence) is used as the loss function, which is:

[0034]

[0035] Where P(i) is the probability distribution of the i-th category output by the teacher network, Q(i) is the probability distribution of the i-th category output by the student network; by minimizing L KL , allowing the student network to learn the knowledge of the teacher network and improve its own semantic information extraction ability.

[0036] S2.1 to S2.5 form a progressive support relationship of "data generation → module design → phased training," which together contribute to the optimization of the prompt learning module. S2.1 provides basic data support for prompt learning training. The generated degraded images are directly used for unsupervised pre-training in S2.3 as encoder input; clean images (original ImageNet-1K) are used in the knowledge distillation stage of S2.5 as input to the momentum encoder.

[0037] Unsupervised pre-training (stage one) enables the cue learning module to learn feature representations for different degraded images. A pre-classification task (stage two) enhances the cue learning module's ability to extract semantic information. Knowledge distillation (stage three) uses a momentum encoder as the teacher network and a standard encoder as the student network to refine features. After three stages of progressive training, the resulting cue learning model combines the training results of all previous stages, resulting in a robust capability for processing degraded images and extracting semantic information.

[0038] The overall process of the S3 image restoration network for de-motion blurring based on prompt learning is as follows:

[0039] First, a dynamic blurred crane beam bolt image I∈R H×W×3 Apply a convolutional layer to extract the shallow feature map F0∈R H×W×C ;

[0040] Then, the shallow feature F0 passes through the 5-level encoder-decoder to generate the deep feature F D ∈R H×W×C For feature downsampling and upsampling, pixel unshuffling and pixel shuffling operations are used; skip connections are added to assist in recovery to bridge continuous intermediate features; a feature fusion module is added to integrate multi-scale information; hint information is embedded in the deblurring network framework through a hint learning module, and acts on the encoder, U-Net bottom, and decoder stages respectively;

[0041] In the encoding stage, the multi-scale feature feas generated by the hint learning module guide the extraction of encoding features layer by layer through the feature fusion module. At the bottom of the U-Net, the global semantic information q generated by the hint learning module is linearly transformed and modulated channel by channel with the bottom features to enhance the global consistency of the recovery.

[0042] In each simplified prompt transformer (SPT) module of the decoder, the prompt information is fused with the decoded features;

[0043] Finally, the deep feature F D Apply the convolution operation to generate a residual image R∈R H×W×3 , add it to the degraded image to get the restored image:

[0044] The proposed network is optimized using the Peak Signal-to-Noise Ratio (PSNR) loss, which is formulated as:

[0045]

[0046] in, represents the real image, ∈ is a minimum value used to avoid the denominator being zero.

[0047] Specifically:

[0048] First, a dynamic blurred crane beam bolt image is used with a 3×3 convolutional layer to extract the shallow feature map F0∈R H ×W×C , H, W, C are the height, width and number of channels of the feature map respectively. Then, the shallow feature F0 is processed through a 5-level encoder-decoder to generate a deep feature F D ∈R H×W×C ;

[0049] The convolution part mainly includes layer normalization (LN), convolution, simple gate (SG) and simplified channel attention (SCA). The process steps of obtaining output Y from input X are as follows:

[0050]

[0051] SG=X f1 ·X f2

[0052]

[0053] in, represents 1×1 convolution, represents 3×3 depth convolution, GAP represents global average pooling, By adding X f0 It is obtained by dividing the features into two parts along the channel dimension and multiplying them, replacing the nonlinear activation function to reduce the computational complexity and selectively determine which frequency information should be retained to restore the potential clear image;

[0054] The Transformer part mainly consists of a multi-head reordered attention with prompts and a simple gated feedforward network. Given the input feature X of the (l-1)th module l-1 , the calculation of a Transformer module is expressed as:

[0055]

[0056] Among them, LN represents layer normalization; and X l They represent the outputs of Multi-Head Re-ranked Attention (MRAP) and Simple Gated Feedforward Network (SGFN), respectively.

[0057] The implementation of Multi-Head Reordered Attention MRAP is as follows: for a given input feature F with a spatial resolution of H×W and the number of channels C, we first apply 1×1 convolution and 3×3 depthwise convolution to aggregate channel context information, obtaining: Next, we will query, key and value matrix Q∈R H×W×C , K∈R H×W×C , V∈R H×W×C Rearrange to Where h is the number of heads;

[0058] The attention matrix is ​​calculated through the self-attention mechanism as follows:

[0059]

[0060] Here, β is a learnable scaling parameter that adjusts the and The size of the dot product, and finally, the attention matrix is ​​rearranged back to its original dimension R∈R H×W×C , and apply 1×1 convolution to generate output features After the above process, the computational complexity is reduced from quadratic complexity to SA Becomes linear complexity O MRAP Calculated as:

[0061] O MRAP =5HWC 2 +HWC

[0062] The specific implementation of the simple gated feedforward network SGFN is: for a given MRAP (multi-head reordered attention) output tensor T∈R H×W×C , the proposed SGFN is expressed as:

[0063]

[0064] Where SG(.) represents the simple gate defined in formula (4);

[0065] Feature Fusion Module (FFM): A feature fusion module (FFM) is introduced to fuse the information of the three-level and four-level hierarchical position transformation Transformer encoders. After upsampling the output feature map E4 of the four-level Transformer encoder, it is seamlessly integrated with the output feature map E4 of the three-level Transformer encoder by element-by-element addition. Subsequently, the resulting combined feature map is input into the feature fusion module FF to generate the final fused feature map F o , the entire feature fusion process is expressed as:

[0066] F o =FF(Up(E4)⊕E3)

[0067] The fusion process of the cue information and the deblurring network is as follows: the cue learning module, as an independent module, generates cue information (q and feas) and introduces it into different parts of the U-Net;

[0068] Among them, q (global prompt information) is used to guide the global features of the deblurring network; feas (multi-scale feature information) is used for local feature guidance to help the deblurring network supplement information at different levels.

[0069] The global information q output by the prompt learning module affects the bottom layer of the U-net network; the local information feas affects the output of the U-net encoder; through skip connections, the features guided by the information output by the prompt module are further used for recovery;

[0070] The prompt information at the bottom of the Unet structure is fused with the output features of the deblurring network: bottom*q is weighted channel by channel (channel dimension multiplication), which is equivalent to modulating the global information of the bottom. Through torch.cat([bottom, bottom*q], dim=1), the original bottom and bottom*q are concatenated to enrich the feature information. Then, 1x1 convolution fusion is performed to further adjust the features of the bottom layer so that the bottom maintains an appropriate number of channels.

[0071] Local information feas affects the output of the U-net encoder: feas is a list containing fea0, fea1, and fea2, which correspond to features of different layers and participate in the jump connection of the encoder (left side of U-Net); torch.cat([x,fea],dim=1) is used to directly concatenate the input feature x and the fea provided by the prompt learning module to expand the information along the channel dimension; 1×1 convolution is used for channel dimensionality reduction so that the downsampling part still maintains the input image size dim dimension, and encs[0], encs[1], encs[2] are used to record these fused features;

[0072] In the decoder stage, encs[2] is fused with bottom upsampling step by step, encs[1] is fused with the output of upsampling module 3, and encs[0] is fused with the output of upsampling module 2 to form a complete U-Net structure.

[0073] The de-motion blurring model trained by S3 was tested. The mean square error (MSE), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators were used to quantitatively evaluate the images after de-motion blurring. The performance of the de-motion blurring model was analyzed based on the evaluation results to determine whether the model meets the actual needs of crane beam bolt detection.

[0074] Specifically:

[0075] S4.1: Test the model trained in S3 using the test set from S1, and make a judgment by calculating the PSNR and SSIM values ​​between the model output image and the GT clear image;

[0076] The peak signal-to-noise ratio index of the blurred image and the restored clear image is compared, as well as the structural similarity index of the blurred image (input), the restored clear image (output) and the clear image (gt) are compared to evaluate the model processing results.

[0077] Beneficial effects of the present invention:

[0078] To address the complex, blurry images found in real-world industrial environments, this paper utilizes the SFGN (Simple Gated Feedforward Network) in the encoder-decoder to split the input features into two parts. A gating mechanism selectively retains the frequency information needed to recover the underlying clear image, thereby mitigating the impact of implicit noise. Furthermore, the MRAP (Multi-Head Rearranged Attention) method is employed to perform attention calculations on the feature dimension rather than the spatial dimension. This adjustment ensures that the attention calculations maintain linear complexity based on the channel dimension. Furthermore, by integrating the hint learning information with the encoder image features at each level, the model effectively captures degraded detail information.

[0079] The present invention utilizes the mechanism of prompt learning. During the process of backpropagation of the model to update the parameters, since the weights of the original model are frozen, only the prompt information parameters are adjusted. However, these parameters are usually relatively small in number, and the gradient descent update speed is faster, which greatly reduces the amount of calculation and reduces the demand for computing resources (such as GPU memory, computing time, etc.), so that it can be deployed more efficiently on edge devices. In addition, by utilizing the prompt learning information, the visual prompts can be customized according to the dynamic blur characteristics of the crane beam bolt image, thereby more accurately guiding the model to perform image restoration. The prompt learning module is improved into a powerful image restoration prompter using unsupervised pre-training, pre-tasks and knowledge distillation technology, which solves the model collapse problem and the training instability problem caused by model limitations. In addition, the redundant operation of manually selecting a specific number of iterations to freeze the weights of different models and tasks in prompt learning is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flow chart of the motion deblurring method of the present invention.

[0081] Figure 2 The present invention prompts a learning and training flow chart.

[0082] Figure 3 This is a structural diagram of the learning module encoder and momentum encoder provided by the present invention.

[0083] Figure 4 This is the prompt-based de-motion blur restoration network structure of the present invention.

[0084] Figure 5 This is the structural diagram of the convolution module of the present invention.

[0085] Figure 6 This is the structural diagram of the Transformer module of the present invention. DETAILED DESCRIPTION

[0086] The present invention will be described in further detail below with reference to the accompanying drawings.

[0087] Reference Figure 1 , the specific steps of the crane beam bolt detection and de-dynamic blur image restoration method based on prompt learning of the present invention are described in detail.

[0088] S1.1: Use an industrial camera to photograph the bolts on the crane beam while the crane is operating, lifting heavy objects, or braking. Photograph the bolts from multiple angles (such as the front, side, and diagonally above, ensuring coverage of all visible surfaces). Because the crane beam vibrates and moves during operation, the captured images will exhibit varying degrees of motion blur. To obtain a clearer image, photograph the bolts in the same position while the crane is stationary and the camera is stable.

[0089] During filming, the raw video data is stored in a large-capacity storage device, with the frame rate set to a value that can clearly capture the dynamic changes of the bolt (e.g., 25 fps or higher). After filming, the raw video is frame-extracted and the extracted valid frames are saved as high-quality image files (e.g., JPEG format, with a resolution of 1920×1080 to ensure clear image details).

[0090] S1.2: Use the professional image annotation tool LabelImg for annotation. Develop detailed annotation specifications: For bolt positions, use rectangular boxes to accurately mark the bolt outlines; bolt status is divided into categories such as normal, loose, and missing, and distinguished by different colors or annotation symbols; the degree of blur is divided into mild, moderate, and severe, determined based on human observation and image clarity assessment standards; blur types are marked as periodic blur, random blur, etc. Divide the dataset into 70% training set, 15% validation set, and 15% test set. When dividing, use random sampling to ensure that each subset contains images of different working conditions, different blur levels, and types to avoid uneven data distribution affecting the model training effect.

[0091] S1.3: Using the Python OpenCV library for image processing, we performed geometric transformations on the images in the training set and some validation sets, including rotation (the rotation angle range was set to -30° to 30° to simulate deviations from different shooting angles), flipping (horizontally and vertically), and scaling (scaling ratios between 0.8 and 1.2 to ensure image feature integrity and some variation). After each transformation, we generated new image samples, and updated the annotations for these new samples based on the original annotations and the transformation rules.

[0092] S2.1: The training dataset of the hint learning module is mainly built based on ImageNet-1K. The ImageNet-C method is used to generate motion blur and low-light effects, and the Real-ESRGAN two-stage degradation method is used to generate Gaussian blur, noise and JPEG artifacts.

[0093] In each batch of images, 40% will randomly add motion blur and adjust the contrast, and 60% will add Gaussian blur, noise and JPEG artifacts. The training data generated in this way contains rich and diverse degradation features, providing a wide range of samples for model learning and helping the model adapt to various real image degradation situations.

[0094] S2.2: Model Component Initialization: Build the cue learning model architecture, including the encoder and momentum encoder, followed by a classifier and softmax layer. These components perform different functions during training: the encoder extracts features from degraded images, the momentum encoder generates higher-quality visual cues based on clean images, and the classifier and softmax layers perform classification and assist the model in learning semantic information.

[0095] Reference Figure 2 , the specific steps of the present invention are described in detail.

[0096] S2.3: Forward propagation process:

[0097] (1) Encoder Processing Degraded Images: The degraded image is input to the encoder, which extracts features from the image through multiple convolutional layers, average pooling layers, and MLP layers. These features contain information such as the image's structure and texture, but are also affected by the degradation of the input image. The features output by the encoder undergo subsequent processing and are ultimately converted into visual cues used to guide the general restoration network.

[0098] (2) Momentum encoder processes clean images: Clean images are input to the momentum encoder. Since the input is a clean image that has not been degraded, the momentum encoder can output cleaner and more representative visual cues. This output will serve as the output of the teacher network to guide the encoder learning in the subsequent knowledge distillation process.

[0099] (3) Classifier and Softmax layer operation: The features output by the encoder and momentum encoder are input into their respective classifiers. The classifiers further process these features and then convert the output into a probability distribution through the Softmax layer to obtain the prediction result of the image category.

[0100] S2.4: Loss function calculation:

[0101] (1) Content Loss: Calculates the difference in content between the encoder output features and the clean image features output by the momentum encoder. It focuses on the semantic content of the image, such as the shape, structure, or contour of the object. It is usually measured using the mean square error (MSE), which is:

[0102]

[0103] Where N is the number of feature dimensions, is the nth dimension of the encoder output feature, is the nth dimension of the clean image feature. The content loss pushes the features generated by the encoder to be closer to the features of the clean image in terms of content, thereby improving the quality of the visual cues.

[0104] (2) Style Loss: Style loss is used to measure the difference in style between the encoder output features and the clean image features, focusing on the style features of the image, such as texture and color distribution. First, the Gram matrix of the feature map is calculated to reflect the correlation between different channels of the feature map, representing the image style information, and then the Gram matrix of the encoder output features and the clean image features is compared to calculate. In actual operation, for the feature map F output by the encoder, encoder and the feature map F of the clean image clean , calculate their Gram matrices G respectively encoder and G clean The size of the feature map is C×H×W (C is the number of channels, H is the height, and W is the width). The element G of the Gram matrix G is ij The calculation method is:

[0105]

[0106] Among them F i and F jare the i-th and j-th channels of the feature map, respectively. After calculating the two Gram matrices, the style loss is calculated by comparing the difference between them. The commonly used calculation method is the mean square error (MSE), that is:

[0107]

[0108] This allows us to accurately measure the stylistic differences between the encoder output features and the clean image features, guiding the model to generate visual cues that are stylistically closer to the clean images.

[0109] (3) KL Divergence Loss: In the knowledge distillation process, the momentum encoder is regarded as the teacher network and the encoder is regarded as the student network. KL divergence is used to measure the difference between the outputs of the two. The calculation object is the output probability distribution of the encoder and momentum encoder after passing through the classifier and Softmax layer. The calculation formula is:

[0110]

[0111] where p teacher (i) is the probability distribution of the momentum encoder’s prediction for the i-th category, p student (i) is the probability distribution of the encoder’s prediction for category i. The KL divergence loss guides the encoder to learn better feature representations for the momentum encoder, improving the encoder’s ability to extract semantic information.

[0112] (4) InfoNCE Loss: The hint learning module is based on the contrastive learning framework MoCo and introduces a queue mechanism. The queue stores features from past training and provides negative samples for the current training sample. InfoNCE loss is calculated by using the negative samples in the queue to enhance the contrastive learning effect. Assume that the feature of the current sample is represented by z and the feature of the positive sample is represented by {z +}, the negative sample feature representation set is {z -}, the InfoNCE loss calculation formula is:

[0113]

[0114] Where sim represents the similarity calculation function (cosine similarity), and τ is the temperature hyperparameter used to adjust the distribution of similarity. InfoNCE loss enables the encoder to learn more discriminative features, improving model performance.

[0115] (5) Cross Entropy Loss: The calculation object is the output of the encoder after the classifier and Softmax layer and the ImageNet label. By comparing the difference between the two, the cross entropy loss measures the performance of the encoder on the classification task. The calculation formula is:

[0116]

[0117] where y i is the probability distribution of the true label (in ImageNet labels, the corresponding category is 1 and the others are 0), p i is the probability distribution predicted by the model. This loss encourages the encoder to learn more discriminative features and enhance its understanding of image semantics so as to better cope with image restoration tasks under different degradation conditions.

[0118] (6) Total loss calculation: The above cross entropy loss, content loss, style loss, KL divergence loss and InfoNCE loss are weighted and summed to obtain the total loss (L total ), the formula is:

[0119] L total =L style +L content +L InfoNCE +L CrossEntropy +L KL

[0120] The total loss is used to measure the difference between the model's current prediction results and the actual situation, and serves as the basis for model optimization.

[0121] S2.4: Backpropagation and parameter update: Based on the calculated total loss, the gradient is calculated through the backpropagation algorithm. Backpropagation starts from the total loss and backpropagates the gradient along the network's computational graph to calculate the gradient value of each parameter. The optimizer uses AdamW (β1 = 0.9, β2 = 0.999, weight decay = 1e -4 ), update the encoder, momentum encoder, and parameters of the related classifier and Softmax layer according to the calculated gradient. During the training process, the initial learning rate is set to 3e -4 And use the Cosine annealing strategy to adjust the learning rate. As the training progresses, the learning rate gradually decreases, making the model converge more stably in the later stage of training and avoiding overtraining.

[0122] S2.5: Model Training Iterations: Repeat the forward propagation, loss calculation, and backpropagation parameter updates described above for multiple rounds of training. With each round, the model gradually learns the feature representations of different degraded images, continuously optimizing its ability to generate visual cues. Training settings are batch size 32, resize 224, and total iterations 200K.

[0123] S2.6: Model Saving and Application: After training is complete, only the encoder portion of the learned structure is retained. The trained encoder is combined with the deblurring network. The encoder generates clean visual cues based on the input degraded image, guiding the general restoration network to restore the degraded image, achieving high-quality image reconstruction.

[0124] Reference Figure 4 , the specific steps of the present invention are described in detail.

[0125] The overall process of the image restoration network for de-motion blurring based on prompt learning is as follows: First, a de-motion blurring image of a crane beam bolt I∈R H×W×3 Apply a 3×3 convolutional layer to extract the shallow feature map F0∈R H×W×C (H, W, C are the height, width and number of channels of the feature map respectively). Then, these shallow features F0 are passed through a 5-level encoder-decoder to generate deep features F D ∈R H×W×C . For the downsampling and upsampling of features, pixel-unshuffle and pixel-shuffle operations are used. To assist the recovery process, jump connections are added to bridge continuous intermediate features. At the same time, a feature fusion module (FFM) is added to integrate multi-scale information. The prompt information is embedded in the de-motion blurring network framework through the prompt learning module, and acts on the encoder, U-Net bottom and decoder stages respectively. In the encoding stage, the multi-scale feature feas generated by the prompt learning module guides the extraction of encoding features layer by layer through the feature fusion module (DC) to enhance the modeling ability of dynamic blur features. At the bottom of U-Net, the global semantic information q generated by the prompt learning module is linearly transformed and modulated channel by channel with the bottom features to enhance the global consistency of recovery. In each simplified prompt transformer (SPT) module of the decoder, the prompt information is fused with the decoded features to enhance the input feature's perception of the degradation type and further guide the recovery process. Finally, the deep feature F D Apply the convolution operation to generate a residual image R∈R H×W×3 , add it to the degraded image to get the restored image: The proposed network is optimized using the Peak Signal-to-Noise Ratio (PSNR) loss, which is formulated as:

[0126]

[0127] in, represents the real image, ∈ is a minimum value used to avoid the denominator being zero.

[0128] The specific implementation of each step of the proposed prompt learning-based crane beam bolt de-fuzzification network is as follows:

[0129] First, a dynamic blurred crane beam bolt image is used with a 3×3 convolutional layer to extract the shallow feature map F0∈R H ×W×C (H, W, C are the height, width and number of channels of the feature map respectively). Then, these shallow features F0 are passed through a 5-level encoder-decoder to generate deep features F D ∈R H×W×C .

[0130] Reference Figure 5 , the specific steps of the present invention are described in detail.

[0131] The convolution part mainly includes layer normalization (LN), convolution, simple gate (SG) and simplified channel attention (SCA). The main steps of the process of obtaining output Y from input X are:

[0132]

[0133] SG=X f1 ·X f2

[0134]

[0135] in, represents 1×1 convolution, represents 3×3 depth convolution, GAP represents global average pooling, By adding X f0 By using a simple gating mechanism (SG) to replace GELU and SCA to replace CA, we can split the features into two parts along the channel dimension and multiply them, replacing the nonlinear activation function to reduce the computational complexity and selectively determine which frequency information should be retained to restore the potential clear image.

[0136] Reference Figure 6 , the specific steps of the present invention are described in detail.

[0137] The Transformer part mainly consists of a multi-head reordered attention with prompts and a simple gated feedforward network. Given the input feature X of the (l-1)th modulel-1 , the calculation of a Transformer module is expressed as:

[0138]

[0139] Among them, LN represents layer normalization; and X l They represent the outputs of Multi-Head Re-ranked Attention (MRAP) and Simple Gated Feedforward Network (SGFN), respectively.

[0140] The implementation of Multi-Head Reordered Attention MRAP is as follows: for a given input feature F with a spatial resolution of H×W and the number of channels C, we first apply 1×1 convolution and 3×3 depthwise convolution to aggregate channel context information, obtaining: Next, we will query, key and value matrix Q∈R H×W×C , K∈R H×W×C , V∈R H×W×C Rearrange to where h is the number of heads.

[0141] Since SA is applied on the channel dimension rather than the spatial dimension to reduce computational complexity, the attention matrix is ​​calculated through the self-attention mechanism as follows:

[0142]

[0143] Here, β is a learnable scaling parameter that adjusts the and Finally, we reshape the attention matrix back to its original dimension R∈R H×W×C , and apply 1×1 convolution to generate output features After the above process, the computational complexity is reduced from quadratic complexity to SA Becomes linear complexity O MRAP Calculated as:

[0144] O MRAP =5HWC 2 +HWC

[0145] The specific implementation of the simple gated feedforward network SGFN is: for a given MRAP (multi-head reordered attention) output tensor T∈R H×W×C , the proposed SGFN can be expressed as:

[0146]

[0147] where SG(.) represents the simple gate defined in Equation (4). In this way, the implicit noise response in the MRAP output result T becomes smaller, thereby weakening the impact of implicit noise.

[0148] Feature Fusion Module (FFM): A feature fusion module (FFM) is introduced to fuse the information of the three-level and four-level hierarchical position transformation transformer encoders. After upsampling the output feature map E4 of the four-level transformer encoder, we seamlessly integrate it with the output feature map E4 of the three-level transformer encoder by element-wise addition. The resulting combined feature map is then input into the feature fusion module FF to generate the final fused feature map F o , the entire feature fusion process can be expressed as:

[0149] F o =FF(Up(E4)⊕E3)

[0150] The fusion process of the hint information and the deblurring network is as follows: the hint learning module acts as an independent module to generate hint information (q and feas) and introduce it into different parts of the U-Net to improve the image restoration quality.

[0151] Among them, q (global prompt information) is used to guide the global features of the deblurring network; feas (multi-scale feature information) is used for local feature guidance to help the deblurring network supplement information at different levels.

[0152] The global information q output by the prompt learning module affects the bottom layer of the U-net network; the local information feas affects the output part of the U-net encoder; through skip connections, the features guided by the information output by the prompt module are continued to be used for recovery.

[0153] The prompt information at the bottom of the Unet structure is fused with the output features of the de-motion blurring network: bottom*q is weighted channel by channel (channel dimension multiplication), which is equivalent to modulating the global information of the bottom, highlighting the key information provided by the prompt learning module. Through torch.cat([bottom, bottom*q], dim=1), the original bottom and bottom*q are spliced ​​to enrich the feature information. Then, 1x1 convolution fusion is performed to further adjust the features of the bottom layer so that the bottom maintains an appropriate number of channels.

[0154] Local information feas influences the output of the U-net encoder: feas is a list containing fea0, fea1, and fea2, corresponding to features at different layers, participating in the jump connection of the encoder (left side of the U-net). Use torch.cat([x,fea],dim=1) to directly concatenate the input feature x and the fea provided by the hint learning module, expanding the information along the channel dimension. Use 1×1 convolution for channel dimensionality reduction, so that the downsampled part still maintains the input image size dim dimension. At the same time, use encs[0], encs[1], and encs[2] to record these fused features.

[0155] In the decoder stage, encs[2] is fused with the bottom upsampling step by step, encs[1] is fused with the output of upsampling module 3, and encs[0] is fused with the output of upsampling module 2 to form a complete U-Net structure. In this way, by prompting learning to guide the recovery process, the network can accurately remove motion blur and improve the accuracy and robustness of bolt detection.

[0156] The training dataset is extracted from the crane beam bolt dynamic fuzzy dataset in step S1. The Adam optimizer (β1 = 0.9, β2 = 0.999) and peak signal-to-noise ratio (PSNR) loss are used to train the model. The training dataset is 5×10 5 iterations, with an initial learning rate of 5×10 -4 and gradually reduced to 1×10 through the cosine annealing strategy -7 .

[0157] S3.1: We constructed an image motion blur restoration network based on a combination of CNN and Transformer. The network adopts the Unet structure and includes an encoder and a decoder.

[0158] Both the decoder and encoder contain convolutional modules and Transformer modules to capture long-distance dependent non-local information and model local invariant features. The convolutional module includes layer normalization (LN), 1x1 Conv, 3x3 Conv, downsampling layer (down), simple gating structure (SG), spatial channel attention mechanism (SCA) and global average pooling (GAP). Its purpose is to stabilize the input distribution through LN, use 1x1 Conv to fuse channel information, 3x3 Conv to extract local spatial features, the downsampling layer to expand the receptive field, SG to selectively retain effective features, SCA combined with GAP to optimize features from the spatial and channel dimensions, and each component forms a pipeline through residual connection to achieve multi-scale local feature extraction and optimization.

[0159] The Transformer's Multi-Head Rearranged Attention (MRAP) mechanism first aggregates local context through convolution, then calculates attention in the channel dimension to reduce complexity. It also injects cues to guide the model in capturing degradation types and modeling global dependencies. The Simple Gated Feedforward Network (SGFN) uses a gating mechanism to suppress noise and refine features. The two collaborate to achieve "global modeling and local optimization." The Multi-Scale Feature Fusion Module (FFM) fuses features from different encoder levels, aggregating multi-scale information through upsampling and element-wise addition to compensate for the shortcomings of a single scale.

[0160] Overall, the convolutional module processes local details, the Transformer module is responsible for global semantics and degradation adaptation, and the FFM connects the two in series to achieve multi-scale feature complementarity, forming an efficient end-to-end restoration framework.

[0161] S3.2: Embed the prompt learning model trained in S2 into the deblurring and restoration network of S3.1, and use the prompt information to guide the model to restore the blurred image. Use the actual industrial plant crane beam bolt dynamic blur dataset extracted in S1 as the input of the deblurring and restoration network embedded with prompt learning for training, and adjust the parameters to obtain the optimal deblurring and restoration model.

[0162] S4.1: In this invention, the performance of the model trained in stage S3 is comprehensively tested using the test set constructed in stage S1. Specifically, the model is fed into a test dataset of blurred images, and the resulting restored, clear image (output) is obtained. To accurately evaluate the model's restoration performance, two key metrics widely recognized in the field of image quality assessment, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), are used. The PSNR and SSIM values ​​are calculated between the model output image (output) and the corresponding GT clear image in the test set.

[0163] The Peak Signal-to-Noise Ratio (PSNR) metric provides a direct reflection of the model's ability to remove blur and restore image details. A higher PSNR value indicates a smaller error between the restored and original clear images, meaning the model's restored image quality is higher and closer to the true clear image. The calculation formula is as follows:

[0164]

[0165] Among them, MAX I Indicates the maximum possible value of the image pixel value (for 8-bit images, MAX I =255), the RMSE formula is:

[0166]

[0167] Here, I ij Represents the pixel value of the coordinate (i, j) in the GT clear image, K ij Represents the pixel value of the corresponding coordinate in the restored clear image output by the model, where m and n are the height and width of the image respectively.

[0168] The Structural Similarity (SSIM) metric evaluates the similarity between images based on brightness, contrast, and structure. Compared to traditional metrics based on pixel error, it better reflects the human eye's perception of image quality. The closer the SSIM value is to 1, the more similar the model output image is to the GT clear image in structure, and the better the model's restoration effect. The calculation formula for SSIM is relatively complex, involving parameters such as mean, variance, and covariance, as follows:

[0169]

[0170] Among them, X and Y represent two images to be compared, μ X and μ Y is the mean of images X and Y, and is the variance of images X and Y, σ XY is the covariance of image X and Y. C1 and C2 are constants introduced to avoid zero denominators. Based on the evaluation results, analyze the performance of the deblurring model to determine whether it meets the actual requirements for crane beam bolt detection.

Claims

1. A method for de-dynamic blurring image restoration for crane beam bolt detection based on prompt learning, characterized in that: The following steps are included: S1: Use industrial cameras to collect images of crane beam bolts under different working conditions to form a crane beam bolt image dataset; obtain images with different degrees of dynamic blur and corresponding clear images; S2: We use the ImageNet-C and Real-ESRGAN methods to degrade the ImageNet-1K dataset to obtain degraded images of various types. We then design and train a hint learning module based on the characteristics of these degraded images, enabling it to generate effective hints containing information related to various image degradations. S3: Embed the prompt learning module trained in S2 into it, build a motion deblurring model based on prompt learning, and use the crane beam bolt image dataset built in S1 to train the motion deblurring model; let the motion deblurring model learn to use prompt information to remove image motion blur, continuously optimize model performance, and obtain the restored image.

2. The method for de-dynamic blurring image restoration for crane beam bolt detection based on prompt learning according to claim 1, characterized in that: The specific steps of S1 are: S1.1: Use an industrial camera to capture the bolts of the crane beam from different angles under various operating conditions, including vibration and movement. Obtain dynamic blurred images with varying degrees of blur and corresponding clear images. Store the raw video data, filter valid frames of the raw video, and save them as image files to obtain a dataset. S1.2: Determine the annotation tools and develop annotation specifications to annotate the location, status, fuzziness, and type of the bolts in the dataset. Select a portion of the dataset as a training set, a portion as a validation set, and another portion as a test set. S1.3: Use geometric transformations such as rotation, flipping, and scaling to perform data augmentation on the images in the training set and some validation sets to generate more diverse image samples and update the corresponding annotation information.

3. The method for de-dynamic blurring image restoration for crane beam bolt detection based on prompt learning according to claim 2, characterized in that: In S1.1, when the crane is stationary and the camera is stable, the bolts at the same position are photographed to obtain corresponding clear images; After the shooting is completed, the original video is frame extracted and the extracted valid frames are saved as high-quality image files.

4. The method for de-dynamic blurring image restoration for crane beam bolt detection based on prompt learning according to claim 1, characterized in that: The S2 is specifically: S2.1: Use the ImageNet-C method to generate motion blur and low-light effects, and use the Real-ESRGAN two-stage degradation method to generate degraded images with Gaussian blur, noise, and JPEG artifacts. In each batch of images, some are randomly motion blurred and contrast adjusted, while others are Gaussian blurred, noisy, and JPEG artifacts. S2.2: Design a cue learning module. The cue learning module consists of an encoder and a momentum encoder. A classifier and a softmax layer are added after the encoder to complete a specific task. The encoder is used to obtain feature information from the degraded image, and the momentum encoder is used to obtain feature information from the clean image to generate a purer feature representation that is closer to the true distribution as the learning goal of the encoder; The classifier and Softmax layer are used to perform classification tasks and assist the model in learning semantic information; Go through the following three methods in stages and progressive training: S2.3: The dataset used for the unsupervised pre-training method is the image processed by ImageNet-C and Real-ESRGAN in S2.

1. The encoder extracts features from the input degraded image to obtain feature representation and calculates the loss using the InfoNCE loss function. The formula is: where z i and z j is the feature representation of the positive sample pair, z k is the feature representation of negative samples, sim(.;.) represents the similarity vector between features, τ is the temperature parameter, and N is the number of negative samples. By minimizing this loss, the model is encouraged to bring the feature representations of positive sample pairs closer and push the feature representations of negative sample pairs further apart. S2.4: Introduction of the pre-task: After the encoder extracts the degraded image features from S2.1, the classifier and Softmax layer map the features to a specific category probability distribution; the loss is calculated using the cross entropy loss function, as follows: Where C is the number of categories, y c is the probability corresponding to the cth category of the true label, p c is the probability corresponding to the cth category predicted by the model; for a batch of samples, the cross entropy loss is the average of the cross entropy losses of all samples; S2.5: Knowledge distillation method: The momentum encoder is used as the teacher network and the ordinary encoder is used as the student network. The momentum encoder inputs a clean image and the ordinary encoder inputs a degraded image. A classifier and a softmax layer are added after the momentum encoder to obtain the output probability distribution of the teacher network. The ordinary encoder passes through the classifier and softmax layer to obtain the output probability distribution of the student network. The KL divergence is used as the loss function, which is: Where P(i) is the probability distribution of the i-th category output by the teacher network, Q(i) is the probability distribution of the i-th category output by the student network; by minimizing L KL , allowing the student network to learn the knowledge of the teacher network and improve its own semantic information extraction ability.

5. The method for de-dynamic blurring image restoration based on prompt learning for crane beam bolt detection according to claim 1, characterized in that: The overall process of the S3 image restoration network for de-motion blurring based on prompt learning is as follows: First, a dynamic blurred crane beam bolt image I∈R H×W×3 Apply a convolutional layer to extract the shallow feature map F0∈R H×W×C ; Then, the shallow feature F0 passes through the 5-level encoder-decoder to generate the deep feature F D ∈R H×W×C ; For feature downsampling and upsampling, pixel unshuffling and pixel shuffling operations are used; skip connections are added to assist in recovery to bridge continuous intermediate features; A feature fusion module is added to integrate multi-scale information; the prompt information is embedded in the de-motion blurring network framework through the prompt learning module, and acts on the encoder, U-Net bottom and decoder stages respectively; In the encoding stage, the multi-scale feature feas generated by the hint learning module guide the extraction of encoding features layer by layer through the feature fusion module. At the bottom of the U-Net, the global semantic information q generated by the hint learning module is linearly transformed and modulated channel by channel with the bottom features to enhance the global consistency of the recovery. In each simplified prompt transformer (SPT) module of the decoder, the prompt information is fused with the decoded features; Finally, the deep feature F D Apply the convolution operation to generate a residual image R∈R H×W×3 , add it to the degraded image to get the restored image: The proposed network is optimized using the Peak Signal-to-Noise Ratio (PSNR) loss, which is formulated as: in, represents the real image, ∈ is a minimum value used to avoid the denominator being zero.

6. The method for de-dynamic blurring image restoration based on prompt learning for crane beam bolt detection according to claim 5, characterized in that: Specifically: First, a dynamic blurred crane beam bolt image is used with a convolutional layer to extract the shallow feature map F0∈R H×W×C , H, W, C are the height, width and number of channels of the feature map respectively. Then, the shallow feature F0 is processed through a 5-level encoder-decoder to generate a deep feature F D ∈R H×W×C ; The dynamic blurred crane beam bolt image first passes through the convolution layer, so the convolution layer is used to extract shallow features. The convolution part mainly includes layer normalization (LN), convolution, simple gate and simplified channel attention. The process of obtaining output Y from input X is as follows: SG=X f1 ·X f2 in, represents 1×1 convolution, represents 3×3 depth convolution, GAP represents global average pooling, X f1 、 By adding X f0 Obtained by dividing by channel dimension; SCA is used to divide the features into two parts along the channel dimension and multiply them, selectively determining which frequency information to retain to restore the potential clear image; Transformer consists of a multi-head reordered attention with prompts and a simple gated feedforward network. Given the input feature X of the (l-1)th module l-1 , the calculation of a Transformer module is expressed as: Among them, LN represents layer normalization; and X l Represent the outputs of multi-head reordered attention and simple gated feedforward networks respectively; The implementation of Multi-Head Reordered Attention MRAP is as follows: for a given input feature F with a spatial resolution of H×W and the number of channels C, first apply 1×1 convolution and 3×3 depthwise convolution to aggregate channel context information, obtaining: Next, the query, key, and value matrix Q∈R H×W×C , K∈R H×W×C , V∈R H×W×C Rearrange to Where h is the number of heads; The attention matrix is ​​calculated through the self-attention mechanism as follows: Here, β is a learnable scaling parameter that adjusts the and The size of the dot product, and finally, the attention matrix is ​​rearranged back to its original dimension R∈R H×W×C , and apply 1×1 convolution to generate output features After the above process, the computational complexity is reduced from quadratic complexity to SA Becomes linear complexity O MRAP Calculated as: Oh MRAP =5HWC 2 +HWC The specific implementation of the simple gated feedforward network SGFN is: for a given MRAP output tensor T∈R H×W×C , the proposed SGFN is expressed as: Where SG(.) represents the simple gate defined in the formula; The feature fusion module is used to fuse the information of the three-level and four-level hierarchical position transformation Transformer encoders; after upsampling the output feature map E4 of the four-level Transformer encoder, it is seamlessly integrated with the output feature map E4 of the three-level Transformer encoder by element-by-element addition; then, the resulting combined feature map is input into the feature fusion module FF to generate the final fused feature F o , the entire feature fusion process is expressed as: The fusion process of the cue information and the deblurring network is as follows: the cue learning module, as an independent module, generates cue information q and feas and introduces them into different parts of the U-Net; Among them, q is used to guide the global features of the deblurring network; feas is used for local feature guidance to help the deblurring network supplement information at different levels. The global information q output by the prompt learning module affects the bottom layer of the U-net network; the local information feas affects the output of the U-net encoder; through skip connections, the features guided by the information output by the prompt module are further used for recovery; The prompt information at the bottom of the Unet structure is fused with the output features of the deblurring network: bottom*q is weighted channel by channel, and the original bottom and bottom*q are concatenated through torch.cat([bottom, bottom*q], dim=1) to enrich the feature information. Then, 1x1 convolution fusion is performed to further adjust the features of the bottom layer so that the bottom layer maintains an appropriate number of channels. Local information feas affects the output of the U-net encoder: feas is a list containing fea0, fea1, and fea2, which correspond to features of different layers and participate in the jump connection of the encoder (left side of U-Net); torch.cat([x,fea],dim=1) is used to directly concatenate the input feature x and the fea provided by the prompt learning module to expand the information along the channel dimension; 1×1 convolution is used for channel dimensionality reduction so that the downsampling part still maintains the input image size dim dimension, and encs[0], encs[1], encs[2] are used to record these fused features; In the decoder stage, encs[2] is fused with bottom upsampling step by step, encs[1] is fused with the output of upsampling module 3, and encs[0] is fused with the output of upsampling module 2 to form a complete U-Net structure.

7. The method for de-dynamic blurring image restoration of crane beam bolt detection based on prompt learning according to claim 1, characterized in that: The de-motion blurring model trained by S3 was tested. The mean square error, peak signal-to-noise ratio and structural similarity indicators were used to quantitatively evaluate the images after de-motion blurring. The performance of the de-motion blurring model was analyzed based on the evaluation results to determine whether the model meets the actual needs of crane beam bolt detection.

8. The method for de-motion blurring image restoration for crane beam bolt detection based on prompt learning according to claim 7, characterized in that: Specifically: S4.1: Test the model trained in S3 using the test set from S1, and make a judgment by calculating the PSNR and SSIM values ​​between the model output image and the GT clear image; The peak signal-to-noise ratio index of the blurred image and the restored clear image is compared, and the structural similarity index of the blurred image, the restored clear image and the clear image (gt) is compared to evaluate the model processing results.

Images