Degraded image defect detection method and system cooperating with text prompt and visual restoration

Abstract

The invention discloses a degraded image defect detection method and system cooperating with text prompt and visual restoration. The method comprises the following steps: generating a degraded image through a high-order degraded model; extracting semantic clues related to the detection target from the degraded image, and encoding the semantic clues into text feature vectors; inputting the degraded image into an anti-degraded backbone network of an integrated low-rank adaptive module, and outputting a robust visual feature map; inputting the text feature vector and the robust visual feature map into an encoder, and outputting an enhanced visual feature of a fusion text prompt; a collaborative learning strategy guided by visual restoration is introduced, the enhanced visual features prompted by the fusion text are restored and enhanced, the feature quality is improved, and meanwhile query distribution under the noise condition is improved; and the enhanced visual features prompted by the fusion text are input to a decoder after being processed by a collaborative learning strategy guided by visual restoration, and the position and category information of the defect is output. According to the method, the degraded image can be directly optimized, and the degraded information is effectively extracted and utilized.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and specifically relates to a method and system for detecting degraded image defects by coordinating text prompts and visual restoration. Background Art

[0002] As an important branch of the target detection field, defect detection's core task is to accurately locate various types of defects in images or videos and identify the predefined categories to which they belong. This technology plays a vital role in ensuring product quality, improving production efficiency, and preventing safety accidents. In recent years, with the rapid development of deep learning technology, especially detectors based on the Transformer architecture (such as the DETR model and its variants), target detection and defect detection technologies have made significant progress. The DETR model achieves end-to-end target detection by introducing the Transformer encoder-decoder structure and learnable target queries, avoiding the complex candidate box generation and post-processing steps in traditional detection methods.

[0003] However, while advanced detection models such as DETR, DINO, and Co-DETR perform well on standard high-quality (HQ) image datasets (e.g., ImageNet and COCO), their performance often degrades significantly when used in real-world industrial applications. Images captured in real-world scenarios often exhibit varying degrees of low-quality (LQ) degradation due to factors such as illumination variations, sensor noise, transmission interference, environmental dust, motion blur, and compression artifacts. This degradation in image quality leads to the loss of effective information and the introduction of noise, making it difficult for detection models that rely on learning features from clear images to accurately extract and identify defect features. This results in a significant decrease in detection accuracy and recall, and frequent missed and false detections.

[0004] To address this challenge, an intuitive strategy is a two-stage approach of “enhancement first, then detection”. That is, first, image enhancement or image restoration techniques (such as denoising, super-resolution SR) are used to pre-process the degraded image to improve its visual quality, and then the enhanced image is fed into the target detection model for detection. For example, super-resolution methods such as MSRN, SRGAN, ESRGAN, and Real-ESRGAN reconstruct image details through complex network structures and adversarial learning to improve the perceived quality of the image. However, such methods have inherent limitations: (1) Inconsistent optimization objectives: The optimization objectives of image enhancement / restoration algorithms are usually to minimize the pixel-level difference with high-quality reference images (such as MSE loss) or to improve human visual perception (such as perceptual loss, adversarial loss), which are not completely aligned with the optimization objectives of downstream detection tasks (such as maximizing mean average precision (mAP)) and may even conflict with them. For example, in order to pursue visual effects, some enhancement algorithms may smooth out subtle defect features that are critical to detection, or introduce new artifacts, which in turn interfere with the judgment of the detection model. (2) Error accumulation and suboptimal solutions: Two-stage separation processing may lead to error accumulation. Any imperfections or distortions produced during the image enhancement step will be passed on to the subsequent detection stage, affecting the final detection performance. Since the two stages are optimized independently, this cascade approach is difficult to achieve global optimality. (3) Increased computational overhead: Introducing additional image enhancement modules increases the computational complexity and processing time of the entire system, which is unacceptable for application scenarios with high real-time requirements.

[0005] Some existing end-to-end models that attempt to combine image enhancement with detection tasks have alleviated the above problems to a certain extent, but often fail to fully explore and utilize the semantic information that is still valuable for detection in degraded images, or fail to effectively adjust the model's feature extraction and attention mechanisms based on the degradation characteristics. In addition, the Transformer-based DETR series of detectors usually rely heavily on backbone networks (such as ViT (Vision Transformer)) pre-trained on large-scale high-quality datasets. Since these backbone networks are optimized for extracting features from clear images, when they are directly applied to low-quality images, the domain gap problem may further aggravate the performance degradation. Summary of the Invention

[0006] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a degraded image defect detection method and system that collaborates text prompts and visual restoration. The method can directly optimize degraded images, effectively extract and utilize degradation information, and combine multimodal information (such as text) for guidance to achieve an end-to-end solution for robust and efficient defect detection.

[0007] In order to achieve the above object, the present invention has the following technical solutions:

[0008] In a first aspect, a method for detecting degraded image defects by coordinating text prompts and visual restoration is provided, comprising:

[0009] Generate degraded images containing real-world noise characteristics through a pre-built high-order degradation model;

[0010] Extract semantic clues related to the detection target from the degraded image and encode the semantic clues into text feature vectors;

[0011] The degraded image is input into the degradation-resistant backbone network integrated with low-rank adaptive modules, and the robust visual feature map is output;

[0012] The text feature vector and robust visual feature map are input into the encoder, and the cross-attention mechanism is used to guide and enhance the visual features with text semantic information, and the enhanced visual features that are integrated with the text hints are output;

[0013] A collaborative learning strategy guided by visual restoration is introduced to repair and enhance the enhanced visual features fused with textual hints, recovering key information lost due to quality degradation, improving feature quality, and improving query allocation under noisy conditions.

[0014] The enhanced visual features fused with textual cues are processed by a collaborative learning strategy guided by visual restoration and then input into the decoder. Combined with learnable target queries, defect targets are located and classified, and the location and category information of the defects are output.

[0015] As a preferred solution, in the step of generating a degraded image containing real-world noise characteristics using a pre-built high-order degradation model, the high-order degradation model performs two or more classic degradation operations, each of which includes at least two operations among blur kernel convolution, downsampling of a specific scale factor, noise addition, and JPEG compression;

[0016] The mathematical expression of the classic degradation operation is: Where, represents the blur kernel k convolution, ↓ r represents downsampling with a scale factor of r, n represents additive noise, [·] JPEG Indicates JPEG compression.

[0017] As a preferred solution, in the step of extracting semantic clues related to the detection target from the degraded image and encoding the semantic clues into a text feature vector, the degraded image is input into a degradation perception hint extractor, and the degradation perception hint extractor extracts semantic keywords or phrases related to the defect target to be detected in the degraded image to form an initial text hint; the initial text hint is input into a text encoder, and the text encoder converts the text hint into a text feature vector with a fixed dimension.

[0018] As a preferred solution, in the step of inputting the degraded image into the anti-degradation backbone network integrated with the low-rank adaptive module and outputting the robust visual feature map, the low-rank adaptive module freezes the original model parameters by adding a low-rank trainable matrix in the weight matrix bypass of the anti-degradation backbone network model, thereby fine-tuning and enhancing the feature extraction capability of the degraded image and outputting a preliminary robust visual feature map;

[0019] The specific implementation of the low-rank adaptive module is as follows: for the weight matrix of the selected linear layer or two-dimensional convolutional layer in the pre-trained anti-degradation backbone network model Its update ΔW is expressed as the product of two low-rank matrices BA, ΔW = BA; where, And the rank r<<min(d,k); during training, only the parameters of matrices A and B are updated, while W0 remains unchanged, and the forward propagation of the layer becomes h=W0x+ΔWx=W0x+BAx.

[0020] As a preferred solution, in the step of inputting the text feature vector and the robust visual feature map into the encoder, adopting the cross-attention mechanism to guide and enhance the visual features with the text semantic information, and outputting the enhanced visual features fused with the text prompts, the encoder is a Transformer encoder including a multi-layer self-attention mechanism and a feedforward network;

[0021] In at least one attention layer of the Transformer encoder, a cross attention mechanism is used to take the robust visual feature map as the query Q vis , using the text feature vector as the key K txt Sum V txt , or vice versa;

[0022] The calculation is performed as follows:

[0023]

[0024] Where, d k is the dimension of the key vector.

[0025] As a preferred solution, the collaborative learning strategy guided by visual restoration is introduced to restore and enhance the enhanced visual features fused with textual cues, recover key information lost due to degradation, improve feature quality, and improve query allocation under noisy conditions. The collaborative learning strategy guided by visual restoration is implemented through a multi-scale feature restoration unit and multiple auxiliary detection heads.

[0026] The multi-scale feature restoration unit adopts a cyclic residual dense learning (RRDL) structure to process the multi-scale features output by the encoder, repair and enhance the enhanced visual features of the fused text prompts, restore the key information lost due to degradation, and improve the feature quality. The cyclic residual dense learning (RRDL) structure is composed of a stack of multiple basic residual blocks, each of which contains multiple convolutional layers or linear layers and ReLU activation functions, and applies multi-level residual connections.

[0027] The multiple auxiliary detection heads are introduced in the middle layer of the decoder to supervise features at different levels, provide additional supervision signals, help the decoder learn the query-target matching relationship, and improve query allocation under noisy conditions.

[0028] As a preferred solution, the enhanced visual features of the fused text prompts are input into the decoder after being processed by the collaborative learning strategy guided by visual restoration, and the defect targets are located and classified in combination with the learnable target query. In the step of outputting the location and category information of the defects, the decoder is a Transformer decoder, which receives the enhanced visual features of the fused text prompts after being processed by the collaborative learning strategy guided by visual restoration, and a set of learnable target queries Object Queries; through the self-attention and cross-attention mechanisms, each query is gradually focused on a specific target instance; the prediction head is a feedforward neural network, which is connected to the output of each query of the decoder and is used to predict the category and bounding box coordinates of the target corresponding to the query; the loss function adopts the set prediction loss of the DETR standard, including classification loss and bounding box regression loss.

[0029] In a second aspect, a degraded image defect detection system that collaborates with text prompts and visual restoration is provided, comprising:

[0030] A degraded image generation module is used to generate degraded images containing real-world noise characteristics through a pre-built high-order degradation model;

[0031] Semantic clue encoding module, which is used to extract semantic clues related to the detection target from the degraded image and encode the semantic clues into text feature vectors;

[0032] A robust visual feature map output module is used to input the degraded image into the degradation-resistant backbone network integrated with the low-rank adaptive module and output a robust visual feature map;

[0033] The text and visual feature fusion encoding module is used to input the text feature vector and the robust visual feature map into the encoder, use the cross-attention mechanism to guide and enhance the visual features with text semantic information, and output the enhanced visual features that are fused with text cues;

[0034] A collaborative learning module guided by visual restoration is used to introduce a collaborative learning strategy guided by visual restoration to repair and enhance the enhanced visual features fused with textual cues, recovering key information lost due to degradation, improving feature quality, and improving query allocation under noisy conditions.

[0035] The decoding output module is used to input the enhanced visual features of the fused text prompts into the decoder after being processed by the collaborative learning strategy guided by visual restoration, and to locate and classify the defect targets in combination with the learnable target query, and output the location and category information of the defects.

[0036] According to a third aspect, an electronic device is provided, including:

[0037] a memory storing at least one instruction; and

[0038] The processor executes the instructions stored in the memory to implement the degraded image defect detection method that coordinates text prompts and visual restoration as described in the first aspect.

[0039] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the degraded image defect detection method that coordinates text prompts and visual repair as described in the first aspect.

[0040] Compared with the prior art, the present invention has at least the following beneficial effects:

[0041] Significantly improves degraded image detection performance: By leveraging the adaptability of high-order degradation models to real-world noise, combined with semantic guidance from textual hints and enhanced feature quality through visual restoration, our method significantly improves the accuracy (AP), recall, and mean average precision (mAP) of defect detection under degraded conditions such as noise, blur, and artifacts. Experimental results show that our method improves mAP by 4.5% on the ABSD dataset (from 67.3% to 71.8%).

[0042] End-to-end optimization avoids suboptimal solutions: Image degradation modeling, text guidance, feature extraction, visual restoration, and defect detection are integrated into a unified end-to-end framework for joint optimization. This allows each module to work together and learn directly towards the final detection goal, avoiding the suboptimal problems caused by inconsistent optimization goals and error accumulation in the traditional "enhancement-then-detection" two-stage approach.

[0043] Leveraging semantic information to enhance robustness: We propose a mechanism for generating and fusing textual cues, enabling the model to leverage relatively stable semantic cues extracted from degraded images to assist in defect localization and identification. When visual information is severely impaired, textual cues can provide additional important contextual information, guiding the model's focus on potential target areas and effectively reducing missed and false detections.

[0044] Parameter-efficient adaptive backbone network: Low-rank adaptation (LoRA) technology is used to fine-tune the pre-trained backbone network. This can effectively adapt the backbone network to the feature distribution of degraded images without significantly increasing the number of model parameters and computational complexity, and extract more robust visual features while maintaining the strong basic capabilities of the pre-trained model.

[0045] Improved query allocation and feature quality: A collaborative learning strategy guided by visual restoration can effectively restore and enhance useful information in encoded features. It also improves the allocation efficiency and accuracy of decoder target queries in noisy environments, accelerating model convergence and improving performance.

[0046] Stronger generalization capability: Because the model is exposed to a large number of images simulating real-world degradation during training and learns to utilize multimodal information, it has better generalization capabilities for unseen degradation scenarios and different types of defects, making it more suitable for deployment in complex and changing real-world industrial environments.

[0047] Broad application prospects: The method proposed in the present invention is suitable for specific defect detection tasks (such as surface defects of aerospace blades) and has potential application value in many fields such as industrial quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 An example of defect detection results of the degraded image defect detection method based on collaborative textual prompts and visual restoration proposed in an embodiment of the present invention on an aerospace blade surface defect (ABSD) dataset is shown in the figure below:

[0050] (a) Detection results with dents and erosion; (b) Detection results with erosion; (c) Detection results with scratches and dents;

[0051] Figure 2 Schematic diagram of the principle of a method for detecting degraded image defects by coordinating text prompts and visual restoration according to an embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the Recurrent Residual Dense Learning (RRDL) architecture used in the collaborative learning strategy guided by visual restoration according to an embodiment of the present invention;

[0053] Figure 4 This is a visual comparison of the detection results of the degraded image defect detection method using collaborative text prompts and visual restoration in an embodiment of the present invention and several other DETR-based SOTA (State-of-the-Art) detection methods on the ABSD dataset:

[0054] (a) DINO; (b) Rank-DETR; (c) Relation-DETR; (d) Co-DETR; (e) H-DETR; (f) the method of the present invention; (g) GT (Ground Truth). DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also derive other embodiments without making any creative work.

[0056] An embodiment of the present invention proposes a method for detecting defects in degraded images by coordinating text prompts and visual restoration. The method is a robust defect detection method that can maintain high detection performance even when image quality is degraded (such as in the presence of noise, blur, artifacts, etc.). The method can be applied to scenarios with high requirements for detection accuracy and robustness, such as product quality inspection on industrial automation production lines, surface defect detection of aerospace components, and security monitoring of critical infrastructure.

[0057] The present invention proposes an end-to-end framework for detecting degraded images by combining text prompts with visual restoration. The framework mainly implements the following:

[0058] (1) Improve the accuracy and robustness of defect detection models under degradation conditions such as noise, blur, and low resolution.

[0059] (2) Jointly optimize visual restoration and defect detection tasks in an end-to-end manner to avoid the problems of inconsistent optimization objectives and error accumulation in the two-stage method.

[0060] (3) The semantic information contained in the text prompts is used to guide the defect detection model to more effectively focus on and identify defect targets in degraded images.

[0061] (4) A parameter-efficient backbone network and feature enhancement mechanism that can adapt to degraded input are designed.

[0062] (5) Reduce excessive reliance on high-quality training data and enhance the generalization ability of defect detection models to complex imaging conditions in the real world.

[0063] See also Figure 2 The method for detecting degraded image defects by coordinating text prompts and visual restoration in an embodiment of the present invention includes the following steps:

[0064] S1. Generate degraded images containing real-world noise characteristics through a pre-built high-order degradation model;

[0065] S2. Extract semantic clues related to the detection target from the degraded image and encode the semantic clues into a text feature vector;

[0066] S3. Input the degraded image into the degradation-resistant backbone network integrated with the low-rank adaptive module, and output a robust visual feature map;

[0067] S4. Input the text feature vector and robust visual feature map into the encoder, use the cross-attention mechanism to guide and enhance the visual features with the text semantic information, and output the enhanced visual features that integrate the text cues.

[0068] S5. Introducing a collaborative learning strategy guided by visual restoration to repair and enhance the enhanced visual features fused with textual cues, restoring key information lost due to degradation, improving feature quality, and improving query allocation under noisy conditions.

[0069] S6. After being processed by the collaborative learning strategy guided by visual restoration, the enhanced visual features fused with textual cues are input into the decoder, and the defective targets are located and classified in combination with the learnable target query, and the location and category information of the defects are output.

[0070] In one possible implementation, in order to enable the defect detection model to learn complex degradation patterns in the real world and bridge the domain gap between training data (usually high-quality images) and low-quality images in actual applications, in step S1 of this embodiment of the present invention, a high-order degradation model is pre-constructed. The input of the high-order degradation model is a high-quality (HQ) original image y. The high-order degradation model performs two or more classic degradation operations, each of which includes at least two operations among blur kernel convolution, downsampling of a specific scale factor, noise addition (such as Gaussian noise, Poisson noise), and JPEG compression. The mathematical expression of the classic degradation operation is: Where, represents the blur kernel k convolution, ↓ r represents downsampling with a scale factor of r, n represents additive noise, [·] JPEG Indicates JPEG compression.

[0071] The present invention adopts a high-order degradation model to apply N d times (such as N d =2) Independent classical degradation process Each process can have different hyperparameters (such as different blur kernels, noise levels, compression factors). Mathematically expressed as:

[0072] Output the simulated low-quality (LQ) image x.

[0073] Through this process, a corresponding LQ image can be generated for each HQ image in the training set to form an LQ-HQ image pair for robust training of the defect detection model.

[0074] In one possible implementation, step S2 inputs the degraded image into a degradation-aware prompt extractor (TExtractor), which extracts semantic keywords or phrases related to the defect target to be detected in the degraded image to form an initial text prompt;

[0075] The initial text prompt is input into the text encoder (TEncoder), which converts the text prompt into a text feature vector with a fixed dimension.

[0076] The degradation-aware cue extractor analyzes the input degraded image and generates a textual cue p describing the image content or indicating the target to be detected. For example, for an image of a leaf containing "cracks" and "dents," the extractor might generate the text "An image of a leaf containing cracks and dents" or, more concisely, "Cracks, dents." This extractor can be a pretrained image captioning model, a large multimodal language model, or a simple classifier designed for a specific defect category. In this invention, the extractor's parameters are typically frozen during main model training to avoid adding additional training burden.

[0077] The text encoder converts the text prompt p generated by the extractor into a fixed-length, dense text feature vector v. Commonly used text encoders include BERT and the text encoding part of CLIP. Similarly, text encoders usually keep their parameters frozen.

[0078] Mathematically expressed as: p = T Extractor (D(y)), v = T Encoder (p).

[0079] In one possible implementation, in step S3, the low-rank adaptation (LoRA) module freezes the original model parameters by adding a low-rank trainable matrix to the weight matrix of the degradation-robust backbone network (Degradation-Robust Backbone) model, thereby fine-tuning and enhancing the feature extraction capability of the degraded image and outputting a preliminary robust visual feature map.

[0080] In order to effectively extract features from degraded images and reduce the performance loss caused by directly using the backbone network pre-trained on HQ images, the present invention adaptively transforms the backbone network.

[0081] Basic backbone network: It can be any advanced visual backbone network, such as ResNet, Swin Transformer or ViT, which are usually pre-trained on large-scale datasets such as ImageNet.

[0082] The specific implementation of the low-rank adaptive module is as follows: for the weight matrix of the selected linear layer or two-dimensional convolutional layer in the pre-trained anti-degradation backbone network model Its update ΔW is expressed as the product of two low-rank matrices BA, ΔW = BA; where, And the rank r<<min(d,k); during training, only the parameters of matrices A and B are updated, while W0 remains unchanged, and the forward propagation of the layer becomes h=W0x+ΔWx=W0x+BAx.

[0083] Compared to full parameter fine-tuning, the Low Rank Adaptation (LoRA) module significantly reduces the number of parameters that need to be trained, lowering storage requirements and reducing the risk of overfitting. By learning low-rank adjustments specific to degraded data based on pre-trained weights, the backbone network can better adapt to degraded inputs and extract more robust visual features. During inference, the product of the two learned low-rank matrices, BA, and W0, is merged into W0+BA, introducing no additional computational latency and enabling fast switching.

[0084] In one possible implementation, the encoder in step S4 is a Transformer encoder including a multi-layer self-attention mechanism and a feedforward network; the extracted text feature vector v and the robust visual feature map need to be effectively fused to play the guiding role of the text prompt. The Transformer encoder follows the structure of DETR, including a multi-layer self-attention mechanism and a feedforward network. In each layer (or specific layers) of the Transformer encoder, a cross attention module is introduced. The input is the visual features from the anti-degradation backbone network (after self-attention processing as the query Query, Q vis ) and the text feature vector generated from the text prompt (as key Key, K txt Sum Value, V txt ), the calculation expression is as follows:

[0085]

[0086] Where, d k is the dimension of the key vector.

[0087] The output is a more discriminative encoder feature enhanced with textual information. This mechanism allows visual features to actively query and absorb relevant semantic information from textual features. For example, if the textual cue indicates "cracks," the encoder will pay more attention to regions and patterns in the image that match the semantic meaning of "cracks" when processing visual features.

[0088] In one possible implementation, in order to further mitigate the negative impact of degradation on feature representation and improve detection performance, especially in terms of query allocation, step S5 of this embodiment of the present invention introduces a collaborative learning strategy guided by visual restoration.

[0089] The visual inpainting-guided collaborative learning strategy is implemented through a multi-scale feature inpainting unit and multiple auxiliary detection heads.

[0090] Multi-scale Feature Restoration Module:

[0091] Residual in Residual Dense Layer (RRDL):

[0092] The design of the embodiment of the present invention adopts the RRDL structure (such as Figure 3 RRDL is typically composed of multiple stacked basic residual blocks, each of which may contain multiple convolutional layers (or linear layers, depending on the type of input features) and ReLU activation functions, and uses dense residual connections. This deep and complex structure has powerful nonlinear representation and feature reconstruction capabilities.

[0093] Fine-tune and enhance the encoded features to restore key information lost due to degradation and improve feature quality.

[0094] Multiple Auxiliary Detection Heads:

[0095] In the early stages of training for DETR-like models, the decoder's Hungarian matching (for query assignment) may be unstable, especially under noisy conditions.

[0096] Auxiliary detection heads are introduced in the middle layer of the Transformer decoder (similar to the main detection head structure, but lighter). These auxiliary detection heads also participate in loss calculation and backpropagation.

[0097] This can provide additional supervisory signals, helping the decoder learn reasonable query-target matching relationships more quickly, improve query allocation under noisy conditions, accelerate the convergence of defect detection models, and improve the final detection performance.

[0098] Collaborative learning: The introduction of a multi-scale feature restoration module and multiple auxiliary detection heads enables the collaborative optimization and mutual promotion of visual information restoration and target detection tasks within a unified framework.

[0099] In a possible implementation, step S6 is decoding and outputting the DETR.

[0100] Transformer decoder: Receives processed features from the collaborative learning module and a set of learnable object queries. Through self-attention and cross-attention mechanisms (interaction between the query and the encoder output features), each query gradually focuses on a specific object instance.

[0101] Prediction head: Usually a FFN (Feed-Forward Network), connected after the output of each query of the decoder, used to predict the category and bounding box coordinates of the target corresponding to the query.

[0102] Loss function: A set prediction loss of the DETR standard is used, which usually includes classification loss (such as cross entropy loss) and bounding box regression loss (such as a combination of L1 loss and GIoU loss). Similar losses are also calculated for the output of the auxiliary detection head.

[0103] Another embodiment of the present invention describes in detail the degraded image defect detection method of the present invention by coordinating text prompts and visual restoration with an aerospace blade surface defect (ABSD) dataset, and specifically includes the following steps:

[0104] 1. Data preparation and preprocessing

[0105] Dataset:

[0106] Task-specific dataset: Aerospace Blade Surface Defect (ABSD) dataset. This dataset is self-constructed and contains 317 training images and 30 validation images, covering four defect categories: crack, dent, erosion, and scratch. The images are collected from real industrial scenes and may contain varying degrees of natural degradation.

[0107] Real noise simulation process:

[0108] Input: High-quality (HQ) image y from ABSD.

[0109] High-order degradation model: using the second-order classical degradation operation (N d =2).

[0110] First degradation

[0111] Blur: Randomly selects a Gaussian blur kernel, and the kernel size and standard deviation are randomly selected within the preset range.

[0112] Downsampling: Downsample with a random scaling factor r1 (e.g. between 1.5 and 4).

[0113] Noise addition: Add Gaussian white noise, and the noise level σ1 is randomly selected within a certain range.

[0114] JPEG compression: JPEG compression is performed with a random compression quality factor q1 (e.g., between 40 and 90).

[0115] Second downgrade right A similar degradation operation is again applied to the output of , but with an independent set of random parameters (blur kernel, r2, σ2, q2).

[0116] Output: Low quality (LQ) image This generates LQ-HQ image pairs for the training set.

[0117] 2. The TPVR-DETR model proposed in this embodiment of the present invention mainly consists of the following core modules connected in series or in parallel:

[0118] 2.1 Text prompt generation process

[0119] Degradation-aware hint extractor (TExtractor):

[0120] You can choose a pre-trained, lightweight image classification model (such as MobileNetV2 pre-trained on ImageNet and fine-tuned for defect categories or general object categories), or a simple image-to-label mapping rule library. In this embodiment, for the sake of versatility and efficiency, you can consider using a small-parameter image captioning model pre-trained on a public captioning dataset and extracting keywords from the generated descriptive text as prompts. For example, for the ABSD dataset, you can pre-define the names of defect categories as potential prompts.

[0121] The parameters of this module are kept frozen during the training of the TPVR-DETR main model.

[0122] Text Encoder (TEncoder):

[0123] The text encoder part of the pre-trained CLIP model is used. The input is the text prompt p generated by TExtractor, and the output is a fixed-dimensional text feature vector v (for example, 512 dimensions).

[0124] The module parameters are also kept frozen during the main model training.

[0125] 2.2 Anti-degradation backbone network

[0126] Basic backbone: Swin Transformer (Swin-T or Swin-B) pre-trained on ImageNet is used as the basis for visual feature extraction.

[0127] The LoRA module is integrated into the query (Q), key (K), value (V) projection linear layers of the multi-head self-attention (MHSA) module and the linear layer of the feed-forward network (FFN) in each Transformer block of the Swin Transformer.

[0128] Specifically, for the weight matrix W0, add a trainable and The rank r of LoRA is set to a smaller value, such as r=4 or r=8.

[0129] During initialization, A is usually initialized with a Gaussian distribution and B is initialized to zero, so that ΔW=BA is zero at the beginning and the model starts learning from the pre-trained weights.

[0130] Output: LoRA-enhanced multi-scale visual feature map that is more adaptable to degraded images.

[0131] 2.3 Object Detection Process for Degradation

[0132] Transformer Encoder:

[0133] It adopts the standard DETR encoder structure, which includes 6 layers of Transformer encoding layers.

[0134] Text-Visual Feature Fusion: After the Multi-Head Self-Attention (MHSA) module in each encoding layer, an additional Cross-Attention module is inserted. This Cross-Attention module uses the visual features of the current layer as the query (Query) and the global text feature vector v generated in step 2.1 as the key (Key) and value (Value). In this way, the semantic information of the text is injected into each layer of visual feature processing, guiding the model to focus on visual areas related to the textual cues.

[0135] Visual prosthesis guided collaborative learning:

[0136] Multi-scale feature restoration (RRDL): The feature sequence output by the Transformer encoder (which may be taken from the last layer or multi-layer fusion) is input into a RRDL module. Figure 3 As shown in the figure, it is composed of several (e.g., 3-5) residual dense blocks (RRDBs) connected in series. Each RRDB contains multiple densely connected convolutional layers (or 1D convolutional / linear layers for sequence features) and ReLU activations with residual connections. The entire RRDL module also has a global residual connection outside.

[0137] Multi-scale feature restoration aims to deeply repair and refine the encoder features that incorporate text information, enhance their representation capabilities, and reduce information loss caused by degradation.

[0138] Auxiliary detection heads: Auxiliary detection heads are added in parallel after the output of the middle layers of the Transformer decoder (for example, layers 3, 4, and 5). These auxiliary heads share the same structure (FFN for classification and regression) as the main detection head (the head after the last layer of the decoder), but calculate independent losses during training and add them to the total loss according to certain weights.

[0139] Transformer Decoder:

[0140] It adopts the standard DETR decoder structure, which includes 6 layers of Transformer decoding layers.

[0141] The input includes: enhanced visual features from the collaborative learning module (ultimately the output of RRDL), and a set of learnable target queries (e.g., 100 or 300).

[0142] Through self-attention inside the decoder and cross-attention on the encoder output, the target query is gradually refined and aligned with the target object in the image.

[0143] Prediction head (FFN):

[0144] Concatenate after each target query output of each decoder layer (primary and auxiliary).

[0145] It consists of a classification branch (predicting the target category, with an output dimension of the number of categories + 1, including background categories) and a regression branch (predicting the normalized bounding box center coordinates, width, and height).

[0146] 3. Training Process

[0147] Optimizer: AdamW.

[0148] Learning rate: The initial learning rate is 5×10 -5 , and adopt a learning rate scheduling strategy (such as cosine annealing or step decay).

[0149] Weight decay: 0.05.

[0150] Batch Size: Adjust according to the GPU memory. For example, on a 4-card NVIDIA RTX A6000 (48GB), you can set an appropriate batch size.

[0151] Training cycles (Epochs):

[0152] ABSD dataset: 50 epochs.

[0153] Loss function: For each detection head (main and auxiliary), the loss function is a weighted sum of classification loss and bounding box regression loss.

[0154] Classification loss: Focal Loss or standard cross entropy loss.

[0155] Bounding box regression loss: a linear combination of L1 loss and GIoU (Generalized Intersection over Union) loss.

[0156] The loss is calculated by performing the optimal match between the prediction and the true target using the Hungarian Algorithm.

[0157] Hardware: 4 NVIDIA RTX A6000 GPUs.

[0158] 4. Evaluation Metrics

[0159] Use standard defect detection evaluation metrics:

[0160] AP: Average Precision (mean of IoU thresholds from 0.5 to 0.95 with an interval of 0.05).

[0161] AP@50: AP when the IoU threshold is 0.5.

[0162] AP@75: AP when the IoU threshold is 0.75.

[0163] APs, APm, APl: AP for small, medium and large size targets respectively.

[0164] 5. Please refer to Figure 1 (a) to (c) and Figure 4 (a) to (g), experimental results and analysis

[0165] Table 1 Performance on the ABSD dataset

[0166] Model Training cycle AP AP@50 AP@75 <![CDATA[AP S ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> DINO 50 53.2 82.3 56.5 45.1 57.1 61.5 Rank-DETR 50 67.1 93.9 71.4 48.2 71.6 79.8 Relation-DETR 50 59.3 91.6 63.2 45.0 69.2 67.0 Co-DETR 50 67.3 91.6 78.7 53.5 64.8 77.7 H-DETR 50 51.2 89.0 46.7 40.8 68.6 54.8 TPVR-DETR 50 71.8 92.8 80.0 64.0 76.1 84.2

[0167] TPVR-DETR (50 epochs): AP reaches 71.8%, AP@75 reaches 80.0%, and APs reaches 64.0%.

[0168] Compared with Co-DETR (AP 67.3%), TPVR-DETR achieves an AP improvement of 4.5%. APs are improved by 10.5% (from 53.5% to 64.0%), showing a significant enhancement in the ability to detect small defects.

[0169] Compared with H-DETR, APm improves by 4.5% and APl improves by 4.4%.

[0170] Figure 1 (a) to (c) intuitively demonstrate the model's ability to detect various defects (such as cracks, dents, etc.) in actual application scenarios. Figure 4 (a) to (g) are visual comparisons of the detection results of the TPVR-DETR model of the embodiment of the present invention and several other SOTA (State-of-the-Art) detection methods based on DETR (such as DINO, Co-DETR, H-DETR, Rank-DETR, Relation-DETR) on the ABSD dataset. Figure 4 (g) represents the true value, Figure 4 (g) is used as the standard for comparison. The detection method whose detection result is most similar to the true value has the best effect.

[0171] Figure 4 The visualization results show that on a sample from the ABSD dataset, other methods (DINO, Co-DETR, H-DETR, Rank-DETR, and Relation-DETR) suffer from missed detections (failure to detect all "dent" objects), false positives (Rank-DETR), or inaccurate bounding boxes (Relation-DETR). However, the TPVR-DETR model of this embodiment successfully detects all objects, and the bounding boxes closely match the ground-truth annotations.

[0172] Figure 1 The examples also further prove that TPVR-DETR can effectively detect different types of surface defects on the ABSD dataset, with clear results and accurate positioning.

[0173] In some other possible implementations, for example, a variation of the text prompt generation process further includes:

[0174] In Example 1, the extractor and encoder of the text prompt generation process are pre-trained and frozen. In this example, the following variants can be considered:

[0175] Task-specific hint extractor: For datasets such as ABSD with clear defect categories, T Extractor A simpler module is allowed, for example, to use the defect category name (such as "crack", "dent") as the text prompt. If an image may contain multiple defects, all possible category names can be combined or processed independently.

[0176] Fine-tunable text module: If computing resources allow and there is sufficient multimodal annotated data, you can considerExtractor and / or T Encoder Fine-tune some parameters of the model to better suit the specific defect detection task and degradation mode. However, this requires careful design to avoid overfitting.

[0177] Learning hints without explicit text input: Another approach is to learn “pseudo-text hints”, that is, instead of generating human-readable text directly from images, we learn a set of trainable vector embeddings that functionally play the role of text hints and interact directly with visual features in the Transformer encoder.

[0178] In some other possible implementations, the LoRA configuration for the degradation-resistant backbone network includes:

[0179] The rank r of LoRA is an important hyperparameter. Ablation experiments can be performed to determine the optimal r value.

[0180] Smaller r (such as 1, 2) may not be able to learn enough.

[0181] A larger r (e.g., 16, 32) will add more parameters, which may lose the advantage of LoRA's parameter efficiency and may lead to overfitting.

[0182] The experiment can test values ​​of r∈{2, 4, 8, 16} and select the configuration that performs best on the validation set and has reasonable parameter increments.

[0183] LoRA can also be selectively applied to different parts of the backbone network, for example only to deeper layers, or only to specific projections in the attention module (such as only Q and V projections).

[0184] In some other possible implementations, component selection of the collaborative learning strategy guided by visual restoration includes:

[0185] The depth and width of RRDL: The number of RRDBs in an RRDL module, as well as the number of convolutional layers and channels within each RRDB, can be adjusted. Deeper and wider RRDLs generally have stronger repair capabilities, but also come with higher computational costs.

[0186] Position and number of auxiliary detection heads: Auxiliary heads can be introduced at different layers of the decoder (e.g., from layer 1 to layer 5, or only in the middle layers). The loss weights of the auxiliary heads also need to be adjusted.

[0187] Through the above detailed description of the implementation method, combined with the explanation of the accompanying drawings, the technical solution, implementation process and the beneficial effects brought about by the present invention are clearly presented. Those skilled in the art should understand that, without departing from the spirit and scope of the present invention, various modifications and variations can be made to the above embodiments. For example, the specific number of network layers, the selection of activation functions, the parameter settings of the optimizer, etc. can all be adjusted according to the actual application scenario and available computing resources. In addition, the core ideas and frameworks proposed in the present invention can also be applied to other types of visual perception tasks, such as image segmentation, posture estimation, etc., as long as these tasks also face the challenges brought about by image degradation.

[0188] Another embodiment of the present invention further provides a degraded image defect detection system that collaborates with text prompts and visual restoration, including:

[0189] A degraded image generation module is used to generate degraded images containing real-world noise characteristics through a pre-built high-order degradation model;

[0190] Semantic clue encoding module, which is used to extract semantic clues related to the detection target from the degraded image and encode the semantic clues into text feature vectors;

[0191] A robust visual feature map output module is used to input the degraded image into the degradation-resistant backbone network integrated with the low-rank adaptive module and output a robust visual feature map;

[0192] The text and visual feature fusion encoding module is used to input the text feature vector and the robust visual feature map into the encoder, use the cross-attention mechanism to guide and enhance the visual features with text semantic information, and output the enhanced visual features that are fused with text cues;

[0193] A collaborative learning module guided by visual restoration is used to introduce a collaborative learning strategy guided by visual restoration to repair and enhance the enhanced visual features fused with textual cues, recovering key information lost due to degradation, improving feature quality, and improving query allocation under noisy conditions.

[0194] The decoding output module is used to input the enhanced visual features of the fused text prompts into the decoder after being processed by the collaborative learning strategy guided by visual restoration, and to locate and classify the defect targets in combination with the learnable target query, and output the location and category information of the defects.

[0195] Another embodiment of the present invention further provides an electronic device, including:

[0196] A memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the degraded image defect detection method with coordinated text prompting and visual restoration.

[0197] Another embodiment of the present invention further proposes a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the degraded image defect detection method with coordinated text prompts and visual repair.

[0198] Exemplarily, the instructions stored in the memory can be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to implement the degraded image defect detection method for coordinated textual prompting and visual restoration of the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the server.

[0199] The electronic device may be a computing device such as a smartphone, laptop, PDA, or cloud server. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components, or a combination of certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, and the like.

[0200] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0201] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk equipped on the server, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is about to be output.

[0202] It should be noted that the information interaction, execution process, etc. between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0204] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0205] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0206] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for detecting degraded image defects by coordinating text prompts and visual restoration, characterized in that: include: Generate degraded images containing real-world noise characteristics through a pre-built high-order degradation model; Extract semantic clues related to the detection target from the degraded image and encode the semantic clues into text feature vectors; The degraded image is input into the degradation-resistant backbone network integrated with low-rank adaptive modules, and the robust visual feature map is output; The text feature vector and robust visual feature map are input into the encoder, and the cross-attention mechanism is used to guide and enhance the visual features with text semantic information, and the enhanced visual features that are integrated with the text hints are output; A collaborative learning strategy guided by visual restoration is introduced to repair and enhance the enhanced visual features fused with textual hints, recovering key information lost due to quality degradation, improving feature quality, and improving query allocation under noisy conditions. The enhanced visual features fused with textual cues are processed by a collaborative learning strategy guided by visual restoration and then input into the decoder. Combined with learnable target queries, defect targets are located and classified, and the location and category information of the defects are output.

2. The method for detecting degraded image defects by coordinating text prompts and visual restoration according to claim 1, characterized in that: In the step of generating a degraded image containing real-world noise characteristics by using a pre-built high-order degradation model, the high-order degradation model performs two or more classic degradation operations, each of which includes at least two operations among blur kernel convolution, downsampling of a specific scale factor, noise addition, and JPEG compression; The mathematical expression of the classic degradation operation is: Where, represents the blur kernel k convolution, ↓ r represents downsampling with a scale factor of r, n represents additive noise, [·] JPEG Indicates JPEG compression.

3. The method for detecting degraded image defects by coordinating text prompts and visual restoration according to claim 1, characterized in that: In the step of extracting semantic clues related to the detection target from the degraded image and encoding the semantic clues into a text feature vector, the degraded image is input into a degradation perception hint extractor, which extracts semantic keywords or phrases related to the defect target to be detected in the degraded image to form an initial text hint; the initial text hint is input into a text encoder, which converts the text hint into a text feature vector with a fixed dimension.

4. The method for detecting degraded image defects by coordinating text prompts and visual restoration according to claim 1, characterized in that: In the step of inputting the degraded image into the anti-degradation backbone network integrated with the low-rank adaptive module and outputting the robust visual feature map, the low-rank adaptive module freezes the original model parameters by adding a low-rank trainable matrix in the weight matrix bypass of the anti-degradation backbone network model, thereby fine-tuning and enhancing the feature extraction capability of the degraded image and outputting a preliminary robust visual feature map; The specific implementation of the low-rank adaptive module is as follows: for the weight matrix of the selected linear layer or two-dimensional convolutional layer in the pre-trained anti-degradation backbone network model Its update ΔW is expressed as the product of two low-rank matrices BA, ΔW = BA; where, And the rank r<<min(d,k); during training, only the parameters of matrices A and B are updated, while W0 remains unchanged, and the forward propagation of the layer becomes h=W0x+ΔWx=W0x+BAx.

5. The method for detecting degraded image defects by coordinating text prompts and visual restoration according to claim 1, characterized in that: In the step of inputting the text feature vector and the robust visual feature map into the encoder, adopting the cross-attention mechanism to guide and enhance the visual features with the text semantic information, and outputting the enhanced visual features fused with the text prompts, the encoder is a Transformer encoder including a multi-layer self-attention mechanism and a feedforward network; In at least one attention layer of the Transformer encoder, a cross attention mechanism is used to take the robust visual feature map as the query Q vis , using the text feature vector as the key K txt Sum V txt , or vice versa; The calculation is performed as follows: Where, d k is the dimension of the key vector.

6. The method for detecting degraded image defects by coordinating text prompts and visual restoration according to claim 1, characterized in that: In the step of introducing a collaborative learning strategy guided by visual restoration to repair and enhance the enhanced visual features fused with textual cues, recover key information lost due to degradation, improve feature quality, and improve query allocation under noisy conditions, the collaborative learning strategy guided by visual restoration is implemented through a multi-scale feature restoration unit and multiple auxiliary detection heads; The multi-scale feature restoration unit uses a cyclic residual dense learning (RRDL) structure to process the multi-scale features output by the encoder, repair and enhance the enhanced visual features of the fused text prompts, recover the key information lost due to degradation, and improve the feature quality; The RRDL structure is composed of multiple basic residual blocks stacked together. Each basic block contains multiple convolutional layers or linear layers and ReLU activation functions, and applies multi-level residual connections. The multiple auxiliary detection heads are introduced in the middle layer of the decoder to supervise features at different levels, provide additional supervision signals, help the decoder learn the query-target matching relationship, and improve query allocation under noisy conditions.

7. The method for detecting degraded image defects by coordinating text prompts and visual restoration according to claim 1, characterized in that: In the step of inputting the enhanced visual features fused with textual cues into a decoder after being processed by a collaborative learning strategy guided by visual restoration, locating and classifying defective targets in combination with learnable target queries, and outputting the location and category information of the defects, the decoder is a Transformer decoder, which receives the enhanced visual features fused with textual cues after being processed by a collaborative learning strategy guided by visual restoration, as well as a set of learnable target queries. Through self-attention and cross-attention mechanisms, each query is gradually focused on a specific target instance. The prediction head is a feedforward neural network connected to the output of each query of the decoder and is used to predict the category and bounding box coordinates of the target corresponding to the query. The loss function adopts the set prediction loss of the DETR standard, including classification loss and bounding box regression loss.

8. A degraded image defect detection system that collaborates with text prompts and visual restoration, characterized in that: include: A degraded image generation module is used to generate degraded images containing real-world noise characteristics through a pre-built high-order degradation model; Semantic clue encoding module, which is used to extract semantic clues related to the detection target from the degraded image and encode the semantic clues into text feature vectors; A robust visual feature map output module is used to input the degraded image into the degradation-resistant backbone network integrated with the low-rank adaptive module and output a robust visual feature map; The text and visual feature fusion encoding module is used to input the text feature vector and the robust visual feature map into the encoder, use the cross-attention mechanism to guide and enhance the visual features with text semantic information, and output the enhanced visual features that are fused with text cues; A collaborative learning module guided by visual restoration is used to introduce a collaborative learning strategy guided by visual restoration to repair and enhance the enhanced visual features fused with textual cues, recovering key information lost due to degradation, improving feature quality, and improving query allocation under noisy conditions. The decoding output module is used to input the enhanced visual features of the fused text prompts into the decoder after being processed by the collaborative learning strategy guided by visual restoration, and to locate and classify the defect targets in combination with the learnable target query, and output the location and category information of the defects.

9. An electronic device, characterized in that: include: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the method for detecting degraded image defects by coordinating text prompts and visual restoration as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the degraded image defect detection method with coordinated text prompting and visual repair as described in any one of claims 1 to 7.

Images