Degraded scene target detection method based on prompt learning

Abstract

The invention discloses a degraded scene target detection method based on prompt learning. A ternary data set including degraded images, degraded type labels and degraded text description of three degraded scenes is constructed for training and verification. The degraded scene target detection network comprises an image recovery encoder, an image recovery decoder, a degradation perception hard prompt module DAPM, a detection task guided semantic soft prompt module DGSPM and a target detector. The semantic soft prompt module guided by the detection task designs a multi-scale learnable prompt pool, and the multi-scale learnable prompt pool and features extracted by the image recovery encoder are subjected to weighted fusion, so that semantic soft prompts related to the target detection task are introduced. And the degradation perception hard prompt module is used for jointly coding degradation text description corresponding to the degradation image and a degradation type label into a cross-scene universal degradation perception hard prompt. After soft and hard prompts are fused, the soft and hard prompts are injected into an image recovery decoder in a jump connection mode, so that the target detection performance is remarkably improved and the relatively good image recovery quality is kept in various degraded scenes.

Description

Technical Field

[0001] This invention pertains to computer image processing methods, and particularly relates to a method for detecting degraded objects in scenes based on cue learning. Background Technology

[0002] With the rapid development of computer vision technology, object detection has been widely applied in visual perception tasks such as autonomous driving and drone surveillance. However, in real-world applications, image acquisition devices often face interference from complex and degrading environments such as fog, rain, and low light. The acquired images frequently exhibit degradation phenomena such as low saturation, color distortion, and blurred details, severely weakening the performance of existing detection models. Therefore, how to accurately detect more targets in complex backgrounds has become a critical problem that urgently needs to be solved in the field of computer vision.

[0003] To address the above problems, some studies [1] A cascaded processing approach of first restoring and then detecting is employed. However, such methods primarily optimize pixel-level visual fidelity, neglecting the applicability of the restored image to downstream detection tasks. This results in improved visual quality after restoration but decreased detection performance. Some researchers have proposed a domain-adaptive method... [2] By aligning the feature distributions of the source and target domains, the model learns domain-invariant feature representations. However, due to the significant differences in the distribution of degraded images under different weather conditions, a single domain alignment strategy is insufficient to effectively handle mixed degradation scenarios. In recent years, the joint optimization paradigm of image restoration and object detection has gradually become the mainstream technical approach for object detection in complex degradation scenarios. This type of method... [3] By introducing an image restoration module into the object detection framework and co-optimizing image restoration and object detection through joint training, image representations that are more favorable for object detection can be generated, thereby improving object detection performance. However, these methods still have two shortcomings: First, in the real world, degradation signals are usually random and diverse, and most methods are designed for a single degradation type, lacking the ability to handle multiple degradation types; second, the optimization objective of the restoration task is fundamentally different from the high-level semantic representation required by the detection task, and joint optimization is difficult to achieve an ideal balance between visual restoration effect and detection accuracy, resulting in improved visual fidelity in the restoration results, but potentially loss of semantic information that is crucial for detection.

[0004] Therefore, how to design a target detection method that can efficiently perceive multiple types of degradation and collaboratively guide image restoration and target detection tasks, and achieve targeted injection of semantic information, is a technical problem that urgently needs to be solved.

[0005] [References]

[0006] [1] Edfdf He, K., Sun, J., & Tang, X. (2010). Single image hazeremoval using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12), 2341-2353.

[0007] [2] Hnewa, M., & Radha, H. (2021, September). Multiscale domainadaptive yolo for cross-domain object detection. In 2021 IEEE InternationalConference on Image Processing (ICIP). IEEE, 3323-3327.

[0008] [3] Wang, Y., Yan, Summary of the Invention

[0009] To address the shortcomings of existing joint optimization methods for image restoration and object detection in multi-degradation scenarios, and the difficulty in co-optimizing image restoration and object detection networks, this invention proposes a cue-learning-based object detection method for degraded scenarios. This method introduces a cue-learning mechanism into the joint optimization framework of image restoration and object detection, constructing a degradation-aware hard cue module and a detection task-guided semantic soft cue module. The two types of cue information are fused at the channel level and input into the image restoration network. The resulting restored image is then input into the object detector for joint optimization, thereby improving object detection performance in complex degradation scenarios while maintaining image restoration accuracy.

[0010] This invention proposes a cue-based object detection method for degraded scenes, comprising an image restoration encoder, an image restoration decoder, a degradation-aware hard cue module (DAPM), a detection task-guided semantic soft cue module (DGSPM), and an object detector. First, a ternary dataset is constructed containing degraded images, degradation type labels, and degradation text descriptions for various degradation scenes. This dataset is generated by applying foggy, rainy, and low-light processing to clear images, respectively, to generate degraded images and their corresponding degradation type labels for each of the three degradation types. A visual language model is then used to perform semantic understanding on the degraded images, generating corresponding degradation text descriptions. During the training phase, the degraded images are first input into the image restoration encoder for feature extraction, and the features extracted by the image restoration encoder are passed to the image restoration decoder via skip connections. During the skip connection process, the detection task-guided semantic soft cue module (DGSPM) constructs a corresponding multi-scale learnable cue pool for each skip connection layer, and weightedly fuses the multi-scale learnable cue pools at the same skip connection layer with the features extracted by the image restoration encoder, thereby introducing semantic soft cues relevant to the object detection task at different skip connection layers. The Degradation-Aware Hard Cueing Module (DAPM) utilizes the degradation text descriptions corresponding to the degradation images in the three-dimensional data set. It extracts text embedding vectors through a pre-trained text encoder and jointly encodes them with the degradation type labels corresponding to the degradation images in the three-dimensional data set, forming a cross-scene universal degradation-aware hard cue. Based on the degradation-aware hard cue, channel attention weights are calculated and fused with the semantic soft cue. These weights are then used as skip connection features input to the image restoration decoder to generate a restored image. Finally, the restored image is input to the object detector, outputting the object detection result. This invention effectively improves object detection performance in various degradation scenarios while maintaining image restoration quality. The method includes the following steps:

[0011] Step 1) Construct the VOC-Deg dataset for network training and validation. The dataset is a ternary dataset containing degraded images of various degradation scenarios, degradation type labels, and degradation text descriptions, where each degraded image corresponds to a set of degradation type labels. and its matching degenerate text description The ternary dataset covers three typical degradation scenarios: foggy, rainy, and low-light conditions. It is built based on the open-source VOC (Video Object Classification) dataset, which contains 9578 training images and 2129 test images, covering 20 target categories. Based on the VOC dataset, different degradation models are applied to the clear images to generate corresponding degraded images, and corresponding degraded text descriptions are simultaneously constructed, thus forming the ternary dataset.

[0012] The construction process of the ternary dataset mainly includes two steps: degraded image generation and degraded text description construction.

[0013] 1-1) During the degradation image generation process, foggy, rainy, and low-light degradation processing are applied to the clear images in the original VOC dataset for three different degradation types, respectively, to generate degradation images of the corresponding degradation types. Here, the original clear image is I.

[0014] For foggy images, an atmospheric scattering model is used to process the clear image I to generate the foggy image:

[0015] (1)

[0016] Where I represents a clear image. This image represents a foggy day. A represents global atmospheric light. Representing atmospheric scattering parameters, set to 0.5 and , respectively. .

[0017] For rainy day images, a physical simulation-based method is used to synthesize the clear image I. First, high-density, uniformly distributed random noise is generated, and then dynamic thresholding is applied. Sparsity reduction is performed to preserve bright noise. Then, a motion blur kernel is used to stretch the noise into stripes. The kernel length is set to 30-70 pixels, and the rotation angle is... To simulate rainfall at different tilt angles. Then, Gamma correction is used ( The stripe brightness was enhanced to simulate the reflective properties of rain streaks. Finally, the rain layer R was superimposed onto the sharp image I using alpha blending.

[0018] (2)

[0019] Where the mixing coefficient The value ranges from 0.6 to 0.9. Additionally, a global brightness attenuation of 0.9 to 1.0 is applied to the synthesized image to simulate a rainy environment.

[0020] For low-light images, a brightness scaling factor is introduced. The global energy of the image is suppressed, and then gamma correction is applied to simulate the nonlinear characteristics of the camera response function. Its degradation model is defined as:

[0021] (3)

[0022] in This processing not only reduces the overall illumination of the image, but also focuses on suppressing the contrast of the dark areas of the image, which greatly weakens the edge features and semantic information of the target object, thus rigorously testing the feature extraction capabilities of the model.

[0023] In addition, the RTTS dataset was used as a real-world foggy test set, which contains 4,322 real foggy images and 5 object categories.

[0024] 1-2) Secondly, degraded text description data is constructed to provide the Degraded Image Awareness Hard Cueing Module (DAPM) with degraded text descriptions corresponding to the degraded images, thereby representing the degradation type, degradation intensity, and visual impact information of the degraded images. To ensure the stability of the generated degraded text descriptions in terms of semantic consistency and linguistic diversity, a visual language model is used to process the degraded images. For each degraded image... By inputting preset prompt words into the visual language model Generate a degraded text description corresponding to the degraded image. :

[0025] (4)

[0026] in, Represents a visual language model. This refers to input prompts that guide the visual language model to generate degenerate text descriptions.

[0027] Step 2) Construct a cue-based object detection network for degraded scenes to process degraded images:

[0028] The network is trained and tested using the triplet dataset constructed in step 1, which includes degraded images of various degradation scenarios, degradation type labels, and degradation text descriptions. The network training process includes the following steps:

[0029] 2-1) For degraded images First, the image is restored using an image restoration network, which employs an encoder-decoder structure. The image restoration encoder... Features are extracted layer by layer through a multi-layer structure, and the encoder features of the corresponding layer are passed to the image restoration decoder through skip connections. To restore high-quality images. The encoder-decoder network consists of four layers of encoding and decoding structure, with three skip connections corresponding to each layer. The features output by the skip connection of the k-th encoder are... Represented as:

[0030] (5)

[0031] in, , indicating a connection of three jumps.

[0032] 2-2) Features of the encoder output for each layer of image recovery A multi-scale learnable cue pool built using the semantic soft cue module DGSPM guided by detection tasks. The DGSPM module is used to obtain semantic soft cues related to the object detection task from the k-th layer skip connections. :

[0033] (6)

[0034] in, This indicates the result processed by the Semantic Soft Hints Module (DGSPM) guided by the detection task.

[0035] 2-3) Using the ternary dataset constructed in step 1, each degraded image in the ternary dataset is associated with a set of degradation type labels. Corresponding degenerate text description The degradation-aware hard cue module DAPM encodes and fuses the data separately to output degradation-aware hard cues. :

[0036] (7)

[0037] in, This indicates the result processed by the Degradation Aware Hard Cueing Module (DAPM). The degeneration type label can be obtained directly from the ternary dataset.

[0038] 2-4) The soft-hard cue fusion mechanism SHCM uses hard cues as conditional information to perform channel-level modulation on soft cues, ultimately obtaining the signal used for hop connections. :

[0039] (8)

[0040] 2-5) The image is injected into the image restoration decoder at each skip connection layer, resulting in a degraded image. The restored image related to the object detection task is obtained by outputting the image restoration decoder. :

[0041] (9)

[0042] in, This indicates the result after processing by the image restoration decoder.

[0043] 2-6) Using the publicly available target detector YOLOv7 For image restoration Perform object detection to obtain the final object detection result. :

[0044] (10)

[0045] in, This represents the result after processing by the object detector Yolov7.

[0046] Furthermore, in the degraded scene target detection method based on cue learning described in this invention, wherein:

[0047] In step 2), the design of the semantic soft cueing module DGSPM guided by the detection task is as follows: the DGSPM module utilizes the feature map extracted by the image recovery encoder in the k-th layer skip connection stage. With instantiated multi-scale learnable cue pool Constructing semantic soft cues relevant to object detection tasks ,include:

[0048] (1) Feature maps extracted by the image reconstruction encoder for each level of the skip connection Each module is configured with a corresponding multi-scale learnable cue pool. In the case of triple-hop connections, three cue pools are configured, each initialized with randomized initial parameters. Each cue pool... Three sets of learnable cue parameters with different receptive ranges are introduced internally to model global contextual information, region semantic information, and local detail information for object detection, respectively:

[0049] (11)

[0050] in, This means that each cue pool contains learnable cue blocks. The total number is 21, where k represents the k-th layer skip connection; The specific settings are: global hints ( ): Consists of 1 full resolution The parameter blocks are responsible for capturing the overall scene structure; region hints ( ) consists of 4 dimensions The parameter blocks are composed of four quadrants of the image, responsible for capturing medium-scale regional features; local cues ( ): Composed of 16 dimensions It consists of parameter blocks that correspond to the fine-grained grid of the image and are responsible for capturing small targets and high-frequency texture details.

[0051] (2) Use skip connections to recover the feature map extracted by the encoder from the image corresponding to the k-th layer. Generate and learnable hint blocks The total number of attention weight vectors with a length of 21 corresponds to the total number of vectors. :

[0052] (12)

[0053] in, This represents the result after global average pooling. This represents the result after processing by the fully connected layer. This represents the result after normalization by the activation function layer.

[0054] Using the attention weight vector mentioned above For the corresponding learnable hint blocks Perform independent weighting to generate activation cue features. :

[0055] (13)

[0056] in, Representing global, regional, and local prompts, It is a 21-dimensional vector. This represents element-wise multiplication;

[0057] Subsequently, Concatenate the components according to spatial quadrant order to construct the semantic soft cue for the final k-th jump connection. :

[0058] (14)

[0059] in, The splicing method is: global prompts are directly generated by... Composition, area prompts will include 4 Strictly follow the spatial stitching order of top left-top right-bottom left-bottom right, with local hints to include 16 [items / items]. according to The grid is then spatially stitched together in sequence; subsequently, the three elements are... Perform channel-level splicing.

[0060] In step 2), the degradation-aware hard cue module DAPM is designed as follows: the DAPM module utilizes the degradation type labels constructed in step 1). Corresponding degenerate text description Generate degradation-perceived hard cues that include degradation type, degradation intensity, and visual impact information. ,include:

[0061] (1) For discrete degenerate type labels One-hot encoding is used to generate degenerate type vectors. :

[0062] (15)

[0063] in, Represents one-hot encoding, for Types of degradation, Converted into a dimensional vector Degraded image The corresponding degenerate type index position is 1, while the positions for all other degenerate types are 0. Set it to 3.

[0064] (2) For degenerate text description The text is processed using a CLIP pre-trained text encoder and normalized using L2. Convert to fixed-dimensional text embedding vectors , It contains information on the degradation type, degradation intensity, and visual impact of the current degraded image, which is used to guide the restoration and detection of degraded images in different degradation scenarios:

[0065] (16)

[0066] in , This represents the CLIP pre-trained text encoder. This indicates that the vector is subjected to L2 normalization, which means dividing the vector by its L2 norm so that its length is 1 while its direction remains unchanged.

[0067] (3) Degenerate type vector (in ) and text embedding vector Perform joint encoding, where :

[0068] First, the degenerate type vector is processed using two fully connected layers (FC) and one activation function layer (GeLU). Mapped twice to the text embedding vector Same channel dimension:

[0069] (17)

[0070] in, This represents a mapping network containing a non-linear activation function, consisting of two fully connected layers (FC) and one activation function layer (GeLU).

[0071] Subsequently, the aligned degenerate type vector With text embedding vectors The results are summed and then passed through L2 normalization and a channel downsampling convolutional layer to generate the final degradation-aware hard cue. :

[0072] (18)

[0073] in, This indicates element-wise addition. This indicates that the vector has been subjected to L2 normalization. This represents a channel downsampling convolutional layer.

[0074] In step 2), the soft and hard cue fusion mechanism (SHCM) is designed as follows: the SHCM utilizes the semantic soft cue of the k-th hop connection. and degradation perception hard cue The degradation-perceived hard cue is used as conditional information to modulate the semantic soft cue. Finally, the modulated semantic soft cue is injected into each level of the image restoration decoder via skip connections, including:

[0075] (1) The degradation-aware hard cue is transmitted through a fully connected layer. Mapped to channel-level weight vectors, and normalized using Softmax to generate channel-gated vectors. :

[0076] (19)

[0077] Among them, the degradation-sensing hard cue Semantic soft cues are shared across all levels, i.e., across different skip connection levels. corresponding same.

[0078] Subsequently, the semantic soft hints Apply channel-level weighting:

[0079] (20)

[0080] in, This indicates element-wise multiplication.

[0081] (2) Finally, use one Convolutional layer pairs Channel compression and feature fusion are performed to adapt the features extracted by the skip connection k-th layer image recovery encoder. and with After fusion Convolutional layers generate the final collaborative hints. The final collaborative prompt The features are injected into the k-th layer via skip connections.

[0082] (twenty one)

[0083] in, This indicates element-wise addition.

[0084] Compared with the prior art, the beneficial effects of the present invention are:

[0085] Compared with methods such as YOLOv7, PromptIR-YoLOv7, TogetherNet, and End-to-End Net, the proposed cue-based object detection method for degraded scenes introduces a cue learning mechanism under a joint optimization paradigm, offering the following significant advantages: Firstly, by constructing a degradation-aware hard cue module (DAPM) and a triplet dataset containing degraded images, degradation type labels, and degradation text descriptions across various degradation scenarios, the model receives explicit and structured degradation prior information. Secondly, by designing a detection task-guided semantic soft cue module (DGSPM), a multi-scale learnable cue pool is introduced, effectively enhancing the model's ability to model semantic information related to object detection. Furthermore, this invention utilizes degradation-aware hard cues to perform channel-level modulation of semantic soft cues, achieving collaborative modeling between degradation prior information and detection semantic information. Excellent object detection performance is achieved on the artificially synthesized foggy, rainy, and low-light dataset VOC-Deg, as well as the real-world foggy dataset RTTS, while maintaining good image restoration results. Attached Figure Description

[0086] Figure 1 This is a schematic diagram of the overall structure of the cue-based learning-based degraded scene object detection network used in this invention.

[0087] Figure 2 This is a schematic diagram of the semantic soft prompting module DGSPM for task guidance in this invention;

[0088] Figure 3 This is a schematic diagram of the Degradation Awareness Hard Cueing Module (DAPM) in this invention;

[0089] Figure 4 The image shows the subjective effect of the method of the present invention on the VOC-Fog-test dataset in foggy weather, where (a) is the original image, (b) is the Yolov7 detection image, (c) is the PromptIR-Yolov7 detection image, (d) is the TogetherNet detection image, (e) is the End-to-End Net detection image, and (f) is the detection image of the present invention.

[0090] Figure 5The image shows the subjective effect of the method of the present invention on the real-world foggy dataset RTTS, where (a) is the original image, (b) is the Yolov7 detection image, (c) is the PromptIR-Yolov7 detection image, (d) is the TogetherNet detection image, (e) is the End-to-End Net detection image, and (f) is the detection image of the present invention.

[0091] Figure 6 The above shows the subjective results of the method of the present invention on the VOC-Rain-test dataset. (a1) and (a2) are two original images, (b1), (c1) and (d1) are the detection images obtained after processing image (a1) using Yolov7, TogetherNet and the method of the present invention, respectively; (b2), (c2) and (d2) are the detection images obtained after processing image (a2) using Yolov7, TogetherNet and the method of the present invention, respectively. Detailed Implementation

[0092] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are not intended to limit the present invention in any way. The VOC dataset involved in the present invention is a publicly available open-source resource, and its use is subject to its license. Even if the data contains faces or other identifiable objects, its use is limited to academic research and model validation purposes and does not involve commercial infringement or illegal activities.

[0093] This invention proposes a degraded scene object detection method based on cue learning. The method mainly consists of an image restoration encoder, an image restoration decoder, a degradation-aware hard cue module (DAPM), a detection task-guided semantic soft cue module (DGSPM), and an object detector. The process of obtaining the final object detection result along the image processing flow is as follows: Figure 1 As shown, the specific steps are as follows:

[0094] Basic Dataset Preparation: A VOC-Deg dataset is constructed for network training and validation. This dataset is a ternary dataset containing degraded images of various degradation scenarios, degradation type labels, and degradation text descriptions. Each degraded image corresponds to a set of degradation type labels. and its matching degenerate text description The VOC-Deg dataset covers three typical degradation scenarios: foggy, rainy, and low-light conditions. It is built based on the open-source VOC clear image dataset, which contains 9578 training images and 2129 test images, covering 20 target categories. Based on this VOC dataset, different degradation models are applied to clear images to generate corresponding degraded images, and corresponding degraded text descriptions are simultaneously constructed, thus forming the ternary dataset. The construction process of the ternary dataset mainly includes two steps: degraded image generation and degraded text description construction.

[0095] For degraded image generation, three types of image construction are included. Fog, rain, and low light degradation processing are applied to the clear images in the VOC dataset to generate degraded images of the corresponding degradation types. Here, the original clear image is I.

[0096] For foggy images, an atmospheric scattering model is used to process the clear image I to generate the foggy image:

[0097]

[0098] Where I represents a clear image. This image represents a foggy day. A represents global atmospheric light. Representing atmospheric scattering parameters, set to 0.5 and , respectively. .

[0099] For rainy day images, a physical simulation-based method is used to synthesize the clear image I. First, high-density, uniformly distributed random noise is generated, and then dynamic thresholding is applied. Sparsity reduction is performed to preserve bright noise. Then, a motion blur kernel is used to stretch the noise into stripes. The kernel length is set to 30-70 pixels, and the rotation angle is... To simulate rainfall at different tilt angles. Then, Gamma correction is used ( The stripe brightness was enhanced to simulate the reflective properties of rain streaks. Finally, the rain layer R was superimposed onto the sharp image I using alpha blending.

[0100]

[0101] Where the mixing coefficient The value ranges from 0.6 to 0.9. Additionally, a global brightness attenuation of 0.9 to 1.0 is applied to the synthesized image to simulate a rainy environment.

[0102] For low-light images, a brightness scaling factor is introduced. The global energy of the image is suppressed, and then gamma correction is applied to simulate the nonlinear characteristics of the camera response function. Its degradation model is defined as:

[0103]

[0104] in This processing not only reduces the overall illumination of the image, but also focuses on suppressing the contrast of the dark areas of the image, which greatly weakens the edge features and semantic information of the target object, thus rigorously testing the feature extraction capabilities of the model.

[0105] In addition, the RTTS dataset was used as a real-world foggy test set, which contains 4,322 real foggy images and 5 object categories.

[0106] Secondly, degraded text description data is constructed to provide the Degraded Image Awareness Hard Cueing (DAPM) module with corresponding degraded text descriptions for each degraded image, thereby representing the degradation type, degradation intensity, and visual impact information of the degraded images. To ensure the stability of the generated degraded text descriptions in terms of semantic consistency and linguistic diversity, a visual language model is used to batch process the degraded images. For each degraded image... By inputting preset prompt words into the visual language model Generate a degraded text description corresponding to the degraded image. :

[0107]

[0108] in, Represents a visual language model. This refers to input prompts that guide the visual language model to generate degenerate text descriptions.

[0109] In this invention, a semantic soft cueing module (DGSPM) is designed to guide the image restoration network to autonomously learn regional features related to the target detection task. In order to achieve target detection in various degradation scenarios and provide rich degradation prior guidance for the model, a degradation-aware hard cueing module (DAPM) is designed.

[0110] The Degradation-Aware Hard Cueing Module (DAPM) utilizes the degradation text descriptions corresponding to the degradation images in the three-dimensional data set. It extracts text embedding vectors through a pre-trained text encoder and jointly encodes them with the degradation type labels corresponding to the degradation images in the three-dimensional data set, forming a cross-scene universal degradation-aware hard cue to enhance the network's ability to distinguish and represent different degradation types. The Detection Task-Guided Semantic Soft Cueing Module (DGSPM) constructs a corresponding multi-scale learnable cue pool for each skip connection layer and weights and fuses the multi-scale learnable cue pools at the same skip connection layer with the features extracted by the image restoration encoder, thereby introducing semantic soft cueing related to the object detection task at different skip connection layers. Based on the degradation-aware hard cueing, channel attention weights are calculated and fused with the semantic soft cueing, then used as skip connection features input to the image restoration decoder to generate a restored image. Finally, the restored image is input to the object detector, outputting the object detection result. This invention effectively improves object detection performance in various degradation scenarios while maintaining image restoration quality.

[0111] like Figure 1 As shown, for the degraded image input to the network First, the image is restored using an image restoration network, which employs an encoder-decoder structure. The image restoration encoder... Features are extracted layer by layer through a multi-layer structure, and the encoder features of the corresponding layer are passed to the image restoration decoder through skip connections. To restore high-quality images. The encoder-decoder network consists of four layers of encoding and decoding structure, with three skip connections corresponding to each layer. The features output by the skip connection of the k-th encoder are... Represented as:

[0112]

[0113] in, , indicating a connection of three jumps.

[0114] In this invention, for the features output by each encoder layer A multi-scale learnable cue pool built using the semantic soft cue module DGSPM guided by detection tasks. Semantic soft cues related to the object detection task are obtained through the DGSPM module. :

[0115]

[0116] in, This indicates the result processed by the Semantic Soft Hints Module (DGSPM) guided by the detection task.

[0117] In this invention, the ternary dataset constructed in step 1 is used, in which each degraded image is associated with a set of degradation type labels. Corresponding degenerate text description The degradation-aware hard cue module (DAPM) encodes and fuses the two separately to output the degradation-aware hard cue. :

[0118]

[0119] in, This indicates the result processed by the Degradation Aware Hard Cueing Module (DAPM). The degeneration type label can be obtained directly from the ternary dataset.

[0120] Subsequently, using the soft-hard cue fusion mechanism SHCM, the hard cues are treated as conditional information, and the soft cues are modulated at the channel level to ultimately obtain the signal used for the hop connection. :

[0121]

[0122] Features , The image is injected into the image restoration decoder of each skip connection, resulting in a degraded image. The restored image related to the object detection task is obtained by outputting the image restoration decoder. :

[0123]

[0124] in, This indicates the result after processing by the image restoration decoder.

[0125] Using the publicly available target detector YOLOv7 For image restoration Perform object detection to obtain the final object detection result. :

[0126]

[0127] in, This represents the result after processing by the object detector Yolov7.

[0128] In this invention, the semantic soft cueing module DGSPM guided by the detection task has the following structure: Figure 2 As shown, the semantic soft cueing module DGSPM module guided by the detection task utilizes features extracted by the image recovery encoder in the k-th layer skip connection stage. With instantiated multi-scale learnable cue pool Constructing semantic soft cues relevant to object detection tasks ,include:

[0129] Feature maps extracted from the image reconstruction encoder corresponding to each level of the skip connections Each module is configured with its corresponding multi-scale learnable cue pool. In the case of triple-hop connections, three cue pools are configured, each initialized with identical initial parameters. Each cue pool... Three sets of learnable parameters with different receptive ranges are introduced internally to model global contextual information, region semantic information, and local detail information for object detection, respectively:

[0130]

[0131] in, This means that each cue pool contains learnable cue blocks. The total number is 21, where k represents the k-th layer skip connection; The specific settings are: global hints ( ): Consists of 1 full resolution The parameter blocks are responsible for capturing the overall scene structure; region hints ( ) consists of 4 dimensions The parameter blocks are composed of four quadrants of the image, responsible for capturing medium-scale regional features; local cues ( ): Composed of 16 dimensions It consists of parameter blocks that correspond to the fine-grained grid of the image and are responsible for capturing small targets and high-frequency texture details.

[0132] The feature map extracted by the image recovery encoder corresponding to the k-th layer of the skip connection Generate and learnable hint blocks The total number of attention weight vectors with a length of 21 corresponds to the total number of vectors. :

[0133]

[0134] in, This represents the result after global average pooling. This represents the result after processing by the fully connected layer. This represents the result after normalization by the activation function layer.

[0135] Using the attention weight vector mentioned above For the corresponding learnable hint blocks Perform independent weighting to generate activation cue features. :

[0136]

[0137] in, Representing global, regional, and local prompts, It is a 21-dimensional vector. This represents element-wise multiplication.

[0138] Subsequently, Concatenate the components according to spatial quadrant order to construct the semantic soft cue for the final k-th jump connection. :

[0139]

[0140] in, The splicing method is: global prompts are directly generated by... Composition, area prompts will include 4 Strictly follow the spatial stitching order of top left-top right-bottom left-bottom right, with local hints to include 16 [items / items]. according to The grid is then spatially stitched together in sequence; subsequently, the three elements are... Perform channel-level splicing.

[0141] In this invention, the structure of the Degradation Awareness Hard Cueing Module (DAPM) is as follows: Figure 3 As shown, the Degradation Aware Hard Cueing (DAPM) module utilizes the degradation type labels constructed in step 1). Corresponding degenerate text description Generate degradation-perceived hard cues that include degradation type, degradation intensity, and visual impact information. ,include:

[0142] For discrete degenerate type labels One-hot encoding is used to generate degenerate type vectors. :

[0143]

[0144] in, Represents one-hot encoding, for Types of degradation, Converted into a dimensional vector Degraded image The corresponding degenerate type index position is 1, while the positions for all other degenerate types are 0. Set it to 3.

[0145] For degenerate text description The text is processed using a CLIP pre-trained text encoder and normalized using L2. Convert to fixed-dimensional text embedding vectors , It contains information on the degradation type, degradation intensity, and visual impact of the current degraded image, which is used to guide the restoration and detection of degraded images in different degradation scenarios:

[0146]

[0147] in , This represents the CLIP pre-trained text encoder. This indicates that the vector is subjected to L2 normalization, which means dividing the vector by its L2 norm so that its length is 1 while its direction remains unchanged.

[0148] Degenerate type vector (in ) and text embedding vector (in Perform joint encoding:

[0149] First, the degenerate type vector is processed using two fully connected layers (FC) and one activation function layer (GeLU). Mapped twice to the text embedding vector Same channel dimension:

[0150]

[0151] in, This represents a mapping network containing a non-linear activation function, consisting of two fully connected layers (FC) and one activation function layer (GeLU).

[0152] Subsequently, the aligned degenerate type vector With text embedding vectors The results are summed and then passed through L2 normalization and a channel downsampling convolutional layer to generate the final degradation-aware hard cue. :

[0153]

[0154] in, This indicates element-wise addition. This indicates that the vector has been subjected to L2 normalization. This represents a channel downsampling convolutional layer.

[0155] In this invention, the design for implementing the soft-hard suggestion fusion mechanism (SHCM) is as follows: the SHCM utilizes semantic soft suggestions during the k-th hop connection. and degradation perception hard cue The degradation-perceived hard cue is used as conditional information to modulate the semantic soft cue. Finally, the modulated semantic soft cue is injected into each level of the image restoration decoder via skip connections, including:

[0156] First, degradation-aware hard cues are implemented through a fully connected layer. Mapped to channel-level weight vectors, and normalized using Softmax to generate channel-gated vectors. :

[0157]

[0158] Among them, the degradation-sensing hard cue Semantic soft cues are shared across all levels, i.e., across different skip connection levels. corresponding same.

[0159] Subsequently, semantic soft prompts Apply channel-level weighting:

[0160]

[0161] in, This indicates element-wise multiplication.

[0162] Finally, use one Convolutional layer pairs Channel compression and feature fusion are performed to adapt the features extracted by the skip connection k-th layer image recovery encoder. and with After fusion Convolutional layers generate the final collaborative hints. The final collaborative prompt The features are injected into the k-th layer via skip connections.

[0163]

[0164] in This indicates element-wise addition.

[0165] In this invention, a joint loss function comprising three core components is designed for the model training phase:

[0166]

[0167] in, use Loss-aligned image restoration effect The loss function specified in Yolov7 is used as the target detection loss. To constrain the consistency loss of the semantic soft cues related to the object detection task, and These are hyperparameters, set to 0.2 and 0.05 respectively.

[0168] Consistency loss of prompt content This is used to constrain the semantic soft cues at different skip connection levels to maintain similarity in the semantic space. Since the semantic soft cues are learnable parameters, and the semantic soft cues at different levels differ in channels and resolution, they should all serve the same downstream task (object detection). This is to ensure that the core semantic content of prompts at different granularities is basically consistent.

[0169] Semantic soft cues for skip connections at layer k Global average pooling is performed to obtain a channel-level feature vector. .because and Number of channels and Unlike other methods, a projection layer is usually required. We map them to the same embedding space. Finally, we calculate the cosine similarity between the average feature vectors of neighboring soft cues:

[0170]

[0171] By minimizing the cosine similarity between the average feature vectors of semantic soft cues from adjacent layers, the semantic consistency of the information learned by the semantic soft cues related to the object detection task is ensured across different depth layers.

[0172] Research materials:

[0173] The method proposed in this invention is compared with eight existing methods, including the object detector Yolov7 (2023) trained on sharp images and the object detector trained on degraded images. (2023) The networks that perform recovery before detection include PromptIR-Yolov7 (2023) and DCP-Yolov7 (2010). The networks that jointly optimize recovery and detection include TogetherNet (2022), RDMNet (2024), DTRDNet (2024), and End-to-End Net (2025). For the ternary dataset (denoted as VOC-Deg dataset) constructed in this invention, during the testing phase, the test set of the ternary dataset is divided into three degradation types according to fog, rain, and dark light: artificially synthesized fog dataset VOC-Fog-test, artificially synthesized rain dataset VOC-Rain-test, and artificially synthesized dark light dataset VOC-Dark-test. At the same time, the real-world fog dataset RTTS is used for testing.

[0174] Figure 4 and Figure 5 The results of this invention on the artificially synthesized fog dataset VOC-Fog-test and the real-world fog dataset RTTS are presented in a visually intuitive way. Figure 6 The results are from the detection dataset VOC-Rain-test, which is a synthetic rainy day dataset.

[0175] Tables 1 and 2 show the average precision (mAP) comparison of each method on the artificial fog dataset VOC-Fog-test and the real-world fog dataset RTTS, respectively. Bold indicates the best result, and underline indicates the second best result.

[0176] Table 1

[0177]

[0178] Table 2

[0179]

[0180] It can be seen that the method of this invention achieves the best results on the VOC-Fog-test, outperforming the state-of-the-art method DTRDNet by 0.73%. Specifically, it achieves the best results in the Car and Bus categories, and the second-best results in the Peron and Bicycle categories. Compared with PromptIR-Yolov7, The mAP metric is 0.69% higher. In the method of restoring before detection, the restored clear image is not necessarily good for detection, thus leading to a decrease in the metric. On a real-world foggy dataset, the method of this invention achieves suboptimal results on RTTS, with the best result in the Car category and suboptimal results in the Bicycle category.

[0181] Table 3 presents a comparison of the mean accuracy (mAP) of each method on the artificially synthesized rain dataset VOC-Rain-test. Bold text indicates the best results, and underlined text indicates the second-best results.

[0182] Table 3

[0183]

[0184] As can be seen, our proposed method achieved suboptimal results on this dataset. Specifically, our method achieved the best results in the Bicycle category, exceeding the suboptimal mAP by 1.19%, and also achieved suboptimal results in the Person category. Furthermore, Table 3 shows that using the method of recovery before detection did not significantly improve detection accuracy.

[0185] Table 4 presents a comparison of the mean accuracy (mAP) of each method on the artificially synthesized dark light dataset VOC-Dark-test. Bold text indicates the best results, and underlined text indicates the second-best results.

[0186] Table 4

[0187]

[0188] As can be seen, compared with other methods, our method achieved the highest mean accuracy (mAP), with an improvement of 0.44% compared to the End-to-End Net network. Specifically, it achieved the best results in the Car category, and the second-best results in the Bicycle, Motorbike, and Bus categories.

[0189] It can be seen that the method proposed in this invention has achieved substantial improvements, especially in the artificially synthesized fog dataset VOC-Fog-test, where its advantages are even more obvious.

[0190] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many improvements and changes under the guidance of the present invention without departing from the spirit of the present invention, and these improvements and changes are all within the protection scope of the present invention.

Claims

1. A method for detecting degraded objects in a scene based on cue learning, characterized in that, The target detection result is obtained by processing the degraded image using a degraded scene target detection network; the degraded scene target detection network includes an image restoration encoder, an image restoration decoder, a degradation-aware hard cueing module DAPM, a detection task-guided semantic soft cueing module DGSPM, and a target detector; First, a ternary dataset containing degraded images, degradation type labels, and degradation text descriptions for various degradation scenarios is constructed. The ternary dataset is constructed by applying fog, rain, and low light processing to clear images respectively to generate degraded images containing three degradation types and their degradation type labels. Then, a visual language model is used to perform semantic understanding on the degraded images to generate corresponding degradation text descriptions. The degradation scene object detection network is trained using the aforementioned ternary dataset. During the training phase, the degraded image is first input into the image restoration encoder for feature extraction. The features extracted by the image restoration encoder at each level of the skip connection are then passed to the image restoration decoder via skip connections. During the skip connection process, the semantic soft cueing module DGSPM, guided by the detection task, constructs a corresponding multi-scale learnable cue pool for each skip connection layer. The multi-scale learnable cue pool at the same skip connection level is weighted and fused with the features extracted by the image restoration encoder, thereby introducing semantic soft cueing related to the object detection task at different skip connection levels. The degradation-aware hard cueing module DAPM uses the degradation text descriptions corresponding to the degraded images in the ternary dataset, extracts text embedding vectors through a pre-trained text encoder, and jointly encodes the text embedding vectors with the degradation type labels corresponding to the degraded images in the ternary dataset to form a cross-scene universal degradation-aware hard cue. Based on the degradation-aware hard cue, channel attention weights are calculated and then fused and encoded with the semantic soft cue. These channel attention weights are then used as features of each level of skip connections and input to the image restoration decoder to generate a restored image. Finally, the restored image is input into the target detector, and the target detection result is output.

2. The degraded scene target detection method based on cue learning according to claim 1, characterized in that, The specific steps are as follows: Step 1) Construct a ternary dataset for network training and validation. The ternary dataset contains degraded images for various degradation scenarios, degradation type labels, and degradation text descriptions, where each degraded image corresponds to a set of degradation type labels. and its matching degenerate text description The ternary dataset covers three typical degradation scenarios: foggy, rainy, and low-light conditions. Based on the open-source VOC dataset, different degradation scenario models are applied to the clear image I to generate corresponding degraded images. Simultaneously, a corresponding degenerate text description is constructed to form the ternary dataset; Step 2) Construct a degraded scene object detection network based on cue learning to process the input degraded image. The process is performed to obtain the final target detection result. ,include: Step 2-1) For degraded images The image restoration encoder Features are extracted layer by layer through a multi-layer structure, and the encoder features of the corresponding layer are passed to the image restoration decoder through skip connections. The features of the output of the k-th layer encoder skip connection Represented as: ； in, , indicating three jump connections; Step 2-2) For the features output by each layer encoder The multi-scale learnable cue pool built in the semantic soft cue module DGSPM guided by the detection task. The semantic soft cueing module DGSPM, guided by the detection task, obtains the semantic soft cues related to the target detection task required for the k-th layer skip connection. : ； in, This indicates the result processed by the Semantic Soft Prompt Module (DGSPM) guided by the detection task. Steps 2-3) Each degraded image in the three-dimensional data set constructed in step 1). All are associated with a set of degradation type labels Corresponding degenerate text description The degradation type label is displayed through the Degradation Aware Hard Cueing Module (DAPM). and degenerate text description Separate encoding and fusion are performed to output degradation-aware hard cues. : ； in, This indicates the result processed by the Degradation Aware Hard Cueing Module (DAPM). Steps 2-4) Hard cue for degradation perception As conditional information, semantic soft prompts Channel-level modulation is performed to achieve soft and hard cue fusion, ultimately obtaining the features of the k-th layer skip connection as input to the image reconstruction decoder. : ； in, This indicates the result of the hardware / software prompt fusion mechanism. Steps 2-5) Features , The image is injected into the image reconstruction decoder of each skip connection layer, ultimately yielding the reconstructed image relevant to the object detection task. : ； in, This represents the result after processing by the image restoration decoder; Steps 2-6) Use the publicly available object detector YOLOv7 to reconstruct the image. Perform object detection to obtain the final object detection result. : ； in, This represents the result after processing by the object detector Yolov7.

3. The degraded scene target detection method based on cue learning according to claim 2, characterized in that, Step 1) involves constructing the ternary dataset, which includes two steps: degraded image generation and degraded text description construction. Step 1-1) Degraded Image Generation: For the three different degradation scenarios, foggy, rainy, and low-light degradation processing are applied to the clear image I in the VOC dataset to generate corresponding degraded images for the three degradation types. Degraded images in a three-dimensional data set Depending on the degradation scene, including foggy images Rainy day images and low-light images ; An atmospheric scattering model is used to process the clear image I to generate a foggy image. A rainy day image is generated by processing a clear image I using a physical simulation method. A brightness scaling factor is introduced to suppress the global energy of the sharp image I, and gamma correction is applied to simulate the nonlinear characteristics of the camera response function to generate a low-light image. ; Steps 1-2) Construct degraded text description data and perform batch processing on degraded images using a visual language model; for each degraded image By inputting preset prompt words into the visual language model Generate a degraded text description corresponding to the degraded image. : ； in, Represents a visual language model. This refers to input prompts that guide the visual language model to generate degenerate text descriptions.

4. The degraded scene target detection method based on cue learning according to claim 2, characterized in that, In step 2-2), the semantic soft cueing module DGSPM guided by the detection task is designed to utilize the features extracted by the image recovery encoder in the k-th layer skip connection stage. With instantiated multi-scale learnable cue pool Constructing semantic soft cues relevant to object detection tasks ,include: S2-2-1 Feature maps extracted from the image recovery encoder corresponding to each level of the skip connection. Each feature map is configured with a multi-scale learnable cue pool module; in the case of triple-hop connections, three cue pools are configured, and each cue pool is initialized with randomized initial parameters; each cue pool... Three sets of learnable cue parameters with different receptive ranges are introduced internally to model global contextual information, region semantic information, and local detail information for object detection, respectively: ； in, This means that each cue pool contains learnable cue blocks. The total number is 21, where k represents the k-th layer skip connection; The specific settings are: global hints ( ): Consists of 1 full resolution The parameter blocks are responsible for capturing the overall scene structure; region hints ( ) consists of 4 dimensions The parameter blocks are composed of four quadrants of the image, responsible for capturing medium-scale regional features; local cues ( ): Consists of 16 dimensions The parameter blocks are composed of fine-grained grids corresponding to the image, which are responsible for capturing small targets and high-frequency texture details; S2-2-2, Recovering the Feature Map Extracted by the Encoder Using Skip Connections to the k-th Layer Image Generate and learnable hint blocks The total number of attention weight vectors with a length of 21 corresponds to the total number of vectors. : ； in, This represents the result after global average pooling. This represents the result after processing by the fully connected layer. This represents the result after normalization by the activation function layer; Using this attention weight vector For the corresponding learnable hint blocks Perform independent weighting to generate activation cue features. : ； in, Representing global, regional, and local prompts, It is a 21-dimensional vector. This represents element-wise multiplication; Subsequently, Concatenate the components according to spatial quadrant order to construct the semantic soft cue for the final k-th jump connection. : ； in, The splicing method is: global prompts are directly generated by... Composition, area prompts will include 4 Strictly follow the spatial stitching order of top left-top right-bottom left-bottom right, with local hints to include 16 [items / items]. according to The grid is then spatially stitched together in sequence; subsequently, the three elements are... Perform channel-level splicing.

5. The degraded scene target detection method based on cue learning according to claim 2, characterized in that, In steps 2-3), the Degradation Aware Hard Cueing Module (DAPM) is designed to utilize the degradation image constructed in step 1). Corresponding degradation type tag With degenerate text description Generate degradation-perceived hard cues that include degradation type, degradation intensity, and visual impact information. ,include: S2-3-1, For discrete degenerate type labels One-hot encoding is used to generate degenerate type vectors. : ； in, Represents one-hot encoding, for Types of degradation, Converted into a dimensional vector Degraded image The corresponding degenerate type index position is 1, and the positions of all other degenerate types are 0; S2-3-2, Regarding degenerate text descriptions The text is processed using a CLIP pre-trained text encoder and normalized using L2. Convert to fixed-dimensional text embedding vectors , It contains information on the current degradation intensity and visual impact of the degraded image, used to guide the restoration and detection of degraded images across different scenes: ； in, , This represents the CLIP pre-trained text encoder. This indicates that the vector is subjected to L2 normalization, which means dividing the vector by its L2 norm so that the length is 1 while the direction remains unchanged; S2-3-3, Degenerate type vector With text embedding vectors Perform joint encoding, where, , The process is as follows: First, the degenerate type vector is processed using two fully connected layers (FC) and one activation function layer (GeLU). Mapped twice to the text embedding vector Same channel dimension: ； in, This represents a mapping network containing a non-linear activation function, consisting of two fully connected layers (FC) and one activation function layer (GeLU). Subsequently, the aligned degenerate type vector With text embedding vectors The results are summed and then passed through L2 normalization and a channel downsampling convolutional layer to generate the final degradation-aware hard cue. : ； in, This indicates element-wise addition. This indicates that the vector has been subjected to L2 normalization. This represents a channel downsampling convolutional layer.

Images