An image feature enhancement method based on scene and target decoupling

Abstract

The application discloses an image feature enhancement method based on scene and target decoupling. The method comprises the following steps: constructing an enhancement model, using an adversarial network to measure the difference between a degraded image and a clear image in scene features, guiding the model to enhance scene features, and constructing an image reconstruction loss to constrain the enhanced image to be consistent with the original image in brightness, color, texture and structure perception. On the basis of scene feature enhancement, the target in the separated image is analyzed by using a downstream task analysis network, so that the target and the scene are decoupled. For target detection, the target feature difference of the detector response is used as a constraint for training. For semantic segmentation, the accuracy of the target segmentation result is used as a measure of the degradation of the target feature, so that the constraint for enhancing the target feature is realized. The application can enhance the image features in bad weather and improve the detection performance of downstream tasks such as target detection and semantic segmentation.

Description

Technical Field

[0001] This invention relates to an image feature enhancement method based on scene and target decoupling. Background Technology

[0002] Currently, video surveillance is increasingly widely used in various fields, becoming an important source of information. With the development of artificial intelligence technology, video surveillance is gradually becoming more intelligent. However, complex and diverse real-world environments are often affected by severe weather conditions and extreme lighting, leading to image quality degradation, such as low contrast, blurred target colors and texture details, which fails to meet the needs of subsequent video content analysis. The impact of severe weather and extreme lighting on image quality seriously hinders the analytical performance of downstream tasks. In foggy weather, particles in the air scatter light, causing the color and texture details of targets to degrade. Under low-light conditions, the ambient light reflected by objects is weak, reducing the light signal captured by the camera, resulting in low image brightness, weak texture, and unclear target features. Similar problems exist under strong light conditions. These degradation factors limit the application of intelligent algorithms in real-world scenarios. To solve this problem, researchers are constantly exploring image enhancement techniques to improve the quality of degraded images and meet the needs of downstream task analysis.

[0003] Currently, image enhancement methods are mainly divided into two categories: physical model-based methods and data-driven deep learning-based methods. Physical model-based methods require physically modeling specific degrading factors and then eliminating these factors by solving for the feature components in the physical model. For example, in low-light scenes, the illumination component can be solved based on the retinal cortex theory, and the brightness can be remapped; in dense fog, the illumination and transmittance components can be solved, and inversely calculated based on an atmospheric scattering model. These methods generally produce good visual effects, but their application remains limited because some degrading factors cannot be modeled.

[0004] Data-driven approaches primarily rely on generative models, employing supervised or semi-supervised methods to drive data processing. These methods implicitly model the degradation process using loss functions, thereby improving image quality. The advantage of this approach is its ability to handle complex degradation scenarios without requiring in-depth knowledge of the physical mechanisms underlying the degradation process.

[0005] While these methods have helped improve machine analysis performance, the improvement remains limited. The main reason is that many degrading factors in images still exist that cannot be explicitly modeled, affecting the machine analysis model's perception and interpretation of image content semantics. The aforementioned enhancement methods sometimes amplify these problems, resulting in limited improvement in downstream tasks such as object detection and semantic segmentation. In practical applications, target analysis tasks in surveillance systems focus more on target features. The difference between the pixel domain and the target feature domain limits the performance of target analysis algorithms. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention considers the performance enhancement of image enhancement for machine analysis tasks. Image features are enhanced through implicit modeling using deep learning models. By decoupling and processing the scene and target in stages, adversarial networks are used to enhance global scene features, and feature differences in the responses of downstream task analysis models are utilized to enhance target features. This achieves image enhancement in various harsh environments such as dense fog and low light, improving the performance of tasks such as target detection and semantic segmentation in these scenarios.

[0007] According to one aspect of the present invention, an image feature enhancement method based on scene and target decoupling is provided, characterized by comprising:

[0008] A) Train the augmentation network to enhance image scene features and obtain the augmentation model. This step includes:

[0009] A1) Extract the image from the degraded image domain, and obtain the preprocessed image after preprocessing such as dehazing or illumination restoration. Extracting sharp images from sharp image domains ;

[0010] A2) Preprocess the image Input to the augmented network In the middle, through the network Get enhanced image Then, using PatchGAN as the discriminator, the preprocessed image is... Image enhancement and clear images Input into PatchGAN for adversarial training;

[0011] A3) Introducing brightness, color, texture, and structure-aware constraints to enhance images. This makes it consistent with the preprocessed image in terms of content. Maintain consistency.

[0012] B) Constructing an Enhanced Model The converged enhancement model Load the parameters into the augmentation model Meanwhile, freeze the enhancement model. The parameters are used to continue training the augmentation model. This enhances the features of the target in the image. The steps include:

[0013] B1) Extract the image from the degraded image domain, and obtain the preprocessed image after preprocessing such as dehazing or illumination restoration. Extracting sharp images from sharp image domains ;

[0014] B2) Preprocess the image Input augmentation model To obtain enhanced images At the same time, preprocessed images Input augmentation model To obtain enhanced images .

[0015] B3) For object detection, enhance the image with labeled information. and clear images The input is fed into a pre-trained Faster R-CNN detector, and training is performed using the target feature differences in the detector's response as constraints; for semantic segmentation, the enhanced image with label information is used. The data is input into the pre-trained Mask2Former segmentation model, and the accuracy of the target segmentation results is used as a measure of target feature degradation and as a constraint for target feature enhancement.

[0016] B4) Introducing structure-aware constraints to enhance images To make it consistent with in content Maintain consistency.

[0017] C) During the testing phase, the degraded image is preprocessed, such as by dehazing or illumination restoration, and then input into the trained enhancement model. In this process, the enhanced results of the present invention are obtained. . Attached Figure Description

[0018] Figure 1 This is a flowchart of an image feature enhancement method based on scene and target decoupling according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of scene difference measurement according to an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of a target difference measurement according to an embodiment of the present invention. Detailed Implementation

[0021] According to one aspect of the present invention, an image feature enhancement method based on scene and target decoupling is provided. The method of the present invention enhances image features, improving the performance of downstream tasks such as target detection and semantic segmentation.

[0022] To achieve image enhancement and improve the performance of downstream task analysis, this invention proposes an image feature enhancement method based on scene and target decoupling, characterized by including:

[0023] A) Train an augmentation network to enhance image scene features, such as Figure 1 As shown in the scene feature enhancement module, the final enhanced model is obtained. ,Right now Figure 1 Generator 1 in the code. This step includes:

[0024] A1) Extract the image from the degraded image domain, and obtain the preprocessed image after preprocessing such as dehazing or illumination restoration. ,correspond Figure 1 Image enhancement based on apparent features. Extracting sharp images from the sharp image domain. ,correspond Figure 1 Clear images in the image;

[0025] A2) Preprocess the image Input to the augmented network In the middle, through the network Get enhanced image ,correspond Figure 1 The scene feature enhancement image is then used. Next, PatchGAN is used as the discriminator to analyze the preprocessed image. Image enhancement and clear images The input is fed into PatchGAN for adversarial training, such as... Figure 2 As shown, PatchGAN takes a sharp image and a degraded image as input to determine whether an image is a sharp or degraded scene, thus achieving scene difference measurement. The degraded image in the input corresponds to the image... and The input sharp image corresponds to the image ;

[0026] A3) Introduces brightness, color, texture, and structure enhancements to the image. This makes it consistent with the preprocessed image in terms of content. Maintain consistency.

[0027] B) Constructing an Enhanced Model ,correspond Figure 1 Generator 2 in the middle will generate the converged augmentation model Load the parameters into the augmentation model In this process, weight sharing is achieved, while the augmentation model is frozen. The parameters are used to continue training the augmentation model. Enhance image target features, such as Figure 1 The target feature enhancement module is shown in the diagram. This step includes:

[0028] B1) Extract the image from the degraded image domain, and obtain the preprocessed image after preprocessing such as dehazing or illumination restoration. ,correspond Figure 1 Image enhancement based on apparent features. Extracting sharp images from the sharp image domain. ,correspond Figure 1 Clear images in the image;

[0029] B2) Preprocess the image Input augmentation model To obtain enhanced images At the same time, preprocessed images Input augmentation model To obtain enhanced images ,correspond Figure 1 Target feature enhancement image.

[0030] B3) will enhance the image and clear images Input to Figure 1 In the machine analysis network shown, for object detection, the enhanced image with labeled information is used. and clear images The input is fed into a pre-trained Faster R-CNN detector, and the training is performed using the target feature differences in the detector's response as constraints. Figure 3 As shown; for semantic segmentation, the enhanced image with label information is used. The data is input into the pre-trained Mask2Former segmentation model, and the accuracy of the target segmentation results is used as a measure of target feature degradation and as a constraint for target feature enhancement.

[0031] B4) Introducing structure-aware constraints to enhance images To make it consistent with in content Maintain consistency.

[0032] C) During the testing phase, the degraded image is preprocessed, such as by dehazing or illumination restoration, and then input into the trained enhancement model. In this process, the enhanced results of the present invention are obtained. .

[0033] According to a further embodiment of the present invention, in steps A) and B) above, the enhanced model and The network structures are similar, both using the ResNet architecture. It consists of two convolutional layers that change the number of channels in the feature map, and four residual blocks that do not change the feature map size. The convolutional layers that change the feature map channels contain 20 kernels, transforming the input three-channel image into a feature map with 20 channels, while maintaining the same height and width as the original input image. Following this are four residual blocks, each containing two convolutional layers. Finally, there is a convolutional layer that changes the number of channels in the feature map, containing three kernels, used to reduce the number of channels in the feature map to three. The output of this layer is the final output of the model. The activation function is ReLU, and the kernel size is 3*3. Overall, this model is a very lightweight network. In step A), the training phase uses Adam as the optimizer, with a batch size of 8 and 200 training epochs. In step B), the batch size is set to 1 and the training phase is 50 epochs. The learning rate is set to 2e-4 and decreases linearly with the number of iterations.

[0034] According to a further embodiment of the present invention, in step A2) above, in order to zoom in and enhance the image... and clear images The scene feature distance is used to implicitly measure the difference between degraded and clear images using Patch GAN, and an adversarial loss is introduced:

[0035]

[0036]

[0037] in, For the adversarial loss of generator 1, For the loss of the discriminator, Indicates the distribution of degraded images. These are images sampled from this distribution. Indicates to The expected value, similarly Indicates the distribution of clear images. This represents an image sampled from this distribution. Indicates to The expected value represents the distribution of the degraded image, and is a sample from that distribution. Representing an image Enhanced model Enhanced image, This is the output of the discriminator, and the content within parentheses is the input of the discriminator. This represents logarithmic operations. To enable the discriminator to better distinguish between degraded and sharp image domains, the discriminator is trained by inputting not only the enhancement model... The generated image also takes the original image as input.

[0038] According to a further embodiment of the present invention, in step A3) above, in order to keep the scene content information of the image unchanged before and after enhancement, constraints are imposed on the brightness, color, structure, and texture gradient of the scene before and after enhancement. The brightness constraint ensures that the enhanced image as a whole is not too bright or too dark, using the maximum channel value of the image as the constraint.

[0039]

[0040] in Indicates brightness loss. This represents the augmentation model, which corresponds to... . This represents an image, specifically a preprocessed image. , This corresponds to the enhanced image. , Representing an image The One channel, To find the maximum value function, and These represent the maximum values ​​of the three channels of the enhancement model's output and the input image, respectively. Number of pixels This is the Manhattan norm. Additionally, because the image may have local color shifts during enhancement, the color information of the enhanced image is constrained as follows:

[0041]

[0042] in Indicates color loss. and These are the functions for finding the maximum value and the function for finding the minimum value, respectively. This represents an image, specifically a preprocessed image. , , , These represent the mean values ​​of the R, G, and B channels of the input image, respectively. To ensure consistent pixel grayscale adjustment across different regions of the image and suppress local artifacts, a first-order difference constraint on the pixel grayscale difference between the input and output images is introduced:

[0043]

[0044] in Indicates texture loss. This represents the augmentation model, which corresponds to... , This represents an image, specifically a preprocessed image. , This corresponds to the enhanced image. , It is the first-order difference operator for images, which measures the difference between adjacent pixels. To constrain the structural consistency of the output image, a pre-trained VGG19 network is used, with the feature maps of each activation layer of the VGG19 network serving as constraints to ensure that the output image maintains overall structural consistency with the input image.

[0045]

[0046] in Indicates structural loss. This represents the augmentation model, which corresponds to... , . This represents an image, specifically a preprocessed image. , This corresponds to the enhanced image. , Indicates the number of activation layers. Represents the length-width product of the feature image. This indicates the VGG19 network's... One activation layer, Indicates the first The response characteristics of each activation layer This represents the Euclidean norm. Compared to pixel-level constraints, this feature, extracted through continuous convolution filtering, is better able to reflect structurally perceptual differences involving relationships between pixels.

[0047] According to a further embodiment of the present invention, in step B3) above, as Figure 3 As shown, the clear image corresponds to the image. Degraded image corresponds to enhanced image For object detection, augmentation models Enhanced output image Global features are obtained through the Faster R-CNN detector backbone network VGG16, based on the image. The label information is obtained from the candidate bounding boxes output by the RPN module of the detector. Based on the candidate bounding boxes, the corresponding target response features are found on the global features. The target features of different sizes are pooled to the same size by the ROI pooling module, and then passed through a fully connected layer in RCNN. Obtain target feature vector Similarly, a clear image can be obtained. Target feature vector Obtain enhanced images and clear images After obtaining the target feature vector, a target feature alignment loss is introduced:

[0048]

[0049] in This indicates the loss as a measure of the difference between the target and the target. Representing an image The response in the detector Class target feature vector, express The response in the detector Each class of target feature vectors is obtained by weighting the historical cumulative value and the current value.

[0050] For semantic segmentation to enhance images As input, the accuracy of the object segmentation result from the Mask2Former model is used as a measure of object feature degradation, which is then transformed into object-aware feature enhancement loss. Mask2Former processes pixel category and mask prediction separately; the mask loss only predicts foreground and background, while the pixel category loss is responsible for predicting the category. For semantic segmentation... Defined as:

[0051]

[0052] in It uses the Dice coefficient to measure the similarity between the predicted result and the true label. It is a binary cross-entropy loss, and Combine the jointly predicted foreground mask region. It predicts pixel categories using multi-class cross-entropy loss.

[0053] According to a further embodiment of the present invention, in step B4) above, the structure-aware loss mentioned in step B2) is utilized. Constraint-enhanced images To make it consistent with in content Maintain consistency.

[0054] It should be noted that the above-disclosed examples are merely specific embodiments of the present invention. Variations conceived by those skilled in the art based on the technical concepts provided by the present invention should fall within the protection scope of the present invention.

Claims

1. An image feature enhancement method based on scene and target decoupling, characterized in that... Includes the following steps: A) Train the augmentation network to enhance image scene features and obtain the augmentation model. ,include: A1) Extract the image from the degraded image domain, and obtain the preprocessed image after dehazing or illumination restoration preprocessing. Extracting sharp images from sharp image domains ; A2) Preprocess the image Input to the augmented network In the middle, through the network Get enhanced image Then, using PatchGAN as the discriminator, the preprocessed images are... Image enhancement and clear images Input into PatchGAN for adversarial training; A3) Introducing brightness, color, texture, and structure-aware constraints to enhance images. This makes it consistent with the preprocessed image in terms of content. Maintain consistency. B) Constructing an Enhanced Model The converged enhancement model Load the parameters into the augmentation model Meanwhile, freeze the enhancement model. The parameters are used to continue training the augmentation model. To enhance the features of the target in the image, this step includes: B1) Extract the image from the degraded image domain, and obtain the preprocessed image after dehazing or illumination restoration preprocessing. Extracting sharp images from sharp image domains ; B2) Preprocess the image Input augmentation model To obtain enhanced images At the same time, preprocessed images Input augmentation model To obtain enhanced images , B3) For object detection, enhance the image with labeled information. and clear images The input is fed into a pre-trained Faster R-CNN detector, and training is performed using the target feature differences in the detector's response as constraints; for semantic segmentation, the enhanced image with label information is used. The data is input into a pre-trained Mask2Former segmentation model, where the accuracy of the target segmentation results is used as a measure of target feature degradation and as a constraint for target feature enhancement. B4) Introducing structure-aware constraints to enhance images To make it consistent with in content Maintain consistency. C) During the testing phase, the degraded image is preprocessed by dehazing or illumination restoration and then input into the trained enhancement model. In the process, enhanced results are obtained. , In step A3) above, in order to keep the scene content information of the image unchanged before and after enhancement, constraints are imposed on the brightness, color, structure, and texture gradient of the scene before and after enhancement. The brightness constraint of the scene can ensure that the enhanced image as a whole will not be too bright or too dark. The maximum channel value of the image is used as the constraint. in: Indicates brightness loss. This represents the augmentation model, which corresponds to... , This represents an image, specifically a preprocessed image. , This corresponds to the enhanced image. , Representing an image The One channel, To find the maximum value function, and These represent the maximum values ​​of the three channels of the enhancement model's output and the input image, respectively. Number of pixels For Manhattan norm, Because the image may have local color cast issues during the enhancement process, the color information of the enhanced image is constrained: in Indicates color loss. Indicates color loss. and These are the functions for finding the maximum value and the function for finding the minimum value, respectively. This represents an image, specifically a preprocessed image. , , , These represent the mean values ​​of the R, G, and B channels of the input image, respectively. To ensure consistent pixel grayscale adjustment across different regions of the image and suppress local artifacts, a first-order difference constraint is introduced between the pixel grayscale differences of the input and output images: in: Indicates texture loss. This represents the augmentation model, which corresponds to... , This represents an image, specifically a preprocessed image. , This corresponds to the enhanced image. , It is a first-order difference operator for images, which measures the difference between adjacent pixels. To ensure the structural consistency of the output image, a pre-trained VGG19 network is used, with the feature maps of each activation layer in the VGG19 network serving as constraints to ensure that the overall structure of the output image remains consistent with that of the input image. in: Indicates structural loss. This represents the augmentation model, which corresponds to... , This represents an image, specifically a preprocessed image. , This corresponds to the enhanced image. , Indicates the number of activation layers. Represents the length-width product of the feature image. This indicates the VGG19 network's... One activation layer, Indicates the first The response characteristics of each activation layer Describes the Euclidean norm. Therefore, compared to pixel-level constraints, this feature, extracted through continuous convolutional filtering, is better able to reflect structurally perceptual differences involving relationships between pixels. In step B3) above, for object detection, for the augmentation model Enhanced output image Global features are obtained after passing through the Faster R-CNN detector backbone network, based on the image. The label information is obtained from the candidate bounding boxes output by the RPN module of the detector. Then, based on the candidate bounding boxes, the features corresponding to the target response are found on the global features, and then passed through a fully connected layer. Obtain target feature vector And obtain a clear image in the same way. Target feature vector , Next, target feature alignment loss is introduced: in: This indicates the loss as a measure of the difference between the target and the target. Representing an image The response in the detector Class target feature vector, Represents a clear image The response in the detector Class target feature vector, Each type of target feature vector is obtained by weighting the historical cumulative value and the current value. For semantic segmentation to enhance images As input, the accuracy of the object segmentation result from the Mask2Former model is used as a measure of object feature degradation, which is then transformed into an object-aware feature enhancement loss. Mask2Former processes pixel category and mask prediction separately; the mask loss only predicts foreground and background, while the pixel category loss is responsible for predicting the category. For semantic segmentation... Defined as: in: It uses the Dice coefficient to measure the similarity between the predicted result and the true label. It is a binary cross-entropy loss, which is related to... Combined with the jointly predicted foreground mask region, It predicts pixel categories using multi-class cross-entropy loss. In step B4) above, structure-aware loss is used. Constraint-enhanced images To make it consistent with in content Maintain consistency.

2. The image feature enhancement method according to claim 1, characterized in that: Augmentation Model and The network structures are similar, both using the ResNet architecture, and each includes: A convolutional layer with 20 convolutional kernels modifies the feature map channels, transforming the input three-channel image into a feature map with 20 channels while maintaining the same height and width as the original input image. Following the convolutional layer that changes the feature map channels are four residual blocks that do not change the feature map size. Each residual block contains two convolutional layers. After four residual blocks that do not change the feature map size, there is a convolutional layer that changes the number of channels in the feature map. It contains three convolutional kernels to reduce the number of channels in the feature map to three. The output of this convolutional layer is used as the final output of the model. Its activation function is ReLU and the kernel size is 3*3.

3. The image feature enhancement method according to claim 1, characterized in that: Step A) uses Adam as the optimizer during the training phase, with a batch size of 8 and 200 training epochs. Step B) Set the data batch size to 1, train for 50 rounds, and set the learning rate to 2e-4, which decreases linearly with the number of iterations.

4. The image feature enhancement method according to claim 1, characterized in that: In step A2) above, in order to zoom in and enhance the image and clear images The scene feature distance is used to implicitly measure the difference between degraded and clear images using Patch GAN, and an adversarial loss is introduced: in, To enhance the model The losses in the fight, For the loss of the discriminator, Indicates the distribution of degraded images. These are images sampled from this distribution. Indicates to Expected value Indicates the distribution of clear images. This represents an image sampled from this distribution. Indicates to Expected value Representing an image Enhanced model Enhanced image, This is the output of the discriminator; the content within parentheses is the input of the discriminator. Represents logarithmic operations. In order to enable the discriminator to better distinguish between degraded and clear image domains, the discriminator is trained by not only inputting the enhancement model. The generated image also takes the original image as input.

Images