Shadow weakening method based on weak supervision image conversion generative adversarial network
Adversarial network is generated through weakly supervised image transformation, and auxiliary attention classifiers and dynamic normalization functions are used, combined with residual modules and mixed loss functions, and the problem of shadow interference in complex scenarios is solved, improving the accuracy of road disease target detection and data quality.
Patent Information
- Application Number
- CN202410980344.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
In road disease target detection, light changes in complex scenarios lead to complex shadows, affecting feature extraction. The existing technology is difficult to effectively reduce shadow interference, resulting in a degradation of the detection model performance.
The shadow weakening method based on weak-supervised image transformation generates an adversarial network, and the adversarial network model is constructed, and an auxiliary attention classifier and dynamic adaptation normalization function are introduced, combining residual modules and mixed shadow weakening loss function. Optimize shadow weakening effect.
It significantly improves the model's recognition accuracy and processing ability of shadow features, reduces shadow interference, enhances the resistance of road disease target detection models in complex scenarios, and improves detection accuracy and image data quality.
Smart Images

Figure CN118941485A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of road damage target detection, and in particular to a shadow weakening method based on a weakly supervised image conversion generative adversarial network. Background Art
[0002] In the detection of road damage targets, the difference in illumination changes under complex road scenes will cause uneven shadows on the road surface. The shadows in the road background complicate the features of the damage to a certain extent, affecting the feature extractability of the damage target. In the training of the road damage target detection model, the limitation of the data annotation method causes the rectangular box annotation file to include the shadows near the road damage into the target positive sample, which reduces the performance of the target detection model and may also cause some shadow edges that are very similar to the road damage features to be mistakenly identified as damage. Therefore, shadows can significantly affect the feature extraction performance of the target detection model and may cause erroneous detection results, thereby affecting the accuracy of computer vision tasks such as target detection, target tracking, and image registration.
[0003] In order to improve the accuracy of road damage target detection, it is urgent to propose a shadow weakening method based on a generative adversarial network to address the problems such as road shadows reducing the quality of image data and indirectly affecting the loss of feature information. This method can realize weakly supervised shadow weakening training based on fewer shadow paired images, and enhance the ability of subsequent road damage target detection models to resist this interference factor in complex scenarios. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a shadow weakening method based on weakly supervised image conversion generative adversarial network. The present invention conducts research on the problems that road shadows reduce the quality of image data and indirectly affect the loss of feature information. Weakly supervised shadow weakening training is realized based on paired images with fewer shadows, thereby enhancing the ability of road disease target detection models to resist this interference factor in complex scenes.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] The present invention provides a shadow weakening method based on weakly supervised image conversion generative adversarial network, comprising the following steps:
[0007] S1: Construct an unsupervised image conversion generative adversarial network model, which introduces a dynamic adaptive normalization function through an auxiliary attention classifier to enhance the recognition and processing capabilities of shadow features;
[0008] S2: Introducing a residual module with improved design, by introducing skip connections between convolutional layers, the efficiency of information flow and gradient propagation in the deep network is improved to alleviate the gradient vanishing problem of the generative adversarial network model;
[0009] S3: Construct a mixed shadow weakening loss function to complete the construction of the generative adversarial network model;
[0010] S4: The difference between the image generated by the generative adversarial network and the real image is combined with the feedback of the auxiliary attention classifier to optimize the shadow weakening effect and achieve accurate weakening of road disease shadows.
[0011] Furthermore, S1 specifically includes:
[0012] S1-1: Design auxiliary attention classifier: Construct an auxiliary attention module, which is integrated into the generator and discriminator of the generative adversarial network. The auxiliary attention module uses class activation mapping technology to generate a spatial attention map through global average pooling and global maximum pooling to highlight the areas that are more important to the target features, thereby guiding the model to pay more attention to the shadow features;
[0013] S1-2: Introduce a dynamic adaptive normalization function: Select a dynamic adaptive instance normalization method and dynamically adjust the normalization parameters to adapt to the statistical characteristics of different input data, so that the generative adversarial network model can adaptively adjust the weight parameters, optimize the processing of shadow features, and improve the model's recognition accuracy and processing capabilities for shadow areas.
[0014] Furthermore, in S1-1, the spatial attention map is obtained by global average pooling and global maximum pooling. Given x is the feature map output from the middle layer of the network, the average value of each channel of the feature map x is calculated by global average pooling:
[0015]
[0016] Among them, H and W are the height and width of the feature map respectively;
[0017] Calculate the maximum value of each channel of feature map x:
[0018]
[0019] Use a fully connected layer to transform the output of GAP and GMP:
[0020] CAM(x)=FC GAP (GAP(x))+FC GMP (GMP(x))
[0021] Among them, FC GAP , FC GMP They are the fully connected layers of GAP and GMP outputs respectively;
[0022] The output of CAM is applied back to the original feature map x to weight and highlight important areas:
[0023]
[0024] Furthermore, in S1-2, the dynamic adaptive instance normalization combines the advantages of instance normalization and layer normalization, does not preset fixed normalization parameters, dynamically adjusts according to the statistical characteristics of the input data, and finds the best balance parameters for different tasks, so that the network can effectively adapt to the diverse input data distribution. The specific calculation method is as follows:
[0025]
[0026]
[0027]
[0028] ρ←——clip [0,1] (ρ-τΔρ)
[0029] in, are the activation function values after instance normalization and layer normalization, respectively, where μ I , μ L and σ I , σ L are the mean and standard deviation of each input data point and all data respectively. γ and β are affine transformation parameters dynamically calculated by the fully connected layer in the attention map. ρ is a weight parameter that can be learned and adjusted during training. The optimal balance parameters for different tasks are found through dynamic adjustment between instance normalization and layer normalization. τ is the learning rate. Δρ is the parameter update vector determined by the optimizer. The clip function limits the update boundary of parameter ρ to [0, 1].
[0030] Furthermore, S2 specifically includes:
[0031] A generator and a discriminator are constructed so that they map each other between the original shadow image and the shadow-weakened image. A residual network is introduced into the generator network. The residual network includes multiple residual modules. The information flow and gradient propagation in the deep network are accelerated through jump connections, so as to avoid gradient vanishing and effectively weaken the local shadow area of the image.
[0032] Furthermore, S3 specifically includes:
[0033] Construct a hybrid shadow weakening loss function for the shadow weakening task, which consists of five parts: cycle consistent loss L cyc , against the consistent loss L adv , residual loss Lres , class activation map loss L cam , shadow edge smoothing loss L boundry ;
[0034] The total loss function is:
[0035] L=λ 1 L cyc +λ 2 L adv +λ 3 L cam +λ 4 L boundry +λ 5 L res
[0036] Where: 1 =10,λ 2 =1,λ 3 =1000,λ 4 =1,λ 5 =10.
[0037] Furthermore, the cycle consistency loss L cyc It can constrain the similarity between the fitted image and the real image. Given an image x∈X a , after transformation, it is image x∈X b , and then convert back to the image domain X a After that, the target image successfully completes the source domain conversion, and the calculation is as follows:
[0038]
[0039] The adversarial consistent loss L adv The generator mapping and discriminator discrimination are optimized to make the fitting data more realistic and the true and false classification judgment more accurate. The adversarial loss matches the fitting image distribution with the target image distribution, which is calculated as follows:
[0040]
[0041] Furthermore, the class activation map loss L cam By processing the auxiliary classifier η a and information, focusing on the shadow area in the image, given an image x∈{X a ,X b}, G a→b and D b The shadow area is obtained by class activation mapping loss, which is calculated as follows:
[0042]
[0043]
[0044] Residual loss L res It measures the difference between the residual image predicted by the model and the true residual image, which enables the generator to more accurately predict the brightness changes in the shadow area. It is calculated as follows:
[0045] L res =||X gb -G a→b (X)|| 1
[0046] Among them, X gb is the shadow image and the shadow-free image I gt The residual image close to the shadow ground truth is obtained.
[0047] Furthermore, the class activation map loss L cam By processing the auxiliary classifier η a and information, focusing on the shadow area in the image, given an image x∈{X a ,X b}, G a→b and D b The shadow area is obtained by class activation mapping loss, which is calculated as follows:
[0048]
[0049]
[0050] Shadow edge smoothing loss L boundry It can smoothly transition the boundary between the shadow area and the non-shadow area. The calculation is as follows:
[0051]
[0052] in is the gradient operator, N is the noise robust function, defined as:
[0053] N(Mask a )=N ax +N ay
[0054]
[0055]
[0056] in, and are the partial derivatives of the image in the horizontal and vertical directions, respectively. p is defined as a single pixel, R p is a 3×3 window of pixels around pixel p, g p,qThe weight function to measure spatial similarity is defined as:
[0057]
[0058] Among them, τ defaults to 0.01;
[0059] Through the shadow mask Mask a Calculate the boundary of the shadow area in Mask a By inputting the shadow image X a With the output shadow-free image X b The difference is calculated, and its value is in the range of [0,1]. The difference is calculated as follows:
[0060]
[0061] Among them, the function S is a normalized function, calculated as follows:
[0062]
[0063] Where X max , X min are the maximum and minimum values of X respectively.
[0064] Furthermore, in S4, the specific process of optimizing the shadow weakening effect includes:
[0065] Based on the shadow discrimination auxiliary classifier, the attention map is obtained, and the attention map is embedded into the generator and discriminator to guide the generative adversarial network model to focus on areas with more shadow-specific features, ignore secondary areas, increase the attention weight of the shadow area of the generator and discriminator, and finally weaken the shadows in the background of the road disease dataset;
[0066] Where: Given a shadow input image I s , generator G sr Convert it to a shadow-free output image O sr , discriminator D s Differentiation I r Whether there is a shadow, the generator and its discriminator optimize the following objective function:
[0067]
[0068] Where ε is the error; p data is the data distribution, I r ~p data (I r ) and I s ~p data (I s ) represents the data distribution p from the unshadowed and shadowed datasets respectively. data Select Ir and I s ;
[0069] Using adversarial loss alone to optimize the generator may result in artifacts on the generated image. rs Convert the resulting shadow-free image back to the original shadowed image;
[0070] The shadow and no shadow are represented by s, s r Represents, generator G rs By encoder Decoder and auxiliary classifier Composition, discriminator D r Determines the quality of the shadow removal output by the encoder Classifier and auxiliary classifier Composition, auxiliary classifier and For G sr and D r The input image is classified and integrated into the generator and its discriminator to guide the network to pay attention to the shadow area. Determine whether the image contains shadows;
[0071] Input shadow image I s The corresponding shadow-free image output is O sr , input unpaired shadow-free image I r , G rs Reconstruct the shadow image and return it to assist the classifier and In G rs and D rs In classify whether the image contains shadows.
[0072] Compared with the prior art, the present invention has the following technical advantages:
[0073] 1) On the overall level, in the detection of road damage targets in complex interference scenarios, the present invention aims to address the problems of road shadows reducing the quality of image data and indirectly affecting the loss of feature information. The shadow weakening algorithm proposed based on the generative adversarial network can weaken or remove the interfering shadows in the road background in the road crack dataset required by computer vision tasks, thereby reducing the impact of shadows and converting the shadows in the road background into a pseudo-road surface similar to the road surface. The ability of the road damage target detection model to resist the interference factors in complex scenarios is enhanced, and the accuracy of computer vision tasks such as target detection, target tracking, and image registration is improved.
[0074] 2) On a specific level, the shadow weakening method of the present invention significantly improves the recognition accuracy and processing capability of the model for shadow features by introducing an auxiliary attention classifier and a dynamic adaptive normalization function. This method can effectively weaken the shadows in road damage images and enhance the recognizability of damage features, thereby providing clearer and more accurate image data for the detection and analysis of road damage.
[0075] 3) On a specific level, through the design of improved residual modules and jump connections, the present invention effectively solves the gradient vanishing problem that may occur during the training process of deep generative adversarial networks. This improvement not only speeds up the training of the model, but also improves the stability and generalization ability of the model, making the generated images more realistic and closer to the real road disease scene.
[0076] 4) The hybrid shadow weakening loss function proposed in this paper combines the difference between the image generated by the generative adversarial network and the real image and the feedback of the auxiliary attention classifier to further optimize the shadow weakening effect. This method not only improves the quality of the generated image, but also enhances the adaptability of the model to the areas with significant changes between the source domain and the target domain, providing strong technical support for the automated detection and intelligent analysis of road diseases. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] Figure 1 It is a network structure diagram of the shadow weakening model of the present invention;
[0078] Figure 2 It is the structural diagram of the ILN module in the algorithm of the present invention;
[0079] Figure 3 It is the ILN calculation flow chart in the algorithm of the present invention;
[0080] Figure 4 is a structural diagram of the auxiliary attention classifier of the present invention;
[0081] Figure 5 This is the result of the shadow weakening experiment;
[0082] Figure 6 This is a comparison chart of the image quality index values after the shadow is weakened;
[0083] Figure 7 These are the effects of weakening shadows using different algorithms;
[0084] Figure 8 This is a comparison chart of different algorithms on the ISTD dataset. DETAILED DESCRIPTION
[0085] On the whole, the present invention discloses a shadow weakening algorithm based on weakly supervised image conversion generative adversarial network, adopts a shadow weakening model based on unsupervised image conversion generative adversarial network, and introduces auxiliary classifiers with the help of the idea of auxiliary attention classifiers in UGI unsupervised image conversion generative adversarial network, selects a dynamic adaptive normalization function strategy suitable for road disease background shadow weakening task to generate adversarial game optimization, improves the residual module, introduces jump connections between standard convolutional layers, alleviates gradient disappearance, and proposes a hybrid shadow weakening loss function suitable for shadow weakening tasks, so as to better complete the weakly supervised shadow weakening work. The present invention can effectively suppress the interference of shadows on the road crack feature extraction process, reduce the probability of image loss feature information, enhance the ability of subsequent road disease target detection to resist shadow interference in complex scenes, and improve the accuracy of road disease target detection.
[0086] In order to achieve the purpose, the present invention mainly adopts the following technical means:
[0087] (1) The present invention adopts a shadow reduction model based on unsupervised image conversion generative adversarial network to reduce shadows in road scenes;
[0088] (2) By using the idea of auxiliary attention classifier in the UGI unsupervised image conversion generative adversarial network, the auxiliary classifier is introduced to promote the generative adversarial model to better learn feature representation when the target data scale is small, increase the attention to the target feature area, ignore minor information, and improve the sensitivity to key information features;
[0089] (3) A dynamic adaptive normalization function strategy suitable for the task of weakening the background shadow of road damage is selected for generative adversarial game optimization, the residual module is improved, and jump connections are introduced between standard convolutional layers to alleviate the gradient disappearance;
[0090] (4) A hybrid shadow weakening loss function suitable for shadow weakening tasks is proposed to optimize the shadow weakening model.
[0091] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Component models, material names, connection structures, control methods, algorithms and other features not clearly described in this technical solution are all considered to be common technical features disclosed in the prior art.
[0092] Example 1
[0093] This embodiment is based on the shadow weakening algorithm of weakly supervised image conversion generative adversarial network. When implemented specifically:
[0094] A shadow weakening model based on unsupervised image conversion generative adversarial network is adopted. The auxiliary attention classifier module is introduced with the help of the idea of auxiliary attention classifier in UGI unsupervised image conversion generative adversarial network. The dynamic adaptive normalization function suitable for the road disease shadow weakening task is selected, the residual module is improved, and jump connections are introduced between standard convolutional layers to alleviate the gradient disappearance. At the same time, a hybrid shadow weakening loss function suitable for the shadow weakening task is proposed.
[0095] In specific implementation, Generative Adversarial Nets (GAN) uses deep learning technology to generate data similar to the real data distribution by simulating the distribution of training data sets. It has shown wide applicability in many fields such as high-resolution image generation, denoising low-quality images, and image style transfer.
[0096] The Deep Convolutional Generative Adversarial Networks (DCGAN) based on the deep convolutional network architecture adopts the deep convolutional network architecture to improve the generator and discriminator, and uses two convolutional neural networks to compete with each other. The pooling layer of the convolutional neural network is discarded, the discriminator maintains the core architecture of the convolutional network, and the generator introduces deconvolution layers and batch normalization layers. This not only optimizes the initial state of the model training, but also improves the training efficiency and stability, and accelerates the model convergence speed.
[0097] The UGI (Unsupervised Image Translation Generative Adversarial Network) proposed on the basis of deep convolutional generative adversarial network has multi-task robustness and includes a generator and a discriminator, which map each other between the original shadow image and the shadow-weakened image. A residual network is introduced in the generator network, which includes multiple residual modules. It accelerates the information flow and gradient propagation in the deep network through jump connections, avoids gradient vanishing, and effectively weakens the local shadow area of the image. Based on the shadow discrimination auxiliary classifier, the attention map is obtained and embedded into the generator and discriminator to guide the model conversion to focus on the areas with more shadow-specific features, ignore the secondary areas, increase the attention weight of the generator and discriminator in the shadow area, and finally weaken the shadows in the road disease background.
[0098] In the specific implementation, the auxiliary attention classifier module is introduced with the help of the idea of auxiliary attention classifier in the UGI unsupervised image conversion generative adversarial network. When the target data scale is small, the generative adversarial model can better learn feature representation through auxiliary classifier, increase the attention of target feature area, ignore minor information, and improve the sensitivity to key information features. The implementation is as follows:
[0099] The auxiliary attention classifier based on class activation mapping is integrated into the generator and discriminator. The attention map in the generator focuses on the specific area that distinguishes the source domain from the target domain, while the attention map in the discriminator assists model fine-tuning by focusing on the difference between the real and fake images in the target image area. The feature map obtained by the improved residual module can capture the different features and details of the input image. After global average pooling and global maximum pooling, the feature vector based on the number of channels is a regional attention map that highlights the features more. Then, a learnable weight parameter weight is created, which is compressed to B×1 dimensions through a fully connected layer. A weight is assigned to each channel of the feature map, and its value determines the importance of the corresponding feature of the channel, thereby strengthening the more targeted target features. After global average pooling and global maximum pooling, the features are fused and the source and target domain classification of the target is judged, and finally the target image is generated. The auxiliary classifier enables the generator to focus on key features, ensuring that the model pays more attention to the areas that change significantly between the source and target domains, improving the quality and relevance of the generated image, and obtaining a more comprehensive spatial attention representation.
[0100] In specific implementation, Class Activation Mapping (CAM) can intuitively explore the key pixel parts when making model classification decisions. It generates a spatial attention map to highlight the areas in the image that are more important to the target features, and is more applicable to images with complex backgrounds or multiple targets. The spatial attention map is obtained through global average pooling (GAP) and global max pooling (GMP). Given x is the feature map output from the middle layer of the network, the average value of each channel of the feature map x is calculated through global average pooling:
[0101]
[0102] Among them, H and W are the height and width of the feature map respectively.
[0103] Calculate the maximum value of each channel of feature map x:
[0104]
[0105] Use a fully connected layer to transform the output of GAP and GMP:
[0106] CAM(x)=FC GAP (GAP(x))+FC GMP (GMP(x))
[0107] Among them, FC GAP , FC GMP They are the fully connected layers of GAP and GMP outputs respectively.
[0108] The output of CAM is applied back to the original feature map x to weight and highlight important areas:
[0109]
[0110] In the specific implementation, a dynamic adaptive normalization function suitable for the task of weakening road damage shadows is selected, the residual module is improved, and jump connections are introduced between standard convolutional layers to alleviate the gradient disappearance. The implementation is as follows:
[0111] Normalization is a data preprocessing method that can improve the training efficiency and generalization performance of the model. Layer Normalization (LN) normalizes the channel dimension of each hidden layer, which can alleviate the gradient disappearance. Instance Normalization (IN) is suitable for image conversion with diverse data. It ignores batch information and independently normalizes each input data point, allowing the model to focus on the overall information of the image, but its generalization ability is limited. The dynamically adaptive instance normalization function calculation method ILN combines the advantages of instance normalization and layer normalization. It does not preset fixed normalization parameters, but dynamically adjusts according to the statistical characteristics of the input data to find the best balance parameters for different tasks, so that the network can effectively adapt to the diverse input data distribution. The calculation method is as follows:
[0112]
[0113]
[0114]
[0115] ρ←——clip [0,1] (ρ-τΔρ)
[0116] in, are the activation function values after instance normalization and layer normalization, respectively, where μ I , μ L and σ I , σ Lare the mean and standard deviation of each input data point and all data, γ and β are the affine transformation parameters dynamically calculated by the fully connected layer in the attention map, ρ is the weight parameter that can be learned and adjusted during the training process, and the optimal balance parameters for different tasks are found through dynamic adjustment between instance normalization and layer normalization. τ is the learning rate, and Δρ is the parameter update vector determined by the optimizer. The clip function limits the update boundary of the parameter ρ to [0, 1]. The value of ρ in instance normalization is close to 1, the value of ρ in layer normalization is close to 0, the initial value of ρ is close to 1 in the task of preserving image detail features, and the initial value of ρ is close to 0 in the task of processing overall image conversion. Generally, the value of ρ is initialized to 0.9 in the residual block of the decoder and to 0.1 in the upsampling block of the decoder.
[0117] Each residual module includes two convolutional layer structures. The improved dynamic adaptive instance normalization function calculation method enables the model to adaptively adjust the weight parameter value to adapt to the diverse data distribution. The input data adjusts the weight parameter value after passing through the reverse padding layer and the convolution layer, and receives the normalized feature map parameters adjusted by the fully connected layer, so that the model can learn more complex feature representations and better handle information transfer between the bottom layer and the upper layer.
[0118] In the specific implementation, in order to achieve a better shadow weakening effect, a hybrid shadow weakening loss function suitable for the shadow weakening task is proposed. The loss function consists of five parts: cycle consistent loss L cyc , against the consistent loss L adv , residual loss L res , class activation map loss L cam and shadow edge smoothing loss L boundry .
[0119] Cycle consistency loss L cyc It can constrain the similarity between the fitted image and the real image. Given an image x∈X a , after transformation, it is image x∈X b , and then convert back to the image domain X a After that, the target image successfully completes the source domain conversion, and the calculation is as follows:
[0120]
[0121] Against the consistent loss L adv The generator mapping and discriminator discrimination are optimized to make the fitted data more realistic and the true and false classification judgment more accurate. The adversarial loss matches the fitted image distribution with the target image distribution, which is calculated as follows:
[0122]
[0123] Class activation map loss L camBy processing the auxiliary classifier η a and information, focusing on the shadow area in the image, given an image x∈{X a ,X b}, G a→b and D b The shadow area is obtained by class activation mapping loss, which is calculated as follows:
[0124]
[0125]
[0126] Shadow edge smoothing loss L boundry It can smoothly transition the boundary between the shadow area and the non-shadow area. The calculation is as follows:
[0127]
[0128] in is the gradient operator, N is the noise robust function, defined as:
[0129] N(Mask a )=N ax +N ay
[0130]
[0131]
[0132] in, and are the partial derivatives of the image in the horizontal and vertical directions, respectively. p is defined as a single pixel, R p is a 3×3 window of pixels around pixel p, g p,q The weight function to measure spatial similarity is defined as:
[0133]
[0134] Here, τ defaults to 0.01.
[0135] Through the shadow mask Mask a Calculate the boundary of the shadow area in Mask. a By inputting the shadow image X a With the output shadow-free image X b The difference is calculated, and its value is in the range of [0,1]. The difference is calculated as follows:
[0136]
[0137] Among them, the function S is a normalized function, calculated as follows:
[0138]
[0139] Where X max , X min are the maximum and minimum values of X respectively.
[0140] Residual loss L res It measures the difference between the residual image predicted by the model and the true residual image, which enables the generator to more accurately predict the brightness changes in the shadow area. It is calculated as follows:
[0141] L res =||X gb -G a→b (X)|| 1
[0142] Among them, X gb is the shadow image and the shadow-free image I gt The residual image close to the shadow ground truth is obtained.
[0143] The total loss function is:
[0144] L=λ 1 L cyc +λ 2 L adv +λ 3 L cam +λ 4 L boundry +λ 5 L res
[0145] where λ 1 =10,λ 2 =1,λ 3 =1000,λ 4 =1,λ 5 =10.
[0146] Application Example 1
[0147] (1) Screening the shadow weakening conversion dataset. The shadow public dataset ISTD dataset (Image ShadowTriplets Dataset) contains 1870 pairs of image shadow pairing groups (shadow images, shadow mask images, and shadow-free images) in 135 different scenes, which are used for shadow detection, generation, and weakening and other shadow-related computer vision tasks. Based on this dataset, 1280 image pairing groups are screened and adjusted, and 320 groups of road crack dataset images with shadows are added at a ratio of 4:1, and finally 1600 pairs of image pairing group datasets are formed as shadow weakening datasets for training and testing the shadow weakening algorithm proposed in this invention. The screened and adjusted shadow datasets are then formed into training sets and test sets at a ratio of 15:1, where the training set includes 1500 pairs of shadow and shadow-free images, and the test set only requires 100 images with shadowed backgrounds containing road diseases.
[0148] (2) Determine the training parameters. The shadow reduction algorithm proposed in the present invention is jointly trained on the ISTD dataset after adding data. The training test environment is: Intel (R) Core (TM) i7-12800H CPU @ GHz, NVIDIA GeForceGTX3060, and the training round is set to 10000 epochs. Set λ = 0.0002 as the starting learning rate, and then gradually decrease it after 500 times. In order to improve the generalization of the algorithm, each training iteration process adjusts the training image resolution to 600 × 600 and then randomly crops it to 512 × 512, randomly rotates α degrees, where the range of α is [-10, 10], and randomly flips. Hybrid shadow removal loss function coefficient λ 1 , 2 , 3 , 4 , 5 The parameters are set to 10, 1, 1000, 1, and 10 respectively.
[0149] (3) Propose image processing quality performance evaluation indicators. Peak signal-to-noise ratio (PSNR), image structural similarity (SSIM) and root mean square error (RMSE) are used to measure and compare the shadow reduction algorithm based on weakly supervised image conversion generative adversarial network proposed in this invention and other contrast networks in the shadow reduction task and the visual quality effect of the image after the task. Its implementation is as follows:
[0150] Peak signal-to-noise ratio (PSNR) is an indicator of image reconstruction quality. It is obtained by calculating the noise level between the original image and the reconstructed image. The higher the value, the better the image quality. Given an image I of size m×n and a noise image K. The mean square error (MSE) is defined as:
[0151]
[0152] The peak signal-to-noise ratio (PSNR) is defined as:
[0153]
[0154] in is the maximum pixel value of the image. If each pixel is represented by 8 bits of binary, it is 255. If the pixel value is represented by B bits of binary, Generally, for uint8 data, the maximum pixel value is 255; for floating-point data, the maximum pixel value is 1.
[0155] Image structural similarity (SSIM) is an indicator used to measure the structural similarity between the original image and the reconstructed image. It takes into account brightness, contrast and structure. Its value range is between [-1, 1]. The closer it is to 1, the more similar the image structure is, providing more detailed and comprehensive image quality information. The calculation formulas for brightness, contrast and structure are as follows:
[0156]
[0157]
[0158]
[0159] Among them, c is generally taken 3 =c 2 / 2; μ x is the mean of x; μ y is the mean of y; σ x 2 is the variance of x; σ y 2 is the variance of y; σ xy is the covariance of x and y; c 1 =(k 1 L) 2 , c 2 =(k 2 L) 2 are two constants to avoid division by zero; L is the range of pixel values, 2 B -1;k 1 =0.01, k 2=0.03 is the default value,
[0160] When calculating, slice an N×N window from the image, then continuously slide the window for calculation, and finally take the average. The calculation formula of SSIM is as follows:
[0161]
[0162] The root mean square error (RMSE) is an image quality indicator that measures the difference between the original image and the reconstructed image. It is obtained by calculating the square root of the average of the square values of the difference between the original image and the reconstructed image. It is highly sensitive to large error values (outliers). The smaller the value, the higher the accuracy of the model image reconstruction. The calculation formula is as follows:
[0163]
[0164] Where n is the number of image data points, y i is the i-th original image data point, is the i-th reconstructed image data point.
[0165] (4) The shadow is weakened. The shadow weakening algorithm proposed in the present invention is based on a generative adversarial network weakly supervised learning network. The network structure is as follows: Figure 1 , including a generator and a discriminator, which map between the original shadow image and the shadow-weakened image. A residual network is introduced into the generator network, which contains multiple residual modules. It accelerates the information flow and gradient propagation in the deep network through jump connections, avoids gradient vanishing, and effectively weakens the local shadow area of the image. Based on the shadow discrimination auxiliary classifier, the attention map is obtained and embedded into the generator and discriminator to guide the model to focus on the areas with more shadow-specific features, ignore the secondary areas, increase the attention weight of the generator and discriminator, and finally weaken the shadows in the background of the road damage dataset.
[0166] Given a shadow input image I s , generator G sr Convert it to a shadow-free output image O sr , discriminator D s Differentiation I r Whether there is a shadow, the generator and its discriminator optimize the following objective function:
[0167]
[0168] Where ε is the error; p data is the data distribution, I r ~p data (I r ) and I s ~p data (I s ) represents the data distribution p from the unshadowed and shadowed datasets respectively. data Select I r and I s . Using adversarial loss alone to optimize the generator may result in artifacts on the generated images, so the generator G rsConvert the resulting unshadowed image back to the original shadowed image.
[0169] The shadow and no shadow are represented by s, s r Represents, generator G rs By encoder Decoder and auxiliary classifier Composition. Discriminator D r Determines the quality of the shadow removal output by the encoder Classifier and auxiliary classifier Composition, auxiliary classifier and For G sr and D r The input image is classified and integrated into the generator and its discriminator to guide the network to pay attention to the shadow area. Determine whether the image contains shadows. The shadow weakening generator uses G rs Indicated by the encoder Decoder and auxiliary classifier Composition, discriminator D rs By encoder Classifier and auxiliary classifier Input shadow image I s The corresponding shadow-free image output is O sr , input unpaired shadow-free image I r , G rs Reconstruct the shadow image and return it to assist the classifier and In G rs and D rs In classify whether the image contains shadows.
[0170] (5) Ablation experiments are conducted on the ISTD dataset with additional road damage shadows to analyze the effect of the shadow reduction algorithm.
[0171] Table 1 Ablation experiments on the ISTD dataset
[0172]
[0173] Note: √ means the improvement point is added in the network, - means the improvement point is not added in the network
[0174] As can be seen from the result table obtained after the ablation experiment, the use of the residual ILN module increases the PSNR and SSIM values by 6.7% and 8.0% respectively. The optimal balance parameters obtained by the dynamic adjustment of the normalization calculation method enable the network to effectively adapt to the complex data distribution with strong randomness, and accelerate the information flow in the deep network, effectively adjust the local shadow area of the image and retain the original image information. The use of the hybrid shadow weakening loss function increases the PSNR and SSIM values by 32.5% and 26.4% respectively. The multi-task loss function improves the model's shadow area recognition ability, thereby more accurately locating and weakening shadows, and making the transition between shadow boundaries and non-shadow areas smoother, reducing color distortion. The ablation experiment shows that the improved shadow weakening algorithm has a good shadow weakening effect in complex road scenes, and enhances the ability of road disease target detection to resist shadow interference in complex scenes.
[0175] (6) Perform a performance comparison experiment on shadow weakening algorithms. Common shadow weakening algorithms include the improved Retinex method based on the physical image method, the supervised shadow removal method Mask-ShadowGAN, and the STCGAN based on image conversion. In order to accurately evaluate the performance of the shadow weakening algorithm based on the weakly supervised image conversion generative adversarial network proposed in this invention, the peak signal-to-noise ratio (PSNR), image structure similarity (SSIM), and image root mean square error (RMSE) in the image quality evaluation index are used to quantitatively evaluate the shadow weakening effects of different algorithms on the mixed ISTD test set.
[0176] Table 2 Comparative experiment on shadow weakening effect on ISTD dataset
[0177]
[0178] As can be seen from the table, the proposed weakly supervised image conversion generative adversarial network algorithm performs well in shadow weakening processing. Compared with the improved Retinex method, Mask-ShadowGAN and STCGAN, the image quality index SSIM is increased by 9.74, 0.583, and 0.442, respectively, and the image quality index PSNR is increased by 0.653, 0.244, and 0.142, respectively. The proposed weakly supervised image conversion generative adversarial network algorithm has the highest index values in PSNR and SSIM, indicating that the image has good quality after shadow weakening and the shadow weakened image has a high structural similarity with the original image. The lowest RMSE index value indicates that the image error is small, and a good shadow weakening effect is achieved while reducing image distortion. Both the Mask-ShadowGAN and STCGAN shadow weakening algorithms are based on generative adversarial networks. Both algorithms achieve high index values, but the SSIM of Mask-ShadowGAN is lower, which means that the image structure similarity after shadow weakening by this algorithm is low, and image color distortion may occur; the SSIM index values of STCGAN and the algorithm of the present invention are slightly different, which may be due to the use of adaptive color equalization processing for color deviation, which reduces the overall color distortion of the image, but its root mean square error is greater than other algorithms, which may be because the algorithm uses some prior knowledge when weakening shadows, and such prior knowledge is not applicable to all scenarios, so it is difficult to adapt to complex and changeable road damage scenarios. Therefore, the shadow weakening algorithm proposed in the present invention has advantages in image quality index values compared to other algorithms. The shadow weakening effect is as follows: Figure 5 As shown, where:
[0179] Mask-ShadowGAN is a shadow weakening algorithm based on deep learning, which uses a binary mask to represent the shadow area. The shadows generated by this method lack details and textures, and even cause image distortion after shadow weakening. As shown in the figure, the pixels at the boundary between the shadow and the actual road surface are relatively blurred. ST-GAN pays more attention to the areas affected by shadows in the image through the differential attention mechanism, while maintaining the details and quality of the non-shadow area. However, after the shadow is weakened, the image may increase the image error due to excessive enhancement of some areas, especially at the boundary between the shadow and non-shadow areas. The algorithm of the present invention can achieve a better shadow weakening effect, retain the original image details and texture after weakening the shadow in the road image, and has high adaptability to illumination changes. The improved Retinex algorithm is a shadow weakening algorithm based on physical methods. The performance of this algorithm in various index values is worse than other algorithms. It improves the image by simulating the adaptability of the human eye to different illumination areas. It is limited by the physical model and assumptions considered in the algorithm design, and cannot be applied to scenes with strong random changes. As a result, the image is severely distorted due to excessive enhancement, some image details become too prominent, and some artifacts are introduced, so it is not compared in the figure.
[0180] (7) Comparison of the accuracy of road damage target detection. In order to evaluate the performance of the shadow weakening algorithm based on weakly supervised image conversion generative adversarial network proposed in the present invention in improving the accuracy of road damage target detection, Mask-ShadowGAN and the shadow weakening network algorithm proposed in the present invention are used to perform shadow weakening processing on the self-owned road damage inspection dataset SD. The weakened road damage dataset is set as RD1 (Mask-ShadowGAN), RD2 (algorithm of the present invention) and SD datasets. The YOLOv8 benchmark network is configured with unified parameters to perform model training on the image datasets processed by different algorithms. The effect of the shadow weakening algorithm is indirectly evaluated by the evaluation index of the detection model. Images disturbed by shadows have the problem that detailed feature information is easily lost, making it difficult for the detection model to effectively characterize small targets. Therefore, if the shadow weakening algorithm can effectively reduce the impact of shadows on images and restore the detailed information of images, it will provide richer information for the detection model, thereby improving the detection effect. The results of the indirect performance comparison experiment are shown in Table 3.
[0181] Table 3 Indirect comparison experiments of different algorithms
[0182]
[0183]
[0184] The experimental results show that when performing target detection on the processed data set, compared with the original SD data set, the detection accuracy of the algorithm of the present invention is improved by 2.5%, 2.9%, 0.24%, 3.2%, 1.6%, and 2.7% on six types of diseases respectively. Compared with Mask-ShadowGAN, the algorithm of the present invention has also improved the detection accuracy of various types of diseases. Shadows may cover up detailed features such as cracks on the road surface, making it difficult for traditional target detection models to identify these areas, while the shadow weakening algorithm can effectively reduce the interference of shadows on the extraction of road crack features. Compared with the data set without shadow image weakening, the effect of target detection on the data set after shadow weakening is significantly improved. The main reason may be that the shadow elimination algorithm can effectively suppress the interference of shadows on the road crack feature extraction process, reduce the probability of image loss of feature information, enhance the ability of subsequent road disease target detection to resist shadow interference in complex scenes, and improve the accuracy of road disease target detection.
[0185] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.
Claims
1. A shadow reduction method based on weakly supervised image conversion generative adversarial network, characterized in that: The following steps are involved: S1: Construct an unsupervised image conversion generative adversarial network model, which introduces a dynamic adaptive normalization function through an auxiliary attention classifier to enhance the recognition and processing capabilities of shadow features; S2: Introducing a residual module with improved design, by introducing skip connections between convolutional layers, the efficiency of information flow and gradient propagation in the deep network is improved to alleviate the gradient vanishing problem of the generative adversarial network model; S3: Construct a mixed shadow weakening loss function to complete the construction of the generative adversarial network model; S4: The difference between the image generated by the generative adversarial network and the real image is combined with the feedback of the auxiliary attention classifier to optimize the shadow weakening effect and achieve accurate weakening of road disease shadows.
2. According to claim 1, a shadow reduction method based on weakly supervised image conversion generative adversarial network is characterized in that: S1 specifically includes: S1-1: Design auxiliary attention classifier: Construct an auxiliary attention module, which is integrated into the generator and discriminator of the generative adversarial network. The auxiliary attention module uses class activation mapping technology to generate a spatial attention map through global average pooling and global maximum pooling to highlight the areas that are more important to the target features, thereby guiding the model to pay more attention to the shadow features; S1-2: Introduce a dynamic adaptive normalization function: Select a dynamic adaptive instance normalization method and dynamically adjust the normalization parameters to adapt to the statistical characteristics of different input data, so that the generative adversarial network model can adaptively adjust the weight parameters, optimize the processing of shadow features, and improve the model's recognition accuracy and processing capabilities for shadow areas.
3. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 2 is characterized in that: In S1-1, the spatial attention map is obtained by global average pooling and global maximum pooling. Given x is the feature map output from the middle layer of the network, the average value of each channel of the feature map x is calculated by global average pooling: Among them, H and W are the height and width of the feature map respectively; Calculate the maximum value of each channel of feature map x: Use a fully connected layer to transform the output of GAP and GMP: CAM(x)=FC GAP (GAP(x))+FC GMP (GMP(x)) Among them, FC GAP , FC GMP They are the fully connected layers of GAP and GMP outputs respectively; The output of CAM is applied back to the original feature map x to weight and highlight important areas:
4. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 2 is characterized in that: In S1-2, the dynamic adaptive instance normalization combines the advantages of instance normalization and layer normalization, does not preset fixed normalization parameters, dynamically adjusts according to the statistical characteristics of the input data, and finds the best balance parameters for different tasks, so that the network can effectively adapt to the diverse input data distribution. The specific calculation method is as follows: ρ←——clip [0,1] (r-tDr) in, are the activation function values after instance normalization and layer normalization, respectively, where μ I , μ L and σ I , σ L are the mean and standard deviation of each input data point and all data respectively. γ and β are affine transformation parameters dynamically calculated by the fully connected layer in the attention map. ρ is a weight parameter that can be learned and adjusted during training. The optimal balance parameters for different tasks are found through dynamic adjustment between instance normalization and layer normalization. τ is the learning rate. Δρ is the parameter update vector determined by the optimizer. The clip function limits the update boundary of parameter ρ to [0, 1].
5. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 1 is characterized in that: S2 specifically includes: A generator and a discriminator are constructed so that they map each other between the original shadow image and the shadow-weakened image. A residual network is introduced into the generator network. The residual network includes multiple residual modules. The information flow and gradient propagation in the deep network are accelerated through jump connections, so as to avoid gradient vanishing and effectively weaken the local shadow area of the image.
6. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 1, characterized in that: S3 specifically includes: Construct a hybrid shadow weakening loss function for the shadow weakening task, which consists of five parts: cycle consistent loss L cyc , against the consistent loss L adv , residual loss L res , class activation map loss L cam , shadow edge smoothing loss L boundry ; The total loss function is: L=λ1L cyc +λ2L adv +λ3L cam +λ4L boundry +λ5L res Among them: λ1=10, λ2=1, λ3=1000, λ4=1, λ5=10.
7. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 6 is characterized in that: The cycle-consistent loss L cyc It can constrain the similarity between the fitted image and the real image. Given an image x∈X a , after transformation, it is image x∈X b , and then converted back to the image domain X a After that, the target image successfully completes the source domain conversion, and the calculation is as follows: The adversarial consistent loss L adv The generator mapping and discriminator discrimination are optimized to make the fitting data more realistic and the true and false classification judgment more accurate. The adversarial loss matches the fitting image distribution with the target image distribution, which is calculated as follows:
8. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 6 is characterized in that: The class activation map loss L cam By processing the auxiliary classifier η a and information, focusing on the shadow area in the image, given an image x∈{X a ,X b }, G a→b and D b The shadow area is obtained by class activation mapping loss, which is calculated as follows: Residual loss L res It measures the difference between the residual image predicted by the model and the true residual image, which enables the generator to more accurately predict the brightness changes in the shadow area. It is calculated as follows: L res =||X gb -G a→b (X)||1 Among them, X gb is the shadow image and the shadow-free image I gt The residual image close to the shadow ground truth is obtained.
9. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 6, characterized in that: The class activation map loss L cam By processing the auxiliary classifier η a and information, focusing on the shadow area in the image, given an image x∈{X a ,X b }, G a→b and D b The shadow area is obtained by class activation mapping loss, which is calculated as follows: Shadow edge smoothing loss L boundry It can smoothly transition the boundary between the shadow area and the non-shadow area. The calculation is as follows: in is the gradient operator, N is the noise robust function, defined as: N(Mask a )=N ax +N ay in, and are the partial derivatives of the image in the horizontal and vertical directions, respectively. p is defined as a single pixel, R p is a 3×3 window of pixels around pixel p, g p,q The weight function to measure spatial similarity is defined as: Among them, τ defaults to 0.01; Through the shadow mask Mask a Calculate the boundary of the shadow area in Mask a By inputting the shadow image X a With the output shadow-free image X b The difference is calculated, and its value is in the range of [0,1]. The difference is calculated as follows: Among them, the function S is a normalized function, calculated as follows: Where X max , X min are the maximum and minimum values of X respectively.
10. The shadow reduction method based on weakly supervised image conversion generative adversarial network according to claim 1, characterized in that: In S4, the specific process of optimizing the shadow weakening effect includes: Based on the shadow discrimination auxiliary classifier, the attention map is obtained, and the attention map is embedded into the generator and discriminator to guide the generative adversarial network model to focus on areas with more shadow-specific features, ignore secondary areas, increase the attention weight of the shadow area of the generator and discriminator, and finally weaken the shadows in the background of the road disease dataset; Where: Given a shadow input image I s , generator G sr Convert it to a shadow-free output image O sr , discriminator D s Differentiation I r Whether there is a shadow, the generator and its discriminator optimize the following objective function: Where ε is the error; p data is the data distribution, I r ~p data (I r ) and I s ~p data (I s ) represents the data distribution p from the unshadowed and shadowed datasets respectively. data Select I r and I s ; Using adversarial loss alone to optimize the generator may result in artifacts on the generated image. rs Convert the resulting shadow-free image back to the original shadowed image; The shadow and no shadow are represented by s, s r Represents, generator G rs By encoder Decoder and auxiliary classifier Composition, discriminator D r Determines the quality of the shadow removal output by the encoder Classifier and auxiliary classifier Composition, auxiliary classifier and For G sr and D r The input image is classified and integrated into the generator and its discriminator to guide the network to pay attention to the shadow area. Determine whether the image contains shadows; Input shadow image I s The corresponding shadow-free image output is O sr , input unpaired shadow-free image I r , G rs Reconstruct the shadow image and return it to assist the classifier and In G rs and D rs In classify whether the image contains shadows.
