Image target detection method in low-illumination superposition fog environment based on APL-Net three-stage distillation

Through the APL-Net three-stage distillation method and adaptive image enhancement technology, the existing object detection algorithm is solved inadequate performance in low-illumination foggy environments, improving the accuracy and performance of object detection, and reducing calculation costs.

CN119027755APending Publication Date: 2024-11-26GUILIN UNIV OF ELECTRONIC TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410884498.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-03
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing object detection algorithms are insufficient in complex environments such as low illumination and foggy days, making it difficult to meet the actual application needs. The existing object detection methods in severe weather usually require additional data enhancement or complex post-processing, which increases the computational cost and Reduced performance.

Method used

Using the three-stage distillation method based on APL-Net, the knowledge is transmitted between the teacher network and the student network through the adaptive perceptual loss network, so that the student network has stronger capabilities in low-illumination fog image object detection, and uses adaptive image enhancement. Technology makes the feature semantic distances in the teacher-student network closer, thereby improving detection performance and accuracy.

Benefits of technology

It significantly improves the target detection performance and accuracy under harsh visual conditions, improves the ability of low-illumination fog image object detection, reduces dependence on labeled data, and reduces calculation costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027755A_ABST
    Figure CN119027755A_ABST
Patent Text Reader

Abstract

The invention discloses an image target detection method in a low-illumination superposition fog environment based on APL-Net three-stage distillation. The method comprises the following steps: 1) processing and dividing a data set; 2) training a teacher network I; and 3) the teacher network 1 is used to guide and train the teacher network 2, and the teacher network 2 is optimized. And 4) the teacher network I and the teacher network II are combined to guide and train the student network. And 5) testing the student network. According to the method, through the multi-level teacher-student network, the student network has the image target detection capability in the low-illumination superposition fog environment. The model can perform adaptive enhancement on the image, and through multi-stage training, the model can effectively extract common features, beneficial to detection, of a clear image and a blurred image, the performance of the model under the low-illumination superposition foggy environment condition is improved, and the model can identify and position a target in the image more accurately.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to deep learning, low-light image enhancement, fog image enhancement, teacher-student network, target detection technology and self-attention mechanism, and specifically to an image target detection method in a low-light superimposed fog environment based on APL-Net three-stage distillation. Background Art

[0002] Object detection is an important task in the field of computer vision, which aims to identify target objects in images or video frames and determine their locations. With the rapid development of computer vision technology, object detection has become one of the key technologies in many fields such as intelligent monitoring, autonomous driving, and robot vision. Although object detection technology has made significant progress under normal lighting conditions, the performance of existing algorithms is often limited in complex environments such as low illumination and foggy weather, making it difficult to meet the needs of practical applications.

[0003] Low-light environments, such as at night or in cloudy weather, can lead to uneven illumination in images, increasing the difficulty of target detection; the increase in noise in the image and the decrease in the contrast between the target and the background make it difficult to extract the features of the target, thus affecting the accuracy and robustness of the detection algorithm. Under foggy conditions, due to the scattering effect of fog, the outlines of objects in the image become blurred and the detailed information is lost, further increasing the complexity of target detection. Traditional target detection algorithms are prone to misjudgment or missed detection in such environments, resulting in decreased detection performance. In addition, many target detection algorithms rely on a large amount of labeled data for training, while a large amount of labeling for low-light foggy images is unrealistic. In addition, existing target detection methods in bad weather only consider simple cases such as low-light images or foggy images. When these models are directly applied to the complex environment of nighttime foggy days, additional data enhancement or complex post-processing steps are usually required, which not only increases the computational cost, but also degrades the performance, thus limiting the practicality of the algorithm. An effective method for low-light superimposed foggy image target detection is urgently needed. Summary of the invention

[0004] In view of the problems of insufficient performance and insufficient data of traditional target detection algorithms under complex lighting and weather conditions, this paper proposes a low-light foggy image target detection method based on APL-Net three-stage distillation. We named it APL-Net (Adaptive Perceptual Loss Network). This method realizes effective knowledge transfer from the teacher network to the student network under low-light and foggy conditions through a three-stage distillation adaptive perceptual loss network. Adaptive image enhancement is used to make the feature semantic distances in the teacher and student networks closer, making the distillation more effective. Through this knowledge distillation process, the student network can inherit and optimize the detection knowledge of the teacher network in clear images and slightly blurred images, improve the model's ability in low-light foggy image target detection, and significantly improve the target detection performance and accuracy under harsh visual conditions.

[0005] The present invention comprises the following steps:

[0006] 1) The low-light image dataset ExDark, the foggy dataset RTTS, and the clear image dataset VOC2012 are processed and divided as follows:

[0007] 1-1) The ExDark dataset is divided into 8:2 ratios, 80% of which is the training set ExDark-Train, with a total of 5891 images; 20% is the test set ExDark-Test, with a total of 1472 images;

[0008] 1-2) The RTTS dataset is divided into 8:2 ratios, 80% of which is the training set RTTS-Train, with a total of 3458 images; 20% is the test set RTTS-Test, with a total of 864 images;

[0009] 1-3) The VOC2012 dataset is divided into a ratio of 8:2, of which 80% is the training set VOC-Train, with a total of 13,700 images; 20% is the test set VOC-Test, with a total of 3,425 images;

[0010] 2) In the first stage, all VOC training set VOC-Train images are fed into APL-Net (see Figure 1 ), train Teacher1-Net, so that Teacher1-Net has the ability to detect targets under normal illumination clear images. The image is first passed through the adaptive image filter AIF (see Figure 2 ), and then pass through the detection module ADE (see Figure 7 ) for training, specifically:

[0011] 2-1) The image is downsampled to a 128*128 thumbnail and input into the parameter prediction module PPM (see Figure 3 ), after three self-attention layers and then two fully connected layers, the output P is obtained, which contains four parameters, namely α, β, ω, and δ, which are used as parameters for dehazing, brightening, contrast, and sharpening enhancement, respectively.

[0012] 2-2) Input the image into the basic image filter BIF (see Figure 4 ), a series of enhancements are performed on the image in BIF. First, the image is histogram equalized, and the equalized image is defogged. The calculation process of defogging enhancement is shown in formula (1):

[0013]

[0014] In formula (1), J(x) represents the clear image after defogging, I(x) represents the image to be defogged, t(x) is the transmittance, and A is the global atmospheric light component. A is estimated by the average brightness of the corresponding position pixels of the brightest 1000 pixels in the dark channel image of I(x). The estimated value of t(x) is shown in formula (2):

[0015]

[0016] In formula (2), Ω(x) represents a window centered at pixel x, and c represents the three channels of R, G, and B. α is calculated by step 2-1).

[0017] The image obtained after dehazing is enhanced in brightness, and the image brightness is adjusted by adjusting the image Gamma value. The specific calculation process is shown in formula (3):

[0018] P y =P i β (3)

[0019] Formula (3) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(P y ,G y ,B y ), where parameter β is calculated by step 2-1).

[0020] The image after brightness enhancement is contrast enhanced. The calculation process is shown in formulas (4) and (5):

[0021]

[0022] EL(P i )=0.5(1-cos(π(0.27R i+0.67G i +0.06B i ))) (5)

[0023] Formula (4) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,B y ), where the parameter ω is calculated by step 2-1).

[0024] The image after contrast enhancement is sharpened, and the calculation process is shown in formula (6):

[0025] F(x)=I(x)+δ(I(x)-G(I(x))) (6)

[0026] Where F is the calculation result, I is the input, G is the Gaussian filter, and the parameter δ is calculated by step 2-1).

[0027] 2-3) The enhanced image obtained in step 2-2) is tested by the ADE detection module. In the ADE module, it first passes through four CBS modules (convolution, batch normalization, SiLu activation), and then passes through the attention residual module ARM (see Figure 8 ), and then through the maximum pooling convolution module MPC (see Fig. 9 ), and then pass through the module ARM, the result is recorded as C1, C1 passes through MPC and ARM in turn, the result is recorded as C2, C2 passes through MPC and ARM in turn, the result is recorded as C3; C3 passes through the convolution module CBS, and then passes through the ARM2 module (see Fig.10 ), the result is recorded as C4; C2 is spliced ​​with C4 in the channel direction through CBS, and the spliced ​​result is input into the ARM2 module, and the result is recorded as C5. C1 is spliced ​​with CBS and C5 through CBS and UPS (up-sampling operation) in turn, and the two results are spliced, and the result is recorded as C6. C6 is input into the MPC module and spliced ​​with C5, and the spliced ​​result is input into ARM2, and the result is recorded as C7. C7 is input into the MPC module and spliced ​​with C4, and the spliced ​​result is input into ARM2, and the result is recorded as C8. C6, C7, and C8 are sent to the detection head Head for detection. Finally, the labeling box loss GTL is calculated and the parameters are updated by back propagation. The loss function is shown in formula (7):

[0028] GTL=0.1L conf +0.125L cls +0.05Lx oc (7)

[0029] Where L conf represents the classification loss, which is the category confidence loss calculated using the binary cross entropy loss function; L cls is the positioning loss, using CIOU Loss to calculate the error between the predicted box and the real box, L loc is the confidence loss, and the binary cross entropy loss function is used to calculate the target confidence loss.

[0030] 2-4) After several rounds of training, Teacher1-Net can detect targets in clear images with normal illumination.

[0031] 3) In the second stage, all the images from the VOC training set VOC-Train, the low-light image dataset ExDark-Train, and the foggy image dataset RTTS-Train are mixed and sent to APL-Net to train Teacher2-Net, and Teacher1-Net is used to perform knowledge distillation on Teacher2-Net. If the image comes from the VOC-Train clear dataset, the clear image is input into Teacher1-Net, and the clear image is added with moderate fog and low-light effects and input into Teacher2-Net to calculate the perceptual loss PL of Teacher1-Net and Teacher2-Net and the annotation box loss GTL of Teacher2-Net for the blurred image; if the image comes from the low-light image dataset ExDark-Train, a slight fog is added to the image and input into Teacher2-Net to calculate the annotation box loss GTL; if the image comes from the foggy image dataset RTTS-Train, a slight low light is added to the image and input into Teacher2-Net to calculate the annotation box loss GTL. The total loss L = PL + GTL, and the specific loss calculation method is as follows: Figure 5 As shown. Through training, Teacher2-Net has the ability to detect targets in images with low illumination and shallow fog, specifically:

[0032] 3-1) If the image comes from the VOC-Train clear dataset, the image is first passed through the low-light module LowLight-Gen to add moderate low light, and the parameter x is set to a random number greater than 0.7 and less than 0.85. The calculation process is shown in formula (8):

[0033]

[0034] Formula (8) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,By ).

[0035] Then, moderate fog is added through the fog module Fog-Gen, and the parameter x is set to a random number greater than 0.25 and less than 0.75. The calculation process is shown in formula (9):

[0036] g(x)=t(x,θ)I(x)+(1-t(x,θ))A (9)

[0037] Where A is a constant of 0.5, I(x) is a clear image, g(x) is a foggy image, and the calculation process of t(x,θ) is shown in formula (10):

[0038] t(x,θ)=exp(-θd(x)) (10)

[0039] The calculation process of d(x) is shown in formula (11):

[0040]

[0041] Where W, H are the width and height of the image, x is the current pixel position, and c is the center position of the image.

[0042] The image with moderate low light and moderate fog is input to Teacher2-Net, while the original clear image is input to Teacher1-Net.

[0043] If the image comes from the ExDark-Train dataset, the image is first passed through the fog module Fog-Gen to add light fog. The parameter x is set to a random number greater than 0 and less than 0.5. The calculation process is shown in formula (9). The obtained image is input into Teacher2-Net.

[0044] If the image comes from the RTTS-Train dataset, the image is first passed through the low-light module LowLight-Gen to add mild low light. The parameter x is set to a random number greater than 0.8 and less than 0.95. The calculation process is shown in formula (8). The obtained image is input into Teacher2-Net.

[0045] 3-2) In Teacher1-Net, the input image first passes through the adaptive image filter AIF described in step 2) and then passes through the ADE module for detection; in Teacher2-Net, the input image first passes through the adaptive image filter AIF described in step 2) and then passes through the ADE module for detection.

[0046] 3-3) Use the first 4 layers of the ADE module in Teacher1-Net and Teacher2-Net in step 3-2) to calculate the perceptual loss PL. The calculation process is shown in formula (12):

[0047]

[0048] Among them, T1 u Represents the calculation result of the uth layer of Teacher1-Net, T2 u represents the calculation result of the uth layer of Teacher2-Net, a u Represents the number of activated neurons in layer u. If the original image comes from ExDark-Train or RTTS-Train, P is 0.

[0049] Then use the output result of Teacher2-Net in step 3-2) and the label value in the data set to calculate the annotation box loss GTL through formula (7), and the total loss function L = PL + GTL.

[0050] 4) In the third stage, all VOC training set VOC-Train, low-light image dataset ExDark-Train, and fog image dataset RTTS-Train images are mixed and sent to APL-Net to train Student-Net, and Teacher1-Net and Teacher2-Net are used to jointly distill the Student network. If the image comes from the VOC-Train clear dataset, the clear image is input into Teacher1-Net, moderate fog and low light effects are added to the clear image and input into Teacher2-Net, and heavy low light and fog are added to the clear image and input into Student-Net to obtain the joint perceptual loss P1L of Teacher1-Net, Teacher2-Net and Student-Net and the annotation box loss GTL of Student-Net; if the image comes from the low-light image dataset ExDark-Train, the image is input into Teacher2-Net, and heavy fog is added to the image and input into Student-Net to obtain the perceptual loss P2L of Teacher2-Net and Student-Net and the annotation box loss GTL of Student-Net; if the image comes from the foggy image dataset RTTS-Train, the image is input into Teacher2-Net, and heavy low light is added to the image and input into Student-Net to obtain the perceptual loss P2L of Teacher2-Net and Student-Net and the annotation box loss GTL of Student-Net. The total loss function L = P1L + P2L + GTL, the specific loss calculation method is as follows Figure 6As shown. Through the joint guidance of Teacher1-Net and Teacher2-Net on the training of Student-Net, Student-Net has the ability to detect targets in low-light foggy images, specifically:

[0051] 4-1) If the image comes from the VOC-Train clear dataset, the image is first passed through the low-light module LowLight-Gen to add moderate low light, and the parameter x is set to a random number greater than 0.7 and less than 0.85. The calculation process is shown in formula (8). Then, the image is passed through the fog module Fog-Gen to add moderate fog, and the parameter x is set to a random number greater than 0.25 and less than 0.75. The calculation process is shown in formula (9). At the same time, the clear source image is passed through the low-light module LowLight-Gen to add heavy low light, and the parameter x is set to a random number greater than 0.6 and less than 0.75. The calculation process is shown in formula (8). Then, the image is passed through the fog module Fog-Gen to add heavy fog, and the parameter x is set to a random number greater than 0.6 and less than 0.95. The calculation process is shown in formula (9). Finally, the clear source image is directly input into Teacher1-Net, the image obtained by adding moderate fog and low light is input into Teacher2-Net, and the image obtained by adding heavy fog and low light is input into Student-Net.

[0052] If the image comes from the ExDark-Train dataset, the image is first passed through the fog module Fog-Gen to add heavy fog. The parameter is set to a random number greater than 0.6 and less than 0.95. The calculation process is shown in formula (9). The obtained image is input into Student-Net, and the low-light source image is directly input into Teacher2-Net.

[0053] If the image comes from the RTTS-Train dataset, the image is first passed through the low-light module LowLight-Gen to add heavy low light. The parameter x is set to a random number greater than 0.6 and less than 0.75. The calculation process is shown in formula (8). The obtained image is input into Student-Net, and the foggy source image is directly input into Teacher2-Net.

[0054] 4-2) Calculate the loss function and update the weights. The total loss L = P1L + P2L + GTL, where P1L is as shown in formula (13):

[0055]

[0056] Among them, T1 u Represents the calculation result of the uth layer of Teacher1-Net, T2 urepresents the calculation result of the uth layer of Teacher2-Net, S u represents the calculation result of the uth layer of Student-Net, a u Represents the number of activated neurons in layer u. If the original image comes from ExDark-Train or RTTS-Train, P is 0.

[0057] P2L is shown in formula (14):

[0058]

[0059] Among them, T2 u represents the calculation result of the uth layer of Teacher2-Net, S u represents the calculation result of the uth layer of Student-Net, a u Represents the number of activated neurons in layer u.

[0060] GTL is shown in formula (7).

[0061] 5) Mix VOC-Test, ExDark-Test, and RTTS-Test and input them into Student-Net for testing.

[0062] 2. The image target detection method in a low-light superimposed fog environment based on APL-Net three-stage distillation according to claim 1 is characterized in that the PPM module in step 2-1) is specifically:

[0063] The module input is a 128*128*3 thumbnail image. After three self-attention layers, the result is a 128*128*3 feature map. After two fully connected layers, the output is a vector of length 4.

[0064] 3. According to the image target detection method in low-light superimposed fog environment based on APL-Net three-stage distillation according to claim 1, it is characterized in that the ARM module in the step 2-3) is specifically:

[0065] The input feature map F1 passes through a self-attention layer, and the result is recorded as F2. F2 passes through two convolution blocks, namely convolution layer, batch normalization layer, and activation layer, and the result is recorded as F3. F3 passes through two convolution blocks, and the result is recorded as F4. F1 passes through another self-attention layer, and the result is recorded as F5. F2, F3, F4, and F5 are spliced ​​as the output of the ARM module.

[0066] 4. The image target detection method in a low-light superimposed fog environment based on APL-Net three-stage distillation according to claim 1, characterized in that the ARM2 module in the step 2-3) is specifically:

[0067] The input feature map F1 passes through a self-attention layer, and the result is recorded as F2. F2 passes through a convolution block, namely the convolution layer, batch normalization layer, and activation layer, and the result is recorded as F3. F3 passes through a convolution block, and the result is recorded as F4. F4 passes through a convolution block, and the result is recorded as F5. F5 passes through a convolution block, and the result is recorded as F6. F1 passes through another self-attention layer, and the result is recorded as F7. F2, F3, F4, F5, F6, and F7 are spliced ​​as the output of the ARM2 module. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 APL-Net overall network structure diagram in the embodiment;

[0069] Figure 2 Schematic diagram of the structure of the adaptive image filtering module AIF in the embodiment;

[0070] Figure 3 This is a schematic diagram of the structure of the parameter prediction module PPM in the embodiment;

[0071] Figure 4 This is a schematic diagram of the structure of a basic image filtering module BIF in the embodiment;

[0072] Figure 5 This is a schematic diagram of the loss calculation in the second stage of the embodiment;

[0073] Figure 6 This is a schematic diagram of the loss calculation in the third stage in the embodiment;

[0074] Figure 7 This is a structural diagram of the attention detection module ADE;

[0075] Figure 8 This is a schematic diagram of the attention residual module ARM structure;

[0076] Fig. 9 It is a schematic diagram of the maximum pooling convolution module MPC structure;

[0077] Fig.10 This is a schematic diagram of the structure of the attention residual module 2 ARM2; DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present invention more clear, the content of the present invention is further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0079] 1) The low-light image dataset ExDark, the foggy dataset RTTS, and the clear image dataset VOC2012 are processed and divided as follows:

[0080] 1-1) The ExDark dataset is divided into 8:2 ratios, 80% of which is the training set ExDark-Train, with a total of 5891 images; 20% is the test set ExDark-Test, with a total of 1472 images;

[0081] 1-2) The RTTS dataset is divided into 8:2 ratios, 80% of which is the training set RTTS-Train, with a total of 3458 images; 20% is the test set RTTS-Test, with a total of 864 images;

[0082] 1-3) The VOC2012 dataset is divided into a ratio of 8:2, of which 80% is the training set VOC-Train, with a total of 13,700 images; 20% is the test set VOC-Test, with a total of 3,425 images;

[0083] 2) In the first stage, all VOC training set VOC-Train images are fed into APL-Net (see Figure 1 ), train Teacher1-Net, so that Teacher1-Net has the ability to detect targets under normal illumination clear images. The image is first passed through the adaptive image filter AIF (see Figure 2 ), and then pass through the detection module ADE (see Figure 7 ) for training, specifically:

[0084] 2-1) The image is downsampled to a 128*128 thumbnail and input into the parameter prediction module PPM (see Figure 3 ), after three self-attention layers and then two fully connected layers, the output P is obtained, which contains four parameters, namely α, β, ω, and δ, which are used as parameters for dehazing, brightening, contrast, and sharpening enhancement, respectively.

[0085] 2-2) Input the image into the basic image filter BIF (see Figure 4 ), a series of enhancements are performed on the image in BIF. First, the image is histogram equalized, and the equalized image is defogged. The calculation process of defogging enhancement is shown in formula (1):

[0086]

[0087] In formula (1), J(x) represents the clear image after defogging, I(x) represents the image to be defogged, t(x) is the transmittance, and A is the global atmospheric light component. A is estimated by the average brightness of the corresponding position pixels of the brightest 1000 pixels in the dark channel image of I(x). The estimated value of t(x) is shown in formula (2):

[0088]

[0089] In formula (2), Ω(x) represents a window centered at pixel x, and c represents the three channels of R, G, and B. α is calculated by step 2-1).

[0090] The image obtained after dehazing is enhanced in brightness, and the image brightness is adjusted by adjusting the image Gamma value. The specific calculation process is shown in formula (3):

[0091] P y =P i β (3)

[0092] Formula (3) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,B y ), where parameter β is calculated by step 2-1).

[0093] The image after brightness enhancement is contrast enhanced. The calculation process is shown in formulas (4) and (5):

[0094]

[0095] EL(P i )=0.5(1-cos(π(0.27R i +0.67G i +0.06B i ))) (5)

[0096] Formula (4) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,B y ), where the parameter ω is calculated by step 2-1).

[0097] The image after contrast enhancement is sharpened, and the calculation process is shown in formula (6):

[0098] F(x)=I(x)+δ(I(x)-G(I(x)))(6)

[0099] Where F is the calculation result, I is the input, G is the Gaussian filter, and the parameter δ is calculated by step 2-1).

[0100] 2-3) The enhanced image obtained in step 2-2) is tested by the ADE detection module. In the ADE module, it first passes through four CBS modules (convolution, batch normalization, SiLu activation), and then passes through the attention residual module ARM (see Figure 8 ), and then through the maximum pooling convolution module MPC (see Fig. 9 ), and then pass through the module ARM, the result is recorded as C1, C1 passes through MPC and ARM in turn, the result is recorded as C2, C2 passes through MPC and ARM in turn, the result is recorded as C3; C3 passes through the convolution module CBS, and then passes through the ARM2 module (see Fig.10 ), the result is recorded as C4; C2 is spliced ​​with C4 in the channel direction through CBS, and the spliced ​​result is input into the ARM2 module, and the result is recorded as C5. C1 is spliced ​​with CBS and C5 through CBS and UPS (up-sampling operation) in turn, and the two results are spliced, and the result is recorded as C6. C6 is input into the MPC module and spliced ​​with C5, and the spliced ​​result is input into ARM2, and the result is recorded as C7. C7 is input into the MPC module and spliced ​​with C4, and the spliced ​​result is input into ARM2, and the result is recorded as C8. C6, C7, and C8 are sent to the detection head Head for detection. Finally, the labeling box loss GTL is calculated and the parameters are updated by back propagation. The loss function is shown in formula (7):

[0101] GTL=0.1L conf +0.125L cls +0.05L loc (7)

[0102] Where L conf represents the classification loss, which is the category confidence loss calculated using the binary cross entropy loss function; L cls is the positioning loss, using CIOU Loss to calculate the error between the predicted box and the real box, L loc is the confidence loss, and the binary cross entropy loss function is used to calculate the target confidence loss.

[0103] 2-4) After several rounds of training, Teacher1-Net can detect targets in clear images with normal illumination.

[0104] 3) In the second stage, all the images from the VOC training set VOC-Train, the low-light image dataset ExDark-Train, and the foggy image dataset RTTS-Train are mixed and sent to APL-Net to train Teacher2-Net, and Teacher1-Net is used to perform knowledge distillation on Teacher2-Net. If the image comes from the VOC-Train clear dataset, the clear image is input into Teacher1-Net, and the clear image is added with moderate fog and low-light effects and input into Teacher2-Net to calculate the perceptual loss PL of Teacher1-Net and Teacher2-Net and the annotation box loss GTL of Teacher2-Net for the blurred image; if the image comes from the low-light image dataset ExDark-Train, a slight fog is added to the image and input into Teacher2-Net to calculate the annotation box loss GTL; if the image comes from the foggy image dataset RTTS-Train, a slight low light is added to the image and input into Teacher2-Net to calculate the annotation box loss GTL. The total loss L = PL + GTL, and the specific loss calculation method is as follows: Figure 5 As shown. Through training, Teacher2-Net has the ability to detect targets in images with low illumination and shallow fog, specifically:

[0105] 3-1) If the image comes from the VOC-Train clear dataset, the image is first passed through the low-light module LowLight-Gen to add moderate low light, and the parameter x is set to a random number greater than 0.7 and less than 0.85. The calculation process is shown in formula (8):

[0106]

[0107] Formula (8) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,B y ).

[0108] Then, moderate fog is added through the fog module Fog-Gen, and the parameter x is set to a random number greater than 0.25 and less than 0.75. The calculation process is shown in formula (9):

[0109] g(x)=t(x,θ)I(x)+(1-t(x,θ))A (9)

[0110] Where A is a constant of 0.5, I(x) is a clear image, g(x) is a foggy image, and the calculation process of t(x,θ) is shown in formula (10):

[0111] t(x,θ)=exp(-θd(x)) (10)

[0112] The calculation process of d(x) is shown in formula (11):

[0113]

[0114] Where W, H are the width and height of the image, x is the current pixel position, and c is the center position of the image.

[0115] The image with moderate low light and moderate fog is input to Teacher2-Net, while the original clear image is input to Teacher1-Net.

[0116] If the image comes from the ExDark-Train dataset, the image is first passed through the fog module Fog-Gen to add light fog. The parameter x is set to a random number greater than 0 and less than 0.5. The calculation process is shown in formula (9). The obtained image is input into Teacher2-Net.

[0117] If the image comes from the RTTS-Train dataset, the image is first passed through the low-light module LowLight-Gen to add mild low light. The parameter x is set to a random number greater than 0.8 and less than 0.95. The calculation process is shown in formula (8). The obtained image is input into Teacher2-Net.

[0118] 3-2) In Teacher1-Net, the input image first passes through the adaptive image filter AIF described in step 2) and then passes through the ADE module for detection; in Teacher2-Net, the input image first passes through the adaptive image filter AIF described in step 2) and then passes through the ADE module for detection.

[0119] 3-3) Use the first 4 layers of the ADE module in Teacher1-Net and Teacher2-Net in step 3-2) to calculate the perceptual loss PL. The calculation process is shown in formula (12):

[0120]

[0121] Among them, T1 u Represents the calculation result of the uth layer of Teacher1-Net, T2 u represents the calculation result of the uth layer of Teacher2-Net, a u Represents the number of activated neurons in layer u. If the original image comes from ExDark-Train or RTTS-Train, P is 0.

[0122] Then use the output result of Teacher2-Net in step 3-2) and the label value in the data set to calculate the annotation box loss GTL through formula (7), and the total loss function L = PL + GTL.

[0123] 4) In the third stage, all VOC training set VOC-Train, low-light image dataset ExDark-Train, and fog image dataset RTTS-Train images are mixed and sent to APL-Net to train Student-Net, and Teacher1-Net and Teacher2-Net are used to jointly distill the Student network. If the image comes from the VOC-Train clear dataset, the clear image is input into Teacher1-Net, moderate fog and low light effects are added to the clear image and input into Teacher2-Net, and heavy low light and fog are added to the clear image and input into Student-Net to obtain the joint perceptual loss P1L of Teacher1-Net, Teacher2-Net and Student-Net and the annotation box loss GTL of Student-Net; if the image comes from the low-light image dataset ExDark-Train, the image is input into Teacher2-Net, and heavy fog is added to the image and input into Student-Net to obtain the perceptual loss P2L of Teacher2-Net and Student-Net and the annotation box loss GTL of Student-Net; if the image comes from the foggy image dataset RTTS-Train, the image is input into Teacher2-Net, and heavy low light is added to the image and input into Student-Net to obtain the perceptual loss P2L of Teacher2-Net and Student-Net and the annotation box loss GTL of Student-Net. The total loss function L = P1L + P2L + GTL. The specific loss calculation method is as follows: Figure 6 As shown. Through the joint guidance of Teacher1-Net and Teacher2-Net on the training of Student-Net, Student-Net has the ability to detect targets in low-light foggy images, specifically:

[0124] 4-1) If the image comes from the VOC-Train clear dataset, the image is first passed through the low-light module LowLight-Gen to add moderate low light, and the parameter x is set to a random number greater than 0.7 and less than 0.85. The calculation process is shown in formula (8). Then, the image is passed through the fog module Fog-Gen to add moderate fog, and the parameter x is set to a random number greater than 0.25 and less than 0.75. The calculation process is shown in formula (9). At the same time, the clear source image is passed through the low-light module LowLight-Gen to add heavy low light, and the parameter x is set to a random number greater than 0.6 and less than 0.75. The calculation process is shown in formula (8). Then, the image is passed through the fog module Fog-Gen to add heavy fog, and the parameter x is set to a random number greater than 0.6 and less than 0.95. The calculation process is shown in formula (9). Finally, the clear source image is directly input into Teacher1-Net, the image obtained by adding moderate fog and low light is input into Teacher2-Net, and the image obtained by adding heavy fog and low light is input into Student-Net.

[0125] If the image comes from the ExDark-Train dataset, the image is first passed through the fog module Fog-Gen to add heavy fog. The parameter is set to a random number greater than 0.6 and less than 0.95. The calculation process is shown in formula (9). The obtained image is input into Student-Net, and the low-light source image is directly input into Teacher2-Net.

[0126] If the image comes from the RTTS-Train dataset, the image is first passed through the low-light module LowLight-Gen to add heavy low light. The parameter x is set to a random number greater than 0.6 and less than 0.75. The calculation process is shown in formula (8). The obtained image is input into Student-Net, and the foggy source image is directly input into Teacher2-Net.

[0127] 4-2) Calculate the loss function and update the weights. The total loss L = P1L + P2L + GTL, where P1L is as shown in formula (13):

[0128]

[0129] Among them, T1 u Represents the calculation result of the uth layer of Teacher1-Net, T2 u represents the calculation result of the uth layer of Teacher2-Net, S u represents the calculation result of the uth layer of Student-Net, a u Represents the number of activated neurons in layer u. If the original image comes from ExDark-Train or RTTS-Train, P is 0.

[0130] P2L is shown in formula (14):

[0131]

[0132] Among them, T2 u represents the calculation result of the uth layer of Teacher2-Net, S u represents the calculation result of the uth layer of Student-Net, a u Represents the number of activated neurons in layer u.

[0133] GTL is shown in formula (7).

[0134] 5) Mix VOC-Test, ExDark-Test, and RTTS-Test and input them into Student-Net for testing.

Claims

1. A method for detecting image targets in low-light superimposed fog environment based on APL-Net three-stage distillation, comprising the following steps: 1) The low-light image dataset ExDark, the foggy dataset RTTS, and the clear image dataset VOC2012 are processed and divided as follows: 1-1) The ExDark dataset is divided into 8:2 ratios, 80% of which is the training set ExDark-Train, with a total of 5891 images; 20% is the test set ExDark-Test, with a total of 1472 images; 1-2) The RTTS dataset is divided into 8:2 ratios, 80% of which is the training set RTTS-Train, with a total of 3458 images; 20% is the test set RTTS-Test, with a total of 864 images; 1-3) The VOC2012 dataset is divided into a ratio of 8:2, of which 80% is the training set VOC-Train, with a total of 13,700 images; 20% is the test set VOC-Test, with a total of 3,425 images; 2) In the first stage, all VOC training set VOC-Train images are sent to APL-Net (see Figure 1) to train Teacher1-Net, so that Teacher1-Net has the ability to detect targets under normal illumination clear images. The images are first passed through the adaptive image filter AIF (see Figure 2) and then through the detection module ADE (see Figure 7) for training. Specifically: 2-1) The image is downsampled to a 128*128 thumbnail and input into the parameter prediction module PPM (see Figure 3). After three self-attention layers and two fully connected layers, the output P is obtained, which contains four parameters, namely α, β, ω, and δ, which are used as parameters for dehazing, brightening, contrast, and sharpening enhancement, respectively. 2-2) The image is input into the basic image filter BIF (see Figure 4), and a series of enhancements are performed on the image in BIF. First, the image is histogram equalized, and the equalized image is defogged. The calculation process of defogging enhancement is shown in formula (1): In formula (1), J(x) represents the clear image after defogging, I(x) represents the image to be defogged, t(x) is the transmittance, and A is the global atmospheric light component. A is estimated by the average brightness of the corresponding position pixels of the brightest 1000 pixels in the dark channel image of I(x). The estimated value of t(x) is shown in formula (2): In formula (2), Ω(x) represents a window centered at pixel x, and c represents the three channels of R, G, and B. α is calculated by step 2-1). The image obtained after dehazing is enhanced in brightness, and the image brightness is adjusted by adjusting the image Gamma value. The specific calculation process is shown in formula (3): Formula (3) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,B y ), where parameter β is calculated by step 2-1). The image after brightness enhancement is contrast enhanced. The calculation process is shown in formulas (4) and (5): EL(P i )=0.5(1-cos(π(0.27R i +0.67C i +0.06B i ))) (5) Formula (4) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,B y ), where the parameter ω is calculated by step 2-1). The image after contrast enhancement is sharpened, and the calculation process is shown in formula (6): F(x)=I(x)+δ(I(x)-G(I(x))) (6) Where F is the calculation result, I is the input, G is the Gaussian filter, and the parameter δ is calculated by step 2-1). 2-3) The enhanced image obtained in step 2-2) is tested by the ADE detection module. In the ADE module, it first passes through four CBS modules (convolution, batch normalization, SiLu activation), then passes through the attention residual module ARM (see Figure 8), then passes through the maximum pooling convolution module MPC (see Figure 9), and then passes through the module ARM, the result is recorded as C1, C1 passes through MPC and ARM in turn, the result is recorded as C2, C2 passes through MPC and ARM in turn, the result is recorded as C3; C3 passes through the convolution module CBS, and then passes through the ARM2 module (see Figure 10), the result is recorded as C4; C2 passes through C BS and C4 are spliced ​​in the channel direction, and the spliced ​​result is input into the ARM2 module, and the result is recorded as C5. C1 passes through CBS and C5 in turn through CBS and UPS (up-sampling operation), and the two results are spliced, and the result is recorded as C6. C6 is input into the MPC module and spliced ​​with C5, and the spliced ​​result is input into ARM2, and the result is recorded as C7. C7 is input into the MPC module and spliced ​​with C4, and the spliced ​​result is input into ARM2, and the result is recorded as C8. C6, C7, and C8 are sent to the detection head Head for detection. Finally, the labeling box loss GTL is calculated and the parameters are updated by back propagation. The loss function is shown in formula (7): GTL=0.1L conf +0.125L cls +0.05L loc (7) Where L conf represents the classification loss, which is the category confidence loss calculated using the binary cross entropy loss function; L cls is the positioning loss, using CIOU Loss to calculate the error between the predicted box and the real box, L loc is the confidence loss, and the binary cross entropy loss function is used to calculate the target confidence loss. 2-4) After several rounds of training, Teacher1-Net can detect targets in clear images with normal illumination. 3) In the second stage, all the images of the VOC training set VOC-Train, the low-light image dataset ExDark-Train, and the fog image dataset RTTS-Train are mixed and sent to APL-Net to train Teacher2-Net, and Teacher1-Net is used to perform knowledge distillation on Teacher2-Net. If the image comes from the VOC-Train clear dataset, the clear image is input into Teacher1-Net, and the clear image is added with moderate fog and low light effects and input into Teacher2-Net to calculate the perceptual loss PL of Teacher1-Net and Teacher2-Net and the annotation box loss GTL of Teacher2-Net for the blurred image; if the image comes from the low-light image dataset ExDark-Train, light fog is added to the image and input into Teacher2-Net to calculate the annotation box loss GTL; if the image comes from the fog image dataset RTTS-Train, light low light is added to the image and input into Teacher2-Net to calculate the annotation box loss GTL. The total loss L = PL + GTL, and the specific loss calculation method is shown in Figure 5. Through training, Teacher2-Net has the ability to detect targets in images with low illumination and shallow fog, specifically: 3-1) If the image comes from the VOC-Train clear dataset, the image is first passed through the low-light module LowLight-Gen to add moderate low light, and the parameter x is set to a random number greater than 0.7 and less than 0.

85. The calculation process is shown in formula (8): Formula (8) converts the input pixel value P i =(R i ,G i ,B i ) is mapped to P y =(R y ,G y ,B y ). Then, moderate fog is added through the fog module Fog-Gen, and the parameter x is set to a random number greater than 0.25 and less than 0.

75. The calculation process is shown in formula (9): g(x)=t(x,θ)I(x)+(1-t(x,θ))A (9) Where A is a constant of 0.5, I(x) is a clear image, g(x) is a foggy image, and the calculation process of t(x,θ) is shown in formula (10): t(x,θ)=exp(-θd(x)) (10) The calculation process of d(x) is shown in formula (11): Where W, H are the width and height of the image, x is the current pixel position, and c is the center position of the image. The image with moderate low light and moderate fog is input to Teacher2-Net, while the original clear image is input to Teacher1-Net. If the image comes from the ExDark-Train dataset, the image is first passed through the fog module Fog-Gen to add light fog. The parameter x is set to a random number greater than 0 and less than 0.

5. The calculation process is shown in formula (9). The obtained image is input into Teacher2-Net. If the image comes from the RTTS-Train dataset, the image is first passed through the low-light module LowLight-Gen to add mild low light. The parameter x is set to a random number greater than 0.8 and less than 0.

95. The calculation process is shown in formula (8). The obtained image is input into Teacher2-Net. 3-2) In Teacher1-Net, the input image first passes through the adaptive image filter AIF described in step 2) and then passes through the ADE module for detection; in Teacher2-Net, the input image first passes through the adaptive image filter AIF described in step 2) and then passes through the ADE module for detection. 3-3) Use the first 4 layers of the ADE module in Teacher1-Net and Teacher2-Net in step 3-2) to calculate the perceptual loss PL. The calculation process is shown in formula (12): Among them, T1 u Represents the calculation result of the uth layer of Teacher1-Net, T2 u represents the calculation result of the uth layer of Teacher2-Net, a u Represents the number of activated neurons in layer u. If the original image comes from ExDark-Train or RTTS-Train, P is 0. Then use the output result of Teacher2-Net in step 3-2) and the label value in the data set to calculate the annotation box loss GTL through formula (7), and the total loss function L = PL + GTL. 4) In the third stage, all VOC training set VOC-Train, low-light image dataset ExDark-Train, and fog image dataset RTTS-Train images are mixed and sent to APL-Net to train Student-Net, and Teacher1-Net and Teacher2-Net are used to jointly distill the Student network. If the image comes from the VOC-Train clear dataset, the clear image is input to Teacher1-Net, the clear image with moderate fog and low light effects is input to Teacher2-Net, and the clear image with heavy low light and fog is input to Student-Net, and the joint perceptual loss P of Teacher1-Net, Teacher2-Net and Student-Net is calculated. 1 L and the annotation box loss GTL of Student-Net; if the image comes from the low-light image dataset ExDark-Train, the image is input into Teacher2-Net, and heavy fog is added to the image and input into Student-Net to find the perceptual loss P of Teacher2-Net and Student-Net 2 L and the annotation box loss GTL of Student-Net; if the image comes from the fog image dataset RTTS-Train, the image is input into Teacher2-Net, and heavy low light is added to the image and input into Student-Net to find the perceptual loss P of Teacher2-Net and Student-Net 2 L and Student-Net’s annotation box loss GTL. Total loss function L = P 1 L+P 2 L+GTL, the specific loss calculation method is shown in Figure 6. Through the joint guidance of Teacher1-Net and Teacher2-Net on the training of Student-Net, Student-Net has the ability to detect targets in low-light foggy images, specifically: 4-1) If the image comes from the VOC-Train clear dataset, the image is first passed through the low-light module LowLight-Gen to add moderate low light, and the parameter x is set to a random number greater than 0.7 and less than 0.

85. The calculation process is shown in formula (8). Then, the image is passed through the fog module Fog-Gen to add moderate fog, and the parameter x is set to a random number greater than 0.25 and less than 0.

75. The calculation process is shown in formula (9). At the same time, the clear source image is passed through the low-light module LowLight-Gen to add heavy low light, and the parameter x is set to a random number greater than 0.6 and less than 0.

75. The calculation process is shown in formula (8). Then, the image is passed through the fog module Fog-Gen to add heavy fog, and the parameter x is set to a random number greater than 0.6 and less than 0.

95. The calculation process is shown in formula (9). Finally, the clear source image is directly input into Teacher1-Net, the image obtained by adding moderate fog and low light is input into Teacher2-Net, and the image obtained by adding heavy fog and low light is input into Student-Net. If the image comes from the ExDark-Train dataset, the image is first passed through the fog module Fog-Gen to add heavy fog. The parameter is set to a random number greater than 0.6 and less than 0.

95. The calculation process is shown in formula (9). The obtained image is input into Student-Net, and the low-light source image is directly input into Teacher2-Net. If the image comes from the RTTS-Train dataset, the image is first passed through the low-light module LowLight-Gen to add heavy low light. The parameter x is set to a random number greater than 0.6 and less than 0.

75. The calculation process is shown in formula (8). The obtained image is input into Student-Net, and the foggy source image is directly input into Teacher2-Net. 4-2) Calculate the loss function and update the weights. Total loss L = P 1 L+P 2 L+GTL, where P 1 L is shown in formula (13): Among them, T1 u Represents the calculation result of the uth layer of Teacher1-Net, T2 u represents the calculation result of the uth layer of Teacher2-Net, S u represents the calculation result of the uth layer of Student-Net, a u Represents the number of activated neurons in layer u. If the original image comes from ExDark-Train or RTTS-Train, P is 0. P 2 L is shown in formula (14): Among them, T2 u represents the calculation result of the uth layer of Teacher2-Net, S u represents the calculation result of the uth layer of Student-Net, a u Represents the number of activated neurons in layer u. GTL is shown in formula (7). 5) Mix VOC-Test, ExDark-Test, and RTTS-Test and input them into Student-Net for testing.

2. According to the image target detection method in low-light superimposed fog environment based on APL-Net three-stage distillation according to claim 1, Features The PPM module in step 2-1) is specifically: The module input is a 128*128*3 thumbnail image. After three self-attention layers, the result is a 128*128*3 feature map. After two fully connected layers, the output is a vector of length 4.

3. According to the image target detection method in low-light superimposed fog environment based on APL-Net three-stage distillation according to claim 1, Features The ARM module in step 2-3) is specifically: The input feature map F1 passes through a self-attention layer, and the result is recorded as F2. F2 passes through two convolution blocks, namely convolution layer, batch normalization layer, and activation layer, and the result is recorded as F3. F3 passes through two convolution blocks, and the result is recorded as F4. F1 passes through another self-attention layer, and the result is recorded as F5. F2, F3, F4, and F5 are spliced ​​as the output of the ARM module.

4. According to the image target detection method in low-light superimposed fog environment based on APL-Net three-stage distillation according to claim 1, Features The ARM2 module in step 2-3) is specifically: The input feature map F1 passes through a self-attention layer, and the result is recorded as F2. F2 passes through a convolution block, namely the convolution layer, batch normalization layer, and activation layer, and the result is recorded as F3. F3 passes through a convolution block, and the result is recorded as F4. F4 passes through a convolution block, and the result is recorded as F5. F5 passes through a convolution block, and the result is recorded as F6. F1 passes through another self-attention layer, and the result is recorded as F7. F2, F3, F4, F5, F6, and F7 are spliced ​​as the output of the ARM2 module.

Citation Information

Cited By

  • Target detection method, detection system, vehicle and visual environment monitoring device thereof

    CN121074801A