Semi-supervised domain adaptation based pedestrian detection method in low-light environment

Abstract

The present application relates to a kind of semi-supervised domain adaptation's weak light environment pedestrian detection method of combining infrared image and visible light image, belong to the technical field of pedestrian detection.The present application includes the following steps: S1: propose the pedestrian detection method in weak light environment, including the following small steps: S11: establish pseudo-image generation network structure;S12: establish MT-YOLOv8 network structure;S13: introduce the mixed attention module based on Transformer;S14: introduce supervision loss function and distillation loss function;S2: using public dataset carries out model evaluation, including the following small steps: S21: KAIST data set is cleaned;S22: the selection of evaluation index;S23: the verification of evaluation effectiveness.The present application can effectively solve the problem of pedestrian detection in weak light environment, and the combined attention module can also improve the detection accuracy to a certain extent, and can be used for accurate detection occasion in weak light environment.

Description

Technical Field

[0001] This invention relates to a semi-supervised adaptive pedestrian detection method in low-light environments that combines infrared and visible light images, belonging to the technical field of pedestrian detection. Background Technology

[0002] Pedestrian detection utilizes computer vision technology to determine the presence and precise location of pedestrians in images or video sequences, and is a crucial research area in autonomous driving technology. With the development of autonomous driving technology, driverless vehicles must accurately perceive their surroundings, especially pedestrians, to ensure driving safety. Therefore, accurate and low-latency pedestrian detection has become a critical issue in the field of autonomous driving.

[0003] Traditional pedestrian detection algorithms perform poorly in low-light environments, prompting researchers to make significant efforts to improve their performance. Currently, multispectral pedestrian detection algorithms utilizing visible light and infrared thermal imaging have attracted increasing attention from experts and scholars because these two technologies can provide more valuable contour information, enhancing the accuracy and robustness of pedestrian detection algorithms. Some research institutions employ a parallel dual-stream mode of visible light and infrared thermal imaging in their multimodal pedestrian detection algorithms. Leveraging the complementarity of visible light and infrared thermal imaging information, they perform image fusion pre-training before model training, then feed the fused image into the pedestrian detection training network for training, resulting in a model with better performance. Cao Hongyan et al. added feature scales to the YOLOv3 algorithm and fused BN network layers with convolutional neural network layers to improve infrared image detection accuracy. Han Hangdi et al. enhanced the feature extraction network of Tiny-YOLOv3 by introducing the MobileNet model, improving the model's high-level semantic feature extraction capabilities and increasing the speed and accuracy of infrared image defect detection. Chen Haiyun et al. designed a spatial channel hybrid attention module and performed deconvolution at the output detection head to expand the output feature map, improving the accuracy of small object detection in infrared images. Fanglin Bao et al. proposed a thermal-assisted detection and ranging (HADAR) technique from the perspective of thermal imaging. This technique utilizes artificial intelligence to analyze thermal signals, thereby clearly identifying the temperature, material, and texture of objects in dark or low-visibility environments. HADAR technology can provide more physical information, including not only temperature but also emissivity and depth. Although the proposed structure can effectively improve the accuracy of pedestrian detection in low-light environments, the challenge of fusing information from two modalities while simultaneously improving detection speed remains. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention proposes a semi-supervised adaptive pedestrian detection method in low-light environments that combines infrared and visible light images.

[0005] The semi-supervised domain adaptive pedestrian detection method in low-light environments described in this invention includes the following steps:

[0006] S1: A pedestrian detection method for low-light environments is proposed, including the following steps:

[0007] S11: Establishing a Pseudo-Image Generation Network Structure: This invention proposes a method for training pseudo-image generation by combining Darkness-free Infrared and Visible Image Fusion (DIVFusion) and Cycle-Consistent Generative Adversarial Networks (CycleGAN). The method extracts the Scene-illumination Disentangled Network (SIDNet) and Texture-contrast Enhancement Fusion Network (TCEFNet) from the DIVFusion network. SIDNet and TCEFNet are then combined with the image input part of CycleGAN. Before image input, the image is fused using SIDNet and TCEFNet. The fused image is then input into CycleGAN for domain transfer to generate pseudo-images.

[0008] S12: Establish the MT-YOLOv8 network structure: Using the Mean Teacher network as the basic framework, a semi-supervised pedestrian detection network is built by introducing the YOLOv8 single-level detector, and a Transformer-based hybrid attention module is introduced to improve the resolution of pseudo images, activate more useful pixels, and further improve detection accuracy.

[0009] S13: Introducing a Transformer-based hybrid attention module: This module combines channel attention and window-based self-attention, leveraging their complementary advantages. Furthermore, to better aggregate cross-window information, an overlapping cross-attention module is introduced to enhance the interaction between features of adjacent windows. This extends the Transformer-based hybrid attention module to more image restoration applications, including improving image super-resolution, Gaussian image denoising, and reducing image compression artifacts.

[0010] S14: Introduction of supervised loss function and distillation loss function: This invention uses the Mean Teacher semi-supervised framework, introduces YOLOv8 as the network detector, and uses the student model to supervise the training of source domain infrared images. The corresponding source domain supervised loss function is defined as follows:

[0011] loss det =loss box (S;L s )+loss cls (S;C s (1)

[0012] In the formula: S represents the set of source domain images;

[0013] L s Represents the set of bounding boxes of the source domain image;

[0014] C s Represents the set of category labels for source domain images;

[0015] loss box The regression loss for predicting bounding boxes is a combination of distributed focusing loss and complete loss.

[0016] loss cls The binary cross-entropy loss represents the classification loss;

[0017] When using a student model to supervise the training of pseudo-infrared images in the source domain, the corresponding pseudo-source domain supervised loss function is defined as:

[0018]

[0019] In the formula: S f Represents a set of pseudo-source domain images;

[0020] Represents the set of bounding boxes for pseudo-source domain images;

[0021] Represents the set of category labels for pseudo-source domain images;

[0022] The target domain image is input into both the student and teacher models for training. During knowledge distillation, high-confidence bounding boxes obtained from the teacher model's predictions are used as pseudo-labels to reduce bias in the target domain image and enhance the model's robustness. The target domain images input to the teacher and student models are used for prediction; the difference between the two model predictions is calculated using the target domain distillation loss function, defined as:

[0023] loss dis =loss det {T s Gbox [L box (T t )],G cls [L cls (T t (3)

[0024] In the formula: T s This represents the target domain image input to the student model;

[0025] T t This represents the target domain image input to the teacher model;

[0026] L box (T t ) represents the bounding box prediction branch from the teacher model;

[0027] L cls (T t ) represents the category prediction branch from the teacher model;

[0028] G box and G cls These represent bounding box filters and category filters, respectively.

[0029] The pseudo-target domain image is used for training the teacher model. To enable it to learn global image features from the source domain, the pseudo-target domain image replaces the target domain image, while the labeled image remains unchanged in the part used to train the student model. The pseudo-target domain distillation loss is then defined as:

[0030]

[0031] In the formula: This represents the pseudo-target domain image input to the teacher model;

[0032] S2: Evaluate the model using a public dataset, including the following steps:

[0033] S21: Data cleaning of the KAIST dataset: The KAIST dataset is taken from consecutive frames of video images, and adjacent images are relatively close, so a certain degree of cleaning is required. This invention mainly studies pedestrian detection in low light conditions, so only the nighttime dataset of KAIST is cleaned for the experiment. The cleaning rules are as follows: (1) Take one image from every two images in the training set, that is, take one image from every three images, and remove all images that do not contain any pedestrians. That is, the selected images must contain at least one target, and remove images in the dataset that only show half of the body or have severe occlusion such as less than 50 pixels. (2) Take one image from every 19 images in the test set, that is, take one image from every 20 images, and retain negative sample images. After the above operations, 2846 training set images and 797 test set images are obtained;

[0034] S22: Selection of Evaluation Metrics: Evaluation metrics include precision, recall, and average precision, among which:

[0035] Accuracy is defined as:

[0036]

[0037] Recall rate is defined as:

[0038]

[0039] The definition of average precision is:

[0040]

[0041] In the formula: TP represents the number of positive samples successfully detected by the model;

[0042] FP represents the number of negative samples that the model incorrectly detected as positive samples;

[0043] FN indicates the number of positive samples that the model did not detect;

[0044] k represents the number of categories;

[0045] AP i This represents the average accuracy, where i = 0.5;

[0046] S23: Validation of effectiveness: The MT-YOLOv8 network was compared with SSDA-YOLO and Efficient Teacher networks using the LLVIP and KAIST datasets. The MT-YOLOv8 network achieved the best detection performance in all cases, with significantly higher prediction accuracy, recall and mean precision than other models.

[0047] Preferably, in step S11, the infrared image is defined as the source domain and the visible light image is defined as the target domain. A pseudo-image generation network is used to perform domain transfer between the two domains, generating pseudo-infrared images and pseudo-visible light images. Correspondingly, the pseudo-infrared image is defined as the pseudo-source domain and the pseudo-visible light image is defined as the pseudo-target domain. Pseudo-cross training is performed using the source domain, target domain, pseudo-source domain, and pseudo-target domain.

[0048] Preferably, in step S12, YOLOv8 is introduced as the detector for both the student and teacher models within the Mean Teacher semi-supervised framework. The student model is used for supervised training, and the teacher model is used for unsupervised training. The teacher model uses the exponential moving average (EMA) weights of the student model, instead of sharing weights with the student model. The consistency cost J is defined as the expected distance between the predictions of the student model and the teacher model.

[0049] J(θ)=Ε x,η',η [||f(x,θ',η')-f(x,θ,η)|| 2 (8)

[0050] In the formula, θ represents the prediction weight of the student model;

[0051] η represents the prediction noise of the student model;

[0052] θ' represents the prediction weights of the teacher model;

[0053] η' represents the prediction noise of the teacher model;

[0054] When θ' = θ, the weighted average of the time set approximation f(x,θ',η') and the continuous prediction, with the EMA of the continuous weights θ at step t, is θ′. t ,but:

[0055] θ′ t =αθ′ t-1 +(1-α)θ t (9)

[0056] Preferably, in step S21, the LLVIP dataset contains 30,976 images, each with both RGB and infrared versions (15,488 pairs). This dataset is strictly aligned in time and space, with an image resolution of 1280×1024. Since pedestrians in the dataset are already labeled, data cleaning is not required. The KAIST pedestrian dataset contains a total of 95,328 images, each 640×480 pixels, with both RGB and infrared versions. It contains a total of 103,128 dense annotations. The dataset captures various typical traffic scenes, including those in campuses, streets, and rural areas, both during the day and at night. Because this dataset is derived from consecutive video frames, adjacent images are relatively close, thus requiring some degree of data cleaning.

[0057] Preferably, in step S23, the SSDA-YOLO network is a semi-supervised adaptive pedestrian detection algorithm proposed by combining the YOLOv5 detector with a Mean Teacher network; the Efficient Teacher network is an efficient knowledge distillation method used to improve the performance of the target detection model by using the knowledge of the teacher model to guide the training of the student model. Comparative experiments verified that the present invention achieved optimal detection performance in the field of pedestrian detection in semi-supervised adaptive low-light environments. Attached Figure Description

[0058] Figure 1 This is a structural diagram of the pseudo-image generation network of the present invention;

[0059] Figure 2(a) is the original image of the dataset used in this invention;

[0060] Figure 2(b) is a visual comparison of the original image after CycleGAN operation;

[0061] Figure 2(c) is a visual comparison of the original image after the pseudo-image generation operation;

[0062] Figure 3 This is a structural diagram of the Transformer-based hybrid attention module of the present invention;

[0063] Figure 4 This is an overall structural diagram of the MT-YOLOv8 network of this invention;

[0064] Figure 5(a) is a visual comparison of the detection results without domain adaptation;

[0065] Figure 5(b) is a visualization comparison of the detection results of the SSDA-YOLO model;

[0066] Figure 5(c) is a visualization comparison of the detection results of the Efficient Teacher model;

[0067] Figure 5(d) is a visualization comparison of the detection results of the MT-YOLOv8 model; Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1:

[0070] Figures 1 to 2(c) The diagram illustrates a pseudo-image generation method. This invention aims to establish a pseudo-image generation network that reduces the differences between the infrared and visible light domains. The SIDNet and TCEFNet modules from the DIVFusion network are extracted and combined with the image input part of CycleGAN. Before the image input, it undergoes image fusion processing via SIDNet and TCEFNet. The fused image is then input into CycleGAN for domain transfer to generate pseudo-images. Using the DIVFusion network for image fusion increases the detail information between the infrared and visible light images, allowing for the generation of more accurate pseudo-images. Adversarial training using the discriminator network in the CycleGAN algorithm reduces the domain difference between pseudo-images, resulting in a smaller difference between the generated and original images.

[0071] Figure 2(a) shows the original images of the dataset used in this invention. The datasets used in this invention consist of paired infrared and visible light images. Figure 2(b) shows the pseudo-images generated by directly performing domain transfer using the original CycleGAN network, i.e., transferring the visible light domain to the infrared domain and vice versa. It can be seen that the transfer effect is very poor; even the human eye cannot distinguish the target, and the target outline is blurred. Figure 2(c) shows the pseudo-images generated by the pseudo-image generation network of this invention using domain transfer. It can be seen that the transfer effect is ideal, the pedestrian outline is clear, and the target can be clearly distinguished. In summary, the pseudo-image generation network proposed in this invention can effectively perform the task of converting between the infrared and visible light domains.

[0072] Example 2:

[0073] Figures 3 to 5(d) As shown, this invention selected the SSDA-YOLO network and the Efficient Teacher network, and conducted comparative experiments on two public datasets, which fully verified the superiority and generalization of the detection performance of this invention.

[0074] The present invention provides a Transformer-based hybrid attention module. Figure 3 As shown, this model combines channel attention and window-based self-attention mechanisms, leveraging their complementary advantages. To better aggregate cross-window information, an overlapping cross-attention module is introduced to enhance the interaction between features of adjacent windows. Furthermore, a task-based pre-training strategy is employed during the training phase to further explore the model's potential. This extends the Transformer-based hybrid attention module to more image restoration applications, including improving image super-resolution, Gaussian image denoising, and reducing image compression artifacts.

[0075] Overall structure of the MT-YOLOv8 of this invention Figure 4 As shown, the network consists of three parts: a pseudo-image generation network, a semi-supervised network, and an object detector. The pseudo-image generation employs domain transfer; unlike ordinary domain transfer, this invention, for the first time, combines the advantages of image fusion and style transfer to generate pseudo-images with less domain diversity, thus improving detection accuracy. The semi-supervised network used in this invention is a Mean Teacher network. The object detector uses a single-stage detector, YOLOv8, and incorporates a Transformer-based hybrid attention module into the YOLOv8 network, ensuring both detection accuracy and speed.

[0076] To verify the effectiveness of the network proposed in this invention, the performance differences between the MT-YOLOv8 network proposed in this invention and the SSDA-YOLO and Efficient Teacher networks were compared on the LLVIP and KAIST datasets.

[0077] Tables 1 and 2 show a comparison of the detection metrics of the MT-YOLOv8 network proposed in this invention with those of the SSDA-YOLO network and the EfficientTeacher network. As can be seen from the tables, the MT-YOLOv8 network proposed in this invention outperforms the other two networks in P, R, and mAP0.5, especially in P and mAP0.5. Compared with the method without domain adaptation (source domain only), this invention has a significant performance improvement, which fully demonstrates the effectiveness of the MT-YOLOv8 network proposed in this invention in pedestrian detection tasks under low light conditions.

[0078] Table 1 shows the test results of different models on the LLVIP dataset.

[0079]

[0080] Table 2 shows the test results of different models on the KAIST dataset.

[0081]

[0082] Figures 5(a) to 5(d) The test results of four models on different samples are shown. Figures 5(a)-5(d)The results correspond to the sample test results for the source domain only, SSDA-YOLO, Efficient Teacher, and the MT-YOLOv8 network proposed in this invention. Among the samples selected in this invention, the detection performance of the source domain only is unsatisfactory. In the LLVIP dataset, the detection accuracy is low, with severe false negatives. In the KAIST dataset, both false negatives and false positives occur, and pedestrians are not detected. The SSDA-YOLO network performs poorly on the LLVIP dataset. Although it detects most pedestrians, it also suffers from low detection accuracy and false negatives. In the KAIST dataset, only one target is detected, while the other is not, resulting in false negatives. The Efficient Teacher network performs well on the LLVIP dataset, only missing one target, but with high detection accuracy. In the KAIST dataset, it also only detects one target, but its detection accuracy is significantly improved compared to the SSDA-YOLO network. The MT-YOLOv8 network proposed in this invention performs excellently on the LLVIP dataset, not only detecting all targets but also achieving high detection accuracy. On the KAIST dataset, it detects both targets, showing a significant performance improvement compared to source domain only, SSDA-YOLO, and Efficient Teacher.

[0083] The detection results of the four models show that the detection accuracy of the MT-YOLOv8 network proposed in this invention is slightly lower than that of SSDA-YOLO and Efficient Teacher on the KAIST dataset. However, MT-YOLOv8 detected more targets. Only MT-YOLOv8 detected the targets, while the other two networks failed to detect them. In summary, the semi-supervised adaptive pedestrian detection method in low-light environments proposed in this invention can effectively perform pedestrian detection tasks in low-light conditions.

[0084] This invention can be widely used in pedestrian detection in low-light environments.

[0085] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0086] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A semi-supervised adaptive pedestrian detection method in low-light environments, characterized in that, Includes the following steps: S1: A pedestrian detection method for low-light environments is proposed, including the following steps: S11: Establishing a pseudo-image generation network structure: Combine the DIVFusion network (a fusion network for infrared and visible light images without dark areas) and the CycleGAN network (a generative adversarial network for cycle consistency) for pseudo-image generation training; extract the scene illumination unwrapping network SIDNet and the texture contrast enhancement fusion network TCEFNet from the DIVFusion network, combine SIDNet and TCEFNet with the image input part of CycleGAN, perform image fusion processing through SIDNet and TCEFNet before image input, and then input the fused image into CycleGAN for domain transfer to generate pseudo-images; S12: Establish the MT-YOLOv8 network structure: Using the Mean Teacher network as the basic framework, introduce the YOLOv8 single-level detector to build a semi-supervised pedestrian detection network, and introduce a hybrid attention module based on Transformer; S13: Configure a Transformer-based hybrid attention module: combine channel attention and window-based self-attention, introduce an overlapping cross-attention module, and aggregate cross-window information; S14: Setting the supervised loss function and distillation loss function: Based on the Mean Teacher semi-supervised framework, using YOLOv8 as the detector, a student model is used for supervised training on the source domain infrared images. The corresponding source domain supervised loss function is defined as follows: （1） In the formula: Represents the set of source domain images; Represents the set of bounding boxes of the source domain image; Represents the set of category labels for the source domain image; The regression loss for predicting bounding boxes is a combination of distributed focusing loss and complete loss. The binary cross-entropy loss represents the classification loss; When using a student model to supervise the training of pseudo-infrared images in the source domain, the corresponding pseudo-source domain supervised loss function is defined as: （2） In the formula: Represents a set of pseudo-source domain images; Represents the set of bounding boxes for pseudo-source domain images; Represents the set of category labels for pseudo-source domain images; The target domain image is input into the student model and the teacher model respectively for training. The high-confidence bounding box predicted by the teacher model is used as the pseudo-label. The target domain distillation loss function is used to constrain the prediction bias between the two models. The target domain distillation loss function is defined as: （3） In the formula: This represents the target domain image input to the student model; This represents the target domain image input to the teacher model; This represents the bounding box prediction branch from the teacher model; This represents the category prediction branch from the teacher model; and These represent bounding box filters and category filters, respectively. The teacher model is used to train the pseudo-target domain image, and the target domain image is replaced by the pseudo-target domain image for student model training. The labeled image remains unchanged. The pseudo-target domain distillation loss function is defined as follows: （4） In the formula: This represents the pseudo-target domain image input to the teacher model; S2: Evaluate the model using a public dataset, including the following steps: S21: Data cleaning of the KAIST dataset: The nighttime dataset in the KAIST dataset was selected for the experiment. The cleaning rules were as follows: (1) One image was selected from every two images in the training set, and all images that did not contain pedestrians were removed. Images that only showed half of the body or had a pixel count of less than 50 were removed. (2) One image was selected from every 19 images in the test set, and negative sample images were retained. After the above data cleaning process, 2846 training set images and 797 test set images were obtained. S22: Selection of Evaluation Metrics: Evaluation metrics include precision, recall, and average precision, among which: Accuracy is defined as: （5） Recall rate is defined as: （6） The definition of average precision is: （7） In the formula: TP represents the number of positive samples successfully detected by the model; FP represents the number of negative samples that the model detects as positive samples; FN indicates the number of positive samples that the model did not detect; k represents the number of categories; This represents the average accuracy, where i = 0.5; S23: Validate the effectiveness: Compare and validate the MT-YOLOv8 network with SSDA-YOLO and Efficient Teacher on the dataset.

2. The semi-supervised adaptive pedestrian detection method in low-light environments according to claim 1, characterized in that, In step S11, the infrared image is defined as the source domain and the visible light image is defined as the target domain. A pseudo-image generation network is used to perform domain transfer between the two domains to generate pseudo-infrared images and pseudo-visible light images. Correspondingly, the pseudo-infrared image is defined as the pseudo-source domain and the pseudo-visible light image is defined as the pseudo-target domain. Pseudo-cross training is performed using the source domain, target domain, pseudo-source domain, and pseudo-target domain.

3. The semi-supervised adaptive pedestrian detection method in low-light environments according to claim 1, characterized in that, In step S23, SSDA-YOLO and Efficient Teacher networks are selected, and comparative experiments are conducted using the LLVIP and KAIST datasets.

Images