A method for salient target detection in nighttime traffic scenes based on improved YOLOv9

Abstract

This invention discloses a salient target detection method for nighttime traffic scenes based on an improved YOLOv9. It utilizes the visual attention of experienced drivers to acquire a dataset of salient targets in nighttime traffic scenes, then establishes and trains an improved YOLOv9 nighttime salient target detection model. The improvement method includes: constructing a custom RepNCSPELAN4_EMA module by fusing the EMA attention mechanism on the original algorithm and replacing layers 3, 5, 7, and 9 of the RepNCSPELAN4 module in the Backbone; adding a set of detection modules that fuse low-level features; and performing nighttime salient target detection after training. This invention is the first to propose a salient target detection method for nighttime traffic scenes that fuses visual attention. Compared to existing all-target nighttime detection models, it not only focuses on important targets related to driving safety by fusing visual attention features, but also detects targets important to the driving environment more accurately and effectively in various nighttime environments through scale prediction by fusing low-level features.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for detecting salient targets in nighttime traffic scenes based on an improved YOLOv9. Background Technology

[0002] In traffic environments, especially at night, road conditions and the surrounding environment are complex and unpredictable, with obstructed visibility. When a vehicle is in motion, the driver encounters numerous factors that can affect driving safety, such as other vehicles, pedestrians, traffic lights, and signs. To ensure safety, experienced drivers typically focus their attention on the most critical areas, concentrating on salient objects directly related to driving safety. For example, when changing lanes to the right, the driver pays particular attention to the right lane and vehicles ahead, while paying less attention to pedestrians or parked vehicles on the roadside. This allocation of attention helps reduce distractions from irrelevant information, thereby ensuring driving safety.

[0003] From a cognitive science perspective, a driver's visual attention can be divided into two key psychological processes: first, the analysis and calculation of salient factors in the driving environment; and second, the identification of important targets within these salient areas. Advanced human-centered driver assistance systems first understand the visual attention patterns of experienced drivers, and then simulate these patterns to detect important targets within salient areas. Such systems can help drivers identify potential hazards in advance, thereby improving driving safety and comfort, and reducing the computational burden.

[0004] However, to date, few studies have simulated the actual attention and allocation of drivers while driving, and correspondingly, even fewer model frameworks can accurately simulate drivers' gaze behavior and salient target detection in nighttime traffic driving environments. In the rapidly changing nighttime driving environment, accurately and in real-time detecting important targets within salient areas of the traffic driving environment by simulating the psychological and biological characteristics of the human visual attention system provides drivers with crucial information and has significant practical implications for nighttime safety monitoring and target localization in complex environments. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a salient target detection method for nighttime traffic scenes based on an improved YOLOv9. This method can accurately and in real-time detect important targets within salient areas in the traffic driving environment, which has significant practical implications for fields such as driver assistance systems and autonomous driving systems.

[0006] The technical solution adopted in this invention is: a salient target detection method for nighttime traffic scenes based on improved YOLOv9, the specific steps of which are as follows:

[0007] S1. Utilize the visual attention of experienced drivers to obtain a dataset of salient targets in nighttime traffic scenes.

[0008] S2. Establish a nighttime salient target detection model based on the improved YOLOv9;

[0009] S3. Divide the nighttime salient target dataset into a training set, a validation set, and a test set. Use the training set to train the improved YOLOv9 nighttime salient target detection model, and use the validation set to validate the model. After training, perform nighttime salient target detection.

[0010] Furthermore, step S1 is specifically as follows:

[0011] S11. Obtain nighttime traffic scene images containing visual attention information through behavioral experiments;

[0012] The initial eye-tracking database provides data, and the recorded eye-tracking data can simulate visual attention. Each frame of the original nighttime traffic scene video is extracted to obtain the original nighttime traffic scene image. Then, the eye-tracking data obtained through behavioral experiments is mapped one by one onto the corresponding original nighttime traffic scene image. The position of each fixation point corresponds to the spatial position of the original image, resulting in a nighttime traffic scene image containing visual attention information.

[0013] S12. Referencing the nighttime traffic scene image containing visual attention information, perform target annotation on the original nighttime traffic scene image to obtain a nighttime salient target dataset.

[0014] The annotation reference consists of five frames of nighttime traffic scene images containing visual attention information, both before and after the original image of the annotated nighttime traffic scene. The specific annotation principles are as follows:

[0015] (1)F t-5 To F t+5 At least 5 frames in a nighttime traffic scene image containing visual attention information have eye-tracking data points.

[0016] Among them, F t This represents the current detected frame, and t represents the frame number.

[0017] (2) When the target is at F t-5 To F t+5 A frame in a nighttime traffic scene image containing visual attention information contains at least 5 eye-tracking data points.

[0018] When a target meets one of the above two conditions, it is marked, and the nighttime salient target dataset is finally obtained.

[0019] Furthermore, step S2 is specifically as follows:

[0020] S21. Based on the YOLOv9 model, a custom RepNCSPELAN4_EMA module is constructed by integrating the EMA attention mechanism and replacing the RepNCSPELAN4 modules in layers 3, 5, 7, and 9 of the Backbone.

[0021] The YOLOv9 RepNCSPELAN4 module includes two RepNCSP+Conv submodules. The EMA attention mechanism is then integrated after RepNCSP to form the RepNCSP+EMA+Conv submodule, resulting in the custom RepNCSPELAN4_EMA module. Finally, the RepNCSPELAN4_EMA module replaces the RepNCSPELAN4 modules in layers 3, 5, 7, and 9 of the Backbone.

[0022] S22. Based on step S21, add another set of detection modules that fuse low-level features;

[0023] A detection module that integrates low-level features is added to perform target detection by integrating features from different stages. High-level features at different scales in four stages are fused with low-level features to simulate visual perception, and object recognition is performed through information processing via two pathways.

[0024] The specific sizes of the four types of features fused at different stages include: features at a scale of 13×13; features at a scale of 26×26 fused with features at a scale of 26×26 and features at a scale of 13×13 that have been upsampled; features at a scale of 52×52 fused with features at a scale of 52×52 and features at a scale of 26×26 that have been upsampled; and features at a scale of 104×104 fused with features at a scale of 104×104 and features at a scale of 52×52 that have been upsampled.

[0025] Furthermore, step S3 is specifically as follows:

[0026] S31. Divide the nighttime salient target dataset into training set, validation set, and test set in a ratio of 7:1:2, and resize the images in the dataset to 416×416×3.

[0027] S32. First, randomly initialize the parameters of the nighttime salient target detection model, input the training set images into the model, and use binary cross-entropy to calculate the loss value between the detected salient targets and their corresponding labels; use an initial learning rate of 10. -2 And set the minimum learning rate to 10. -4 The momentum value is 0.937, and the decay rate is 5. -4 The SGD optimizer is trained, and the model parameters are saved and the optimal model parameters are updated every 10 epochs.

[0028] S33. After each training epoch, the model is validated using a validation set. Steps S31-S33 are repeated iteratively until the calculated mAP and loss values ​​converge, indicating that the parameters in the network are basically stable and the optimal model parameters are obtained. Finally, the optimal model parameters are used to detect nighttime traffic scene datasets and obtain detection results.

[0029] The beneficial effects of this invention are as follows: The method of this invention utilizes the visual attention of experienced drivers to acquire a dataset of salient targets in nighttime traffic scenes. Then, it establishes a nighttime salient target detection model that integrates visual attention and trains and optimizes it. After training, it performs nighttime salient target detection. This invention is the first to propose a salient target detection method for nighttime traffic scenes that integrates visual attention. Compared to existing all-target nighttime detection models, it not only focuses on important targets related to driving safety by integrating visual attention features, but also, through scale prediction that integrates low-level features, more accurately and effectively detects targets important to the driving environment in various nighttime environments. Attached Figure Description

[0030] Figure 1 This is a flowchart of a salient target detection method for nighttime traffic scenes based on an improved YOLOv9 according to the present invention.

[0031] Figure 2 Reference F is used in the embodiments of the present invention. t-5 To F t+5 Frames containing visual attention information of nighttime traffic scenes are used to label the Fth frame. t A schematic diagram of the original image of a nighttime traffic scene.

[0032] Figure 3 This is a diagram showing the overall structure of the nighttime salient target detection model in an embodiment of the present invention.

[0033] Figure 4 This is a result image of nighttime salient target detection performed on a nighttime traffic scene map in an embodiment of the present invention. Detailed Implementation

[0034] The method of the present invention will be further explained below with reference to the accompanying drawings and embodiments.

[0035] like Figure 1 The flowchart of a salient target detection method for nighttime traffic scenes based on improved YOLOv9 is shown below. The specific steps are as follows:

[0036] S1. Utilize the visual attention of experienced drivers to obtain a dataset of salient targets in nighttime traffic scenes.

[0037] S2. Establish a nighttime salient target detection model based on the improved YOLOv9;

[0038] S3. Divide the nighttime salient target dataset into a training set, a validation set, and a test set. Use the training set to train the improved YOLOv9 nighttime salient target detection model, and use the validation set to validate the model. After training, perform nighttime salient target detection.

[0039] In this embodiment, step S1 is specifically as follows:

[0040] S11. Obtain nighttime traffic scene images containing visual attention information through behavioral experiments;

[0041] For research on visually salient targets in nighttime traffic scenes, integrating visual saliency mechanisms is crucial. Eye-tracking data obtained through behavioral experiments can vividly and accurately display visual attention information, which can be perfectly integrated with subsequent nighttime salient target detection. An initial eye-tracking database is provided, containing multiple traffic videos of normal driving in nighttime scenes saved by dashcams. In this embodiment, the traffic videos are 1280×720 pixels in size and last from 150 to 181 seconds. A total of 28 participants were involved in the behavioral experiment, with a male-to-female ratio of 4:3. To ensure the accuracy of the experiment, all participants had at least two years of driving experience and were between 23 and 45 years old. Because the experimental process requires the subjects to watch the video in a completely simulated real driving scenario, the recorded eye movement data can simulate visual attention. In this embodiment, each frame of the existing 16 original night traffic scene videos is extracted to obtain the original night traffic scene image. Then, the eye movement data obtained through the behavioral experiment is mapped one by one onto the corresponding original night traffic scene image. The position of each fixation point corresponds to the spatial position of the original image, resulting in a night traffic scene image containing visual attention information.

[0042] S12. Referencing the nighttime traffic scene image containing visual attention information, perform target annotation on the original nighttime traffic scene image to obtain a nighttime salient target dataset.

[0043] Considering the time cost of labeling and the similarity between adjacent frames, and given that research indicates the latency of attention shift is approximately 200-300ms, the labeling reference is five frames of nighttime traffic scene images containing visual attention information before and after the original labeled nighttime traffic scene image, such as... Figure 2 As shown, to ensure that targets important to the driving environment are not omitted, the specific labeling principles are as follows:

[0044] (1)F t-5 To F t+5At least 5 frames in a nighttime traffic scene image containing visual attention information have eye-tracking data points, because when a driver spends a lot of time focusing on a target, it is reasonable to infer that the target is a relevant and important target in the driving environment.

[0045] Among them, F t This represents the current detected frame, and t represents the frame number.

[0046] (2) When the target is at F t-5 To F t+5 A frame in a nighttime traffic scene image containing visual attention information contains at least 5 eye-tracking data points. When a target attracts the attention of multiple drivers at the same time, although the target may be noticed for a very short time, it is likely to be a suddenly appearing target, such as overtaking. This is obviously an important target related to the driving environment.

[0047] When a target meets one of the above two conditions, it is marked, and the nighttime salient target dataset is finally obtained.

[0048] The structure of the improved YOLOv9 nighttime salient target detection model is as follows: Figure 3 As shown.

[0049] In this embodiment, step S2 is specifically as follows:

[0050] S21. Based on the YOLOv9 model, a custom RepNCSPELAN4_EMA module is constructed by integrating the EMA attention mechanism and replacing the RepNCSPELAN4 modules in layers 3, 5, 7, and 9 of the Backbone (used for extracting attention features);

[0051] In this embodiment, the nighttime salient target detection model (improved YOLOv9 nighttime salient target detection model) that integrates visual attention uses the EMA attention mechanism, a novel and efficient multi-scale attention module that demonstrates significant effectiveness in computer vision tasks. The custom RepNCSPELAN4_EMA module, which integrates the EMA attention mechanism, combines global information and local features of the image through global pooling and cross-spatial attention mechanisms, thereby improving the perception capability of nighttime scenes.

[0052] YOLOv9's RepNCSPELAN4 module includes two RepNCSP+Conv sub-modules. The EMA attention mechanism is fused after RepNCSP to form the RepNCSP+EMA+Conv sub-module, resulting in the custom RepNCSPELAN4_EMA module. Finally, replacing layers 3, 5, 7, and 9 of the RepNCSPELAN4 module in the backbone with the RepNCSPELAN4_EMA module improves the model's ability to extract attention features. Through global pooling and cross-spatial attention mechanisms, the RepNCSPELAN4_EMA module can accurately locate targets like a spotlight, enhancing visual perception in nighttime scenes.

[0053] S22. Based on step S21, add another set of detection modules that fuse low-level features;

[0054] In this embodiment, the nighttime salient target detection model incorporating visual attention adds a detection module that integrates low-level features to capture edge and texture features in the image. Edge features, as an intuitive expression of target boundaries—a direct mapping of target contours—play a crucial role in accurately locating and identifying the contours and shapes of targets in nighttime environments. They are essential for the accurate identification of target positions and shapes in nighttime conditions, providing clear definition of the target and enabling effective identification of target positions and shapes even in low-light conditions. Simultaneously, the integration of texture features helps distinguish targets from the background in nighttime scenes, further enhancing the model's discriminative ability. It can identify subtle texture differences between different targets, which is particularly important for distinguishing similar targets, thus significantly improving the model's ability to detect salient targets in nighttime scenes. This comprehensive utilization of texture and edge features greatly improves the model's accuracy in detecting salient targets in nighttime environments, enabling it to maintain high accuracy and robustness in varied nighttime scenes.

[0055] By fusing features from different stages for object detection, high-level features at different scales across four stages are fused with low-level features to simulate visual perception. Object recognition is achieved through information processing via two pathways, providing more accurate location and boundary information. Furthermore, it can simultaneously utilize detailed and semantic information, thereby improving the accuracy of object detection.

[0056] The specific sizes of the four types of feature fusion at different stages include: 13×13 scale features; 26×26 scale features fused with 13×13 scale features after upsampling; 52×52 scale features fused with 26×26 scale features after upsampling; and 104×104 scale features fused with 52×52 scale features after upsampling. Larger scale features are more accurate at detecting small targets, while smaller scale features are better at detecting large targets.

[0057] In this embodiment, step S3 is specifically as follows:

[0058] S31. Divide the nighttime salient target dataset into training set, validation set, and test set in a ratio of 7:1:2. In order to shorten the training time, resize the images in the dataset to 416×416×3 (height H×width W×number of channels C).

[0059] S32. First, randomly initialize the parameters of the nighttime salient target detection model, input the training set images into the model, and use binary cross-entropy to calculate the loss value between the detected salient targets and their corresponding labels (targets within the standard salient area); use an initial learning rate of 10. -2 And set the minimum learning rate to 10. -4 The momentum value is 0.937, and the decay rate is 5. -4 The SGD optimizer is trained, and the model parameters are saved and the optimal model parameters are updated every 10 epochs.

[0060] S33. After each training epoch, the model is validated using a validation set. Steps S31-S33 are repeated iteratively until the calculated mAP and loss values ​​converge, indicating that the parameters in the network are basically stable and the optimal model parameters are obtained. Finally, the optimal model parameters are used to detect nighttime traffic scene datasets and obtain detection results.

[0061] In this embodiment, the method of the present invention further includes step S4, which tests and evaluates the performance of the nighttime salient target detection model that incorporates visual attention, as follows:

[0062] Import the optimal model parameters obtained in step S33, and input random video frames from the test set to test the performance of the nighttime salient object detection model that incorporates visual attention.

[0063] The test results were analyzed qualitatively and quantitatively. Qualitative analysis included... Figure 4 As shown. Figure 4The image shows the results of nighttime salient target detection on a nighttime traffic scene map. Because the method of this invention not only integrates a visual attention mechanism but also incorporates more low-level features, it can more accurately and effectively detect targets important to the driving environment, especially in scenarios involving small-scale objects and sudden overtaking maneuvers. This demonstrates that the visual attention-integrated nighttime salient target detection model of this invention can effectively simulate real-world driving scenarios and perform nighttime salient target detection, proving the reliability and effectiveness of the method.

[0064] Table 1 shows the comparison results of the evaluation indicators of the method of the present invention with other existing methods. This is a quantitative analysis, and the main evaluation indicators used include: mAP value, car AP value, traffic signs AP value, Person value, motorcycle value, and traffic-light value. The comprehensive quantitative indicator analysis also demonstrates the superior performance of the method of the present invention.

[0065] Table 1

[0066] Method mAP Car Traffic signs Person Motorcycle Traffic light YOLOv9 89.2 94.3 85.6 87 91.8 87.2 Method of the present invention 92.6 95 90.7 90.6 92.3 94.2

[0067] In summary, the method of this invention is applied to the field of computer vision technology. By using salient videos of experienced drivers' visual attention, a nighttime salient target detection dataset is constructed. Then, an improved YOLOv9 nighttime salient target detection model is established and trained and optimized. Compared with existing all-target nighttime detection models, the nighttime salient target detection model that integrates visual attention not only focuses on important targets related to driving safety by integrating visual attention features, but also detects targets important to the driving environment more accurately and effectively in various nighttime environments by integrating scale prediction of low-level features.

[0068] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A salient target detection method for nighttime traffic scenes based on improved YOLOv9, the specific steps of which are as follows: S1. Utilize the visual attention of experienced drivers to obtain a dataset of salient targets in nighttime traffic scenes. S2. Establish a nighttime salient target detection model based on the improved YOLOv9; Step S2 is as follows: S21. Based on the YOLOv9 model, a custom RepNCSPELAN4_EMA module is constructed by integrating the efficient multi-scale attention (EMA) mechanism and replacing the RepNCSPELAN4 modules in layers 3, 5, 7, and 9 of the Backbone. The YOLOv9 RepNCSPELAN4 module includes two RepNCSP+Conv sub-modules. The EMA attention mechanism is fused into RepNCSP to form the RepNCSP+EMA+Conv sub-module, resulting in the custom RepNCSPELAN4_EMA module. Finally, the RepNCSPELAN4 modules in layers 3, 5, 7, and 9 of the Backbone are replaced with the RepNCSPELAN4_EMA module. S22. Based on step S21, add another set of detection modules that fuse low-level features; Add a detection module that integrates low-level features. Target detection is performed by integrating features from different stages. High-level features at different scales in four stages are fused with low-level features to simulate visual perception. Object recognition is performed through information processing via two pathways. in, The four specific sizes for fusing features from different stages include: 13×13 scale features; 26×26 scale features fused with 13×13 scale features after upsampling; 52×52 scale features fused with 26×26 scale features after upsampling; and 104×104 scale features fused with 52×52 scale features after upsampling. S3. Divide the nighttime salient target dataset into a training set, a validation set, and a test set. Use the training set to train the improved YOLOv9 nighttime salient target detection model, and use the validation set to validate the model. After training, perform nighttime salient target detection.

2. The salient target detection method for nighttime traffic scenes based on improved YOLOv9 according to claim 1, characterized in that, The specific steps of S1 are as follows: S11. Obtain nighttime traffic scene images containing visual attention information through behavioral experiments; The initial eye-tracking database provides data, and the recorded eye-tracking data can simulate visual attention. Each frame of the original nighttime traffic scene video is extracted to obtain the original nighttime traffic scene image. Then, the eye-tracking data obtained through behavioral experiments is mapped one by one onto the corresponding original nighttime traffic scene image. The position of each fixation point corresponds to the spatial position of the original image, resulting in a nighttime traffic scene image containing visual attention information. S12. Referencing the nighttime traffic scene image containing visual attention information, perform target annotation on the original nighttime traffic scene image to obtain a nighttime salient target dataset. The annotation reference consists of five frames of nighttime traffic scene images containing visual attention information, both before and after the original image of the annotated nighttime traffic scene. The specific annotation principles are as follows: (1)F t-5 To F t+5 At least 5 frames in a nighttime traffic scene image containing visual attention information have eye-tracking data points. Among them, F t This represents the currently detected frame, where t represents the frame number; (2) When the target is at F t-5 To F t+5 A frame in a nighttime traffic scene image containing visual attention information must contain at least 5 eye-tracking data points. When a target meets one of the above two conditions, it is marked, and the nighttime salient target dataset is finally obtained.

3. The salient target detection method for nighttime traffic scenes based on improved YOLOv9 according to claim 1, characterized in that, The specific details of step S3 are as follows: S31. Divide the nighttime salient target dataset into training set, validation set, and test set in a ratio of 7:1:2, and resize the images in the dataset to 416×416×3. S32. First, randomly initialize the parameters of the nighttime salient target detection model, input the training set images into the model, and use binary cross-entropy to calculate the loss value between the detected salient targets and their corresponding labels. The initial learning rate is 10. -2 And set the minimum learning rate to 10. -4 The momentum value is 0.937, and the decay rate is 5. -4 The SGD optimizer is trained, and the model parameters are saved and the optimal model parameters are updated every 10 epochs. S33. After each training epoch, the model is validated using a validation set. Iterative training is performed continuously until the calculated mAP and loss values ​​converge, meaning the parameters in the network are basically stable, and the optimal model parameters are obtained. Finally, the optimal model parameters are used to detect nighttime traffic scene datasets and obtain detection results.

Images