Night traffic scene salient target detection method based on improved YOLOv9

Abstract

The invention discloses a night traffic scene salient target detection method based on improved YOLOv9, and the method comprises the steps: obtaining a night salient target data set of a night traffic scene through the visual attention of an experienced driver, building a night salient target detection model of the improved YOLOv9, and carrying out the training optimization. The improved method comprises the following steps: on the basis of an original algorithm, fusing an EMA attention mechanism to construct a user-defined RepNCSPELAN4 EMA module, replacing a third layer RepNCSPELAN4 module, a fifth layer RepNCSPELAN4 module, a seventh layer RepNCSPELAN4 module, a ninth layer RepNCSPELAN4 module and a ninth layer RepNCSPELAN4 module in Backbone, adding a group of detection modules fusing low-level features, and performing night salient target detection after training is completed. According to the method, visual attention fused night traffic scene salient target detection is put forward for the first time, and compared with an existing full-target night detection model, only important targets related to driving safety are concerned through fusion of visual attention features, and scale prediction is carried out through fusion of low-level features, so that the driving safety is greatly improved in various night environments. And targets important to the driving environment can be detected more accurately and effectively.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for detecting salient targets in night traffic scenes based on improved YOLOv9. Background Art

[0002] In traffic environments, especially at night, the road and its surroundings are complex and changeable, and the line of sight is obstructed. When the vehicle is driving, the driver will be exposed to many factors that may affect driving safety, such as other vehicles, pedestrians, traffic lights and signs. In order to ensure safety, experienced drivers usually focus on the most critical areas and focus on those significant targets that are directly related to driving safety. For example, when making a right lane change, the driver will pay special attention to the right lane and the vehicle in front, but pay less attention to pedestrians or parked vehicles on the roadside. This distribution of attention helps to reduce the interference of irrelevant information, thereby ensuring driving safety.

[0003] From the perspective of cognitive science, the driver's visual attention can be divided into two key psychological processes: first, the analysis and calculation of salient factors in the driving environment, and second, the identification of important targets within these salient areas. Advanced human-centric driver assistance systems must first understand the visual attention patterns of experienced drivers and then simulate this pattern to detect important targets within salient areas. Such systems can help drivers identify potential dangers in advance, thereby improving driving safety and comfort and reducing the computational burden.

[0004] However, so far, few studies have simulated the real attention and allocation of drivers while driving, and accordingly, there are fewer model frameworks that can accurately simulate the driver's gaze behavior and salient target detection in night traffic driving environments. In the ever-changing night driving environment, by simulating the psychological and biological characteristics of the human visual attention system, important targets in the salient area of ​​the traffic driving environment can be accurately and real-time detected to provide key information to the driver, which has important practical significance for night safety monitoring, target positioning in complex environments, and other issues. Summary of the invention

[0005] To solve the above problems, the present invention proposes a salient target detection method for night traffic scenes based on improved YOLOv9, which can accurately and real-time detect important targets in the salient area of ​​the traffic driving environment, and has important practical significance for the fields of assisted driving systems and automatic driving systems.

[0006] The technical solution adopted by the present invention is: a method for detecting salient targets in night traffic scenes based on improved YOLOv9, and the specific steps are as follows:

[0007] S1. Using the visual attention of experienced drivers to obtain a night-time salient target dataset of night-time traffic scenes;

[0008] S2. Establish a night salient target detection model based on improved YOLOv9;

[0009] S3. Divide the night salient target data set into a training set, a validation set, and a test set. Use the training set to train the improved YOLOv9 night salient target detection model, use the validation set to verify the model, and perform night salient target detection after training.

[0010] Furthermore, the step S1 is specifically as follows:

[0011] S11, obtain night traffic scene images containing visual attention information through behavioral experiments;

[0012] The initial eye movement database provides data. The recorded eye movement data can simulate visual attention. Each frame of the original night traffic scene video is extracted to obtain the original night traffic scene image. Then the eye movement data obtained through behavioral experiments are mapped one by one to the corresponding original night traffic scene image. The position of each gaze point corresponds to the spatial position of the original image, and a night traffic scene image containing visual attention information is obtained.

[0013] S12, referring to the night traffic scene image containing visual attention information, annotating the target of the original night traffic scene image to obtain a night salient target dataset;

[0014] The annotation reference is the night traffic scene image containing visual attention information 5 frames before and after the annotated night traffic scene original image. The annotation principles are as follows:

[0015] (1)F t-5 to F t+5 In the night traffic scene image, at least 5 frames contain eye movement data points;

[0016] Among them, F t represents the current detection frame, and t represents the frame number.

[0017] (2) When the target is in F t-5 to F t+5 There are at least 5 eye movement data points in one frame of the night traffic scene image containing visual attention information.

[0018] When the target meets one of the above two conditions, it is marked, and finally a night salient target dataset is obtained.

[0019] Furthermore, the step S2 is specifically as follows:

[0020] S21. Based on the YOLOv9 model, the EMA attention mechanism is integrated to build a custom RepNCSPELAN4_EMA module and replace the 3rd, 5th, 7th, and 9th layer RepNCSPELAN4 modules in Backbone;

[0021] The RepNCSPELAN4 module of YOLOv9 includes two RepNCSP+Conv submodules. The EMA attention mechanism is fused behind RepNCSP to form RepNCSP+EMA+Conv submodules, and a custom RepNCSPELAN4_EMA module is obtained. Finally, the RepNCSPELAN4_EMA module replaces the 3rd, 5th, 7th, and 9th layer RepNCSPELAN4 modules in Backbone.

[0022] S22, based on step S21, adding another set of detection modules integrating low-level features;

[0023] A set of detection modules that fuse low-level features is added to perform target detection by fusing features at different stages. High-level features of four stages and different scales are fused with low-level features to simulate visual perception, and object recognition is performed through information processing of two pathways.

[0024] Among them, the specific sizes of the four types of features fused at different stages include: features of 13×13 scale; features of 26×26 scale after fusion of features of 26×26 scale and features upsampled from 13×13 scale; features of 52×52 scale after fusion of features of 52×52 scale and features upsampled from 26×26 scale; features of 104×104 scale after fusion of features of 104×104 scale and features upsampled from 52×52 scale.

[0025] Furthermore, the step S3 is as follows:

[0026] S31. Divide the night salient target dataset into a training set, a validation set, and a test set in a ratio of 7:1:2, and reshape the images in the dataset to 416×416×3.

[0027] S32, first randomly initialize the parameters of the night salient target detection model, input the training set images into the model, and use binary cross entropy to calculate the loss value between the detected salient targets and the corresponding labels; use the initialization learning rate of 10 -2 And set the minimum learning rate to 10 -4 , the momentum value is 0.937, and the decay rate is 5 -4 The SGD optimizer is used for training, and the model parameters are saved and the optimal model parameters are updated every 10 epochs;

[0028] S33. After each epoch of training, the model is verified with the validation set, and steps S31-S33 are repeated for iterative training until the calculated mAP and loss values ​​converge, that is, the parameters in the network are basically stable, and the optimal model parameters are obtained. Finally, the optimal model parameters are used to detect the night traffic scene dataset to obtain the detection results.

[0029] Beneficial effects of the present invention: The method of the present invention uses the visual attention of experienced drivers to obtain a night-time salient target data set of night traffic scenes, then establishes a night-time salient target detection model integrating visual attention and performs training optimization, and performs night-time salient target detection after the training is completed. The method of the present invention proposes for the first time salient target detection of night traffic scenes integrating visual attention. Compared with the existing full-target night detection model, it not only focuses on important targets related to driving safety by integrating visual attention features, but also detects targets important to the driving environment more accurately and effectively in a variety of night environments by integrating scale prediction of low-level features. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a flowchart of a method for detecting salient targets in night traffic scenes based on improved YOLOv9.

[0031] Figure 2 Reference F is used in the embodiment of the present invention t-5 to F t+5 The night traffic scene image containing visual attention information is used to mark the F t Schematic diagram of the original image of a night traffic scene with a frame.

[0032] Figure 3 2 is an overall structural diagram of a night salient target detection model in an embodiment of the present invention.

[0033] Figure 4 This is a result diagram of night-time salient target detection on a night-time traffic scene diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The method of the present invention is further explained below in conjunction with the accompanying drawings and embodiments.

[0035] like Figure 1 As shown, a flow chart of a method for detecting salient targets in night traffic scenes based on improved YOLOv9 of the present invention is shown, and the specific steps are as follows:

[0036] S1. Using the visual attention of experienced drivers to obtain a night-time salient target dataset of night-time traffic scenes;

[0037] S2. Establish a night salient target detection model based on improved YOLOv9;

[0038] S3. Divide the night salient target data set into a training set, a validation set, and a test set. Use the training set to train the improved YOLOv9 night salient target detection model, use the validation set to verify the model, and perform night salient target detection after training.

[0039] In this embodiment, the step S1 is specifically as follows:

[0040] S11, obtain night traffic scene images containing visual attention information through behavioral experiments;

[0041] For the study of visually salient targets in night traffic scenes, it is very important to integrate the visual saliency mechanism. The eye movement data obtained through behavioral experiments can show visual attention information very vividly and accurately, which can be perfectly connected with the subsequent detection of salient targets at night. The initial eye movement database provides data. The initial eye movement database contains a number of traffic videos of normal driving in night scenes saved by driving recorders. The size of the traffic video provided in this embodiment is 1280×720, and the duration of the traffic video lasts from 150 seconds to 181 seconds. A total of 28 subjects participated in the behavioral experiment, with a male-female ratio of 4:3. To ensure the accuracy of the experiment, all subjects had at least two years of driving experience and were between 23 and 45 years old. Because the experimental process requires the subjects to watch the video in a completely simulated real driving scene, the recorded eye movement data can simulate visual attention. In this embodiment, each frame of the 16 existing original night traffic scene videos is extracted to obtain the original night traffic scene image, and then the eye movement data obtained through the behavioral experiment are mapped one by one to the corresponding original night traffic scene image. The position of each gaze point corresponds to the spatial position of the original image, and a night traffic scene image containing visual attention information is obtained.

[0042] S12, referring to the night traffic scene image containing visual attention information, annotating the target of the original night traffic scene image to obtain a night salient target dataset;

[0043] Among them, considering the labeling time cost and the similarity between adjacent frames, and studies have shown that the latency of attention transfer is about 200-300ms, the annotation reference is the night traffic scene image containing visual attention information 5 frames before and after the annotated night traffic scene original image, such as Figure 2 As shown, to ensure that the targets important to the driving environment are not missed, the marking principles are as follows:

[0044] (1)F t-5 to F t+5At least 5 frames of the night traffic scene image contain eye movement data points, because when the driver spends a lot of time focusing on a target, it can be reasonably inferred that the target is a relevant and important target in the driving environment;

[0045] Among them, F t represents the current detection frame, and t represents the frame number.

[0046] (2) When the target is in F t-5 to F t+5 There are at least 5 eye movement data points in one frame of the night traffic scene image containing visual attention information. When a target attracts the attention of multiple drivers at the same time, although the target may be noticed for a short time, it is likely to be a sudden target, such as overtaking, which is obviously an important target related to the driving environment.

[0047] When the target meets one of the above two conditions, it is marked, and finally a night salient target dataset is obtained.

[0048] The structure of the improved YOLOv9 night salient target detection model is as follows Figure 3 shown.

[0049] In this embodiment, step S2 is specifically as follows:

[0050] S21. Based on the YOLOv9 model, the EMA attention mechanism is integrated to build a custom RepNCSPELAN4_EMA module and replace the 3rd, 5th, 7th, and 9th layer RepNCSPELAN4 modules in Backbone (used to extract attention features);

[0051] In this embodiment, in the night salient target detection model that integrates visual attention (improved YOLOv9 night salient target detection model), the EMA attention mechanism is a new type of efficient multi-scale attention module, which shows remarkable effectiveness in computer vision tasks. The EMA attention mechanism is integrated to build a custom RepNCSPELAN4_EMA module through global pooling and cross-space attention mechanism, which combines the global information and local features of the image to improve the perception ability of night scenes.

[0052] The RepNCSPELAN4 module of YOLOv9 includes two RepNCSP+Conv submodules. The EMA attention mechanism is integrated behind RepNCSP to form RepNCSP+EMA+Conv submodules, and a custom RepNCSPELAN4_EMA module is obtained. Finally, the RepNCSPELAN4_EMA module replaces the 3rd, 5th, 7th, and 9th layers of the RepNCSPELAN4 module in Backbone, which helps the model improve its ability to extract attention features. Through global pooling and cross-spatial attention mechanisms, the RepNCSPELAN4_EMA module can accurately locate the position of the target like a spotlight, improving the visual perception of night scenes.

[0053] S22, based on step S21, adding another set of detection modules integrating low-level features;

[0054] In this embodiment, in the night salient target detection model that integrates visual attention, adding a set of detection modules that integrate low-level features can capture edge and texture feature information in the image. Edge features, as an intuitive expression of the target boundary, that is, a direct mapping of the target contour, play a key role in accurately locating and identifying the contour and shape of the target in the night environment. They are crucial for accurately identifying the position and shape of the target in the night environment. They provide a clear definition of the target, so that the position and shape of the target can be effectively identified even under low light conditions. At the same time, the integration of texture features helps to distinguish between targets and backgrounds in night scenes, further enhancing the model's ability to distinguish, and can identify subtle texture differences between different targets, which is particularly important for distinguishing similar targets, thereby significantly improving the model's ability to detect salient targets in night scenes. This comprehensive use of texture and edge features greatly improves the model's detection accuracy for salient targets in night environments, enabling it to maintain high accuracy and robustness in changing night scenes.

[0055] Target detection is performed by fusing features from different stages, and high-level features of different scales in four stages are fused with low-level features to simulate visual perception. Object recognition is performed through information processing from two pathways, which can provide more accurate position and boundary information, and can utilize both detail information and semantic information at the same time, thereby improving the accuracy of target detection.

[0056] Among them, the specific sizes of the four types of features fused at different stages include: 13×13 scale features; 26×26 scale features fused with 13×13 scale upsampled features; 52×52 scale features fused with 26×26 scale upsampled features; 104×104 scale features fused with 52×52 scale upsampled features. Features with larger scales are more accurate in detecting small targets, and features with smaller scales are easier to detect large targets.

[0057] In this embodiment, the step S3 is as follows:

[0058] S31. Divide the night salient target dataset into a training set, a validation set, and a test set in a ratio of 7:1:2. In order to shorten the training time, reshape the size of the images in the dataset to 416×416×3 (height H×width W×number of channels C).

[0059] S32, first randomly initialize the parameters of the night salient target detection model, input the training set images into the model, and use binary cross entropy to calculate the loss value between the detected salient targets and the corresponding labels (targets in the standard salient area); use the initialization learning rate of 10 -2 And set the minimum learning rate to 10 -4 , the momentum value is 0.937, and the decay rate is 5 -4 The SGD optimizer is used for training, and the model parameters are saved and the optimal model parameters are updated every 10 epochs;

[0060] S33. After each epoch of training, the model is verified with the validation set, and steps S31-S33 are repeated for iterative training until the calculated mAP and loss values ​​converge, that is, the parameters in the network are basically stable, and the optimal model parameters are obtained. Finally, the optimal model parameters are used to detect the night traffic scene dataset to obtain the detection results.

[0061] In this embodiment, the method of the present invention further includes step S4, testing and evaluating the effect of the night salient target detection model integrating visual attention, as follows:

[0062] The optimal model parameters obtained in step S33 are imported, and random video frames in the test set are input to test the performance of the night salient target detection model integrating visual attention.

[0063] Conduct qualitative and quantitative analysis of the test results. Figure 4 shown. Figure 4This is the result of night salient target detection for night traffic scene graph. Because the method of the present invention not only integrates the visual attention mechanism, but also integrates more low-level features, it can more accurately and effectively detect targets important to the driving environment in small-scale object detection and sudden overtaking scenes. This shows that the night salient target detection model integrating visual attention of the present invention can well simulate the real driving scene and perform night salient target detection, and the method of the present invention is reliable and effective.

[0064] Table 1 is a comparison of the evaluation indicators of the method of the present invention and other existing methods. It is the result of quantitative analysis. The main evaluation indicators used include: mAP value, car AP value, traffic signs AP value, Person value, motorcycle value, and traffic-light value. Comprehensive quantitative indicator analysis also shows that the method of the present invention has superior performance.

[0065] Table 1

[0066] Method mAP Car Traffic signs Person Motorcycle Traffic light YOLOv9 89.2 94.3 85.6 87 91.8 87.2 Method of the present invention 92.6 95 90.7 90.6 92.3 94.2

[0067] In summary, the method of the present invention is applied to the field of computer vision technology. A night salient target detection data set is constructed through salient videos of visual attention of experienced drivers, and then an improved night salient target detection model of YOLOv9 is established and trained and optimized. The night salient target detection model that integrates visual attention, compared with the existing full-target night detection model, not only focuses on important targets related to driving safety by integrating visual attention features, but also detects targets that are important to the driving environment more accurately and effectively in a variety of night environments by integrating scale predictions of low-level features.

[0068] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting salient targets in night traffic scenes based on improved YOLOv9, the specific steps are as follows: S1. Using the visual attention of experienced drivers to obtain a night-time salient target dataset of night-time traffic scenes; S2. Establish a night salient target detection model based on improved YOLOv9; S3. Divide the night salient target data set into a training set, a validation set, and a test set. Use the training set to train the improved YOLOv9 night salient target detection model, use the validation set to verify the model, and perform night salient target detection after training.

2. According to claim 1, a method for detecting salient targets in night traffic scenes based on improved YOLOv9 is characterized in that: The step S1 is specifically as follows: S11, obtain night traffic scene images containing visual attention information through behavioral experiments; The initial eye movement database provides data. The recorded eye movement data can simulate visual attention. Each frame of the original night traffic scene video is extracted to obtain the original night traffic scene image. Then, the eye movement data obtained through behavioral experiments are mapped one by one to the corresponding original night traffic scene image. The position of each fixation point corresponds to the spatial position of the original image, and a night traffic scene image containing visual attention information is obtained. S12, referring to the night traffic scene image containing visual attention information, annotating the target of the original night traffic scene image to obtain a night salient target dataset; The annotation reference is the night traffic scene image containing visual attention information 5 frames before and after the annotated night traffic scene original image. The annotation principles are as follows: (1)F t-5 to F t+5 In the night traffic scene image, at least 5 frames contain eye movement data points; Among them, F t represents the current detection frame, and t represents the frame number; (2) When the target is in F t-5 to F t+5 There are at least 5 eye movement data points in one frame of the night traffic scene image containing visual attention information; When the target meets one of the above two conditions, it is marked, and finally a night salient target dataset is obtained.

3. The method for detecting salient targets in night traffic scenes based on improved YOLOv9 according to claim 1, characterized in that: The step S2 is specifically as follows: S21. Based on the YOLOv9 model, the EMA attention mechanism is integrated to build a custom RepNCSPELAN4_EMA module and replace the 3rd, 5th, 7th, and 9th layer RepNCSPELAN4 modules in Backbone; The RepNCSPELAN4 module of YOLOv9 includes two RepNCSP+Conv submodules. The EMA attention mechanism is fused behind RepNCSP to form RepNCSP+EMA+Conv submodules, and a custom RepNCSPELAN4_EMA module is obtained. Finally, the RepNCSPELAN4_EMA module replaces the 3rd, 5th, 7th, and 9th layer RepNCSPELAN4 modules in Backbone. S22, based on step S21, adding another set of detection modules integrating low-level features; Add a set of detection modules that fuse low-level features, perform target detection by fusing features at different stages, fuse high-level features of four stages and different scales with low-level features to simulate visual perception, and perform object recognition through information processing of two pathways; Among them, the specific sizes of the four types of features fused at different stages include: features of 13×13 scale; features of 26×26 scale after fusion of features of 26×26 scale and features upsampled from 13×13 scale; features of 52×52 scale after fusion of features of 52×52 scale and features upsampled from 26×26 scale; features of 104×104 scale after fusion of features of 104×104 scale and features upsampled from 52×52 scale.

4. The method for detecting salient targets in night traffic scenes based on improved YOLOv9 according to claim 1, characterized in that: The specific details of step S3 are as follows: S31. Divide the night salient target dataset into a training set, a validation set, and a test set in a ratio of 7:1:2, and reshape the images in the dataset to 416×416×3. S32, first randomly initialize the parameters of the night salient target detection model, input the training set images into the model, and use binary cross entropy to calculate the loss value between the detected salient targets and the corresponding labels; The initial learning rate is 10. -2 And set the minimum learning rate to 10 -4 , the momentum value is 0.937, and the decay rate is 5 -4 The SGD optimizer is used for training, and the model parameters are saved and the optimal model parameters are updated every 10 epochs; S33. After each epoch of training, the model is verified with the validation set, and steps S31-S33 are repeated for iterative training until the calculated mAP and loss values ​​converge, that is, the parameters in the network are basically stable, and the optimal model parameters are obtained. Finally, the optimal model parameters are used to detect the night traffic scene dataset to obtain the detection results.

Images