An improved YOLOV3 traffic video salient object detection method integrating temporal information

Through the improvement of YOLOV3 traffic video significant object detection method combined with Hard Attention and time information, the problem of difficulty in simulating driver gaze behavior in the prior art is solved, and the accurate detection of significant objects in complex traffic environments is achieved. Improved driving safety.

CN115984816BActive Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310017951.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-01-06
Publication Date
2024-11-22
Estimated Expiration
2043-01-06

AI Technical Summary

Technical Problem

The prior art is difficult to accurately simulate the driver's gaze behavior and significant object detection in complex traffic driving environments, resulting in hidden dangers in driving safety.

Method used

A method of improving YOLOV3 traffic video significant object detection using fusion time information is adopted to obtain significant object maps of traffic scenes by utilizing the Hard Attention of experienced drivers, and combining the Hard Attention information and time information fusion module, ResNets feature extraction module and fusion Feature prediction modules at different stages are trained and optimized to achieve real-time significant object detection.

Benefits of technology

This method can accurately detect important targets in significant areas in traffic driving environments, and more accurately and effectively detect important targets in driving environments in dark scenes and sudden overtaking scenarios, improving driving safety and performance of assisted driving systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115984816B_ABST
    Figure CN115984816B_ABST
Patent Text Reader

Abstract

The invention discloses a traffic video salient target detection method of an improved YOLOV3 that integrates time information. The salient target map of a traffic scene is obtained by using the Hard Attention of an experienced driver, and then an improved YOLOV3 that integrates time information is established and trained and optimized. After the training is completed, salient target detection is performed. The method of the invention proposes for the first time the salient target detection of a traffic scene that integrates Hard Attention information and time information. Compared with the existing full target detection model, the improved YOLOV3 that integrates time information not only focuses on important targets related to driving safety by integrating Hard Attention information features, but also greatly expands the scope of time information by integrating time information within the first 10 frames of the current detection frame. By integrating the scale prediction of more low-level features, targets important to the driving environment can be more accurately and effectively detected in dark scenes and scenes such as sudden overtaking; compared with the existing salient prediction model, it can focus on the salient area and accurately detect targets and positions in the salient area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an improved YOLOV3 traffic video salient target detection method integrating time information. Background Art

[0002] In traffic scenes, the environment on and around the road is complex and changeable. The vehicle is filled with a lot of information that affects driving safety during driving, such as other vehicles, pedestrians, traffic lights, traffic signs and other related targets. For driving safety, experienced drivers tend to focus on the most important areas and only pay attention to important targets related to driving safety in the salient area. For example, when drivers merge to the left, they are highly alert to the left lane and the traffic in front, but pay little attention to irrelevant street pedestrians or roadside parking. This hard attention mechanism helps drivers reduce the interference of irrelevant scene information and ensure driving safety. From the perspective of cognitive modeling, the driver's hard attention can be divided into two psychological processes, one is the driving environment saliency analysis and calculation, and the other is the target recognition in the salient area. First, the human-centered advanced driver assistant system understands the hard attention of experienced drivers, and then simulates the important target detection processing in the salient area, which can help drivers judge possible dangerous targets in advance, improve driving safety and comfort, and reduce the calculation burden of the assistant system.

[0003] However, so far, few studies have simulated the real attention and allocation of drivers while driving, and correspondingly, there are fewer model frameworks that can accurately simulate drivers' gaze behavior and salient target detection in complex traffic driving environments. Most existing methods are almost salient region prediction models. In a rapidly changing driving environment, if the intelligent driving system only provides salient regions, but requires the driver to further judge the important targets and locations that should be paid attention to in the salient regions, this will pose a serious hidden danger to driving safety. Summary of the invention

[0004] To solve the above problems, the present invention proposes an improved YOLOV3 traffic video salient target detection method that integrates time information, which can accurately and real-time detect important targets in salient areas in traffic driving environments, and has important practical significance for fields such as assisted driving systems and autonomous driving systems.

[0005] The technical solution adopted by the present invention is: an improved YOLOV3 traffic video salient target detection method integrating time information, and the specific steps are as follows:

[0006] S1. Use the Hard Attention of experienced drivers to obtain salient target maps in traffic scenes;

[0007] S2. Establish an improved YOLOV3 that integrates time information and perform training optimization. After training, perform salient target detection.

[0008] Furthermore, the step S1 is specifically as follows:

[0009] S11. Obtain pictures containing Hard Attention information through behavioral experiments;

[0010] The initial eye movement database provides data. The recorded eye movement data can simulate Hard Attention. Each frame of the original traffic scene video is extracted, and then the eye movement data obtained through behavioral experiments are mapped one by one to the corresponding traffic scene original image. The position of each gaze point corresponds to the spatial position of the original image, and an image containing Hard Attention information is obtained.

[0011] S12, refer to the picture containing Hard Attention information to mark the target of the original image of the traffic scene, and obtain a salient target map;

[0012] Use the 5 frames before and after the annotated frame as an annotation reference to ensure that important targets in the driving environment are not missed. There are two principles for annotation:

[0013] (1) From F t-5 to F t+5 There are at least 5 frames with eye movement data points in the traffic scene graph containing Hard Attention information;

[0014] Among them, F t represents the current detection frame, and t represents the frame number.

[0015] (2) When the target is in F t-5 to F t+5 There are at least 5 eye movement data points in a frame of the traffic scene image containing Hard Attention information.

[0016] When the target meets one of the above two conditions, it is marked and finally a salient target map is obtained.

[0017] Furthermore, in step S2, the improved YOLOV3 integrating time information includes a Hard Attention information and time information fusion module, a ResNets feature extraction module and a feature prediction module integrating different stages.

[0018] Furthermore, in the improved YOLOV3 that integrates time information, the Hard Attention information and time information fusion module includes a Hard Attention information feature enhancement module and a time information feature extraction module, which are as follows:

[0019] The Hard Attention information feature enhancement module structure includes: a group of 160×160 scale residual blocks and a group of CBAM modules; the 160×160 scale residual block contains 1 residual unit, and the CBAM contains a channel attention mechanism layer and then a spatial attention mechanism layer with a convolution kernel size of [7, 7] in series to obtain the feature T cb .

[0020] The temporal information feature extraction module structure includes: 4 groups of convolution blocks and a convolutional long short-term memory network; each group of convolution blocks specifically includes: a 3×3 convolution, a batch normalization unit, and a Leaky ReLU unit; the convolution kernels of the 4 groups of convolution blocks are 8, 16, 32, and 64, respectively, and the 3×3 convolution step size of the last group of convolution blocks is 2; the convolutional long short-term memory network includes 3 hidden layers, and the number of nodes in the hidden layers is 32, 32, and 64, respectively.

[0021] Furthermore, the Hard Attention information feature enhancement module is specifically as follows:

[0022] The current detection frame is reshaped to 320×320×3, and is subjected to feature extraction by the convolution block and the residual block. It is then input into the channel attention mechanism layer, and finally connected in series to the spatial attention mechanism layer with a convolution kernel size of [7,7]. CBAM calculates and assigns feature weights through the channel attention mechanism layer and the spatial attention mechanism layer to enhance the HardAttention information in the image.

[0023] Furthermore, in the improved YOLOV3 that integrates time information, the ResNets feature extraction module structure includes 4 groups of convolution blocks and residual blocks of different scales, 80×80, 40×40, 20×20, and 10×10, to extract features, as follows:

[0024] The residual block of 80×80 scale contains 2 residual units, the residual block of 160×160 scale contains 4 residual units, the residual block of 40×40 scale contains 4 residual units, and the residual block of 20×20 scale contains 2 residual units.

[0025] Furthermore, in the improved YOLOV3 that integrates temporal information, the structure of the module for integrating feature prediction at different stages includes integrating features at all different stages for target detection, integrating high-level features of four different scales with low-level features to simulate visual perception for object recognition through information processing of two pathways, constraining position prediction, making parameterization easier to learn, and making the network more stable, as follows:

[0026] The specific sizes of the four stages include: features of 10×10×21 scale; features of 20×20 scale are fused with features of 10×10 scale upsampled to 20×20×21 scale; features of 40×40 scale are fused with features of 20×20 scale upsampled to 40×40×21 scale; features of 80×80 scale are fused with features of 40×40 scale upsampled to 80×80×21 scale.

[0027] Furthermore, in step S2, the training process of improving YOLOV3 by integrating time information is as follows:

[0028] S21, the Hard Attention information and time information fusion module structure is used to enhance the Hard Attention information of the current detection frame and the time information features missing from the current detection frame;

[0029] The current detection frame F t Together with the previous 10 frames, a total of 6 frames form a time series. The time information features are extracted through convolutional LSTM. The Hard Attention information features enhanced by the current detection frame are first fused with the time information features extracted from the multi-frame time series, and then the next step of target detection processing is performed.

[0030] Among them, the 10 frames before the current detection frame are selected to take a frame every 2 frames, that is, (F t-10 , F t-8 , …, F t-2 , F t ).

[0031] S22, the ResNets feature extraction module is used to extract low-level and high-level feature information from the fused features;

[0032] S23, the fusion prediction module of features at different stages can be used to accurately detect objects of different sizes and positions in the salient area of ​​the traffic video;

[0033] S24, divide the data set into training set, validation set, and test set in a ratio of about 8:3:2. In order to shorten the training time, the size of the input image is changed to 320×320×3 (height H×width W×number of channels C);

[0034] S25, first randomly initialize the parameters of the improved YOLOV3 that integrates time information (the network features are all expressed as H×W×C), and take the training set pictures (320×320×3) and each frame picture and its previous 10 frames (take one frame every 2 frames, (F t-10 , F t-8 , …, F t-2 , F t )) A total of 6 frames form a time series, which are input into the Hard Attention information and time information fusion modules respectively. The fused features are input into the ResNets feature extraction module, and the binary cross entropy is used to calculate the loss value between the detected salient targets and the corresponding labels (targets in the standard salient area); the initial learning rate is 10 -2 And set the minimum learning rate to 10 -4 , the momentum value is 0.937, and the decay rate is 5 -4 The SGD optimizer is used for training, and the model parameters are saved every 10 epochs and the optimal model parameters are updated;

[0035] S26. After each epoch of training, the model is verified with the validation set, and steps S21-23 are repeated for iterative training until the calculated mAP and loss values ​​have almost no fluctuations, that is, the parameters in the network are basically stable. The optimal model parameters are obtained and saved for subsequent testing.

[0036] Beneficial effects of the present invention: The method of the present invention uses the Hard Attention of experienced drivers to obtain a salient target map of a traffic scene, then establishes an improved YOLOV3 that integrates time information and performs training optimization, and performs salient target detection after the training is completed. The method of the present invention proposes for the first time the salient target detection of a traffic scene that integrates Hard Attention information and time information. Compared with the existing full-target detection model, the improved YOLOV3 that integrates time information not only focuses on important targets related to driving safety by integrating Hard Attention information features, but also greatly expands the scope of time information by integrating time information within the 10 frames before the current detection frame. By integrating more low-level feature scale predictions, targets that are important to the driving environment can be more accurately and effectively detected in dark scenes and scenes such as sudden overtaking; compared with the existing saliency prediction model, it can focus on salient areas and accurately detect targets and positions within salient areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention is a flow chart of a method for detecting salient objects in traffic videos by integrating time information.

[0038] Figure 2 Reference F is used in the embodiment of the present invention. t-5 to F t+5 The original traffic scene image of the Ft-th frame is marked with the image containing Hard Attention information.

[0039] Figure 3 This is an overall structural diagram of an improved YOLOV3 that integrates time information in the present invention.

[0040] Figure 4 This is a diagram showing salient target detection on a traffic scene graph by the present invention and other algorithms in an embodiment of the present invention.

[0041] Figure 5 This is a comparison diagram of saliency prediction and salient target detection in a traffic scene graph in an embodiment of the present invention.

[0042] Figure 6 This is a comparison diagram of all-target detection and salient-target detection in a traffic scene graph in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further explained below in conjunction with the accompanying drawings and embodiments.

[0044] like Figure 1 As shown, an improved YOLOV3 traffic video salient target detection method integrating time information of the present invention has the following specific steps:

[0045] S1. Use the Hard Attention of experienced drivers to obtain salient target maps in traffic scenes;

[0046] S2. Establish an improved YOLOV3 that integrates time information and perform training optimization. After training, perform salient target detection.

[0047] In this embodiment, the step S1 is specifically as follows:

[0048] S11. Obtain pictures containing Hard Attention information through behavioral experiments;

[0049] For the study of visually salient targets in traffic scenes, it is very important to integrate the visual saliency mechanism. The eye movement data obtained through behavioral experiments can show the Hard Attention information very vividly and accurately, which can perfectly connect with the subsequent salient target detection. The initial eye movement database provides data. The initial eye movement database contains multiple traffic videos of normal driving in urban scenes and high-speed scenes saved by driving recorders. The size of the traffic video is 1280×720, and the duration of the traffic video lasts from 52 seconds to 181 seconds (the average duration of each video is 161.4 seconds, the standard deviation is 38.0, and most videos are 181 seconds). A total of 28 subjects participated in the behavioral experiment, with a male-female ratio of 4:3. To ensure the accuracy of the experiment, all subjects have at least two years of driving experience and are between 23 and 45 years old. Because the experimental process requires the subjects to completely simulate the real driving scene to watch the video, the recorded eye movement data can simulate Hard Attention.

[0050] Each frame of the 16 existing original traffic scene videos is extracted, and a total of 75,050 frames of traffic scene original images are obtained. Then the eye movement data obtained through behavioral experiments are mapped one by one to the corresponding traffic scene original images. The position of each gaze point corresponds to the spatial position of the original image, and an image containing Hard Attention information is obtained.

[0051] S12, refer to the picture containing Hard Attention information to mark the original image of the traffic scene, and obtain a salient target map;

[0052] Considering the time cost of labeling and the similarity between adjacent frames, and studies have shown that the latency of attention transfer is about 200-300ms, the images of the 5 frames before and after the labeled frame are used as the labeling reference, such as Figure 2 As shown, there are two principles for marking to ensure that important targets in the driving environment are not missed:

[0053] (1) From F t-5 to F t+5 At least 5 frames of the traffic scene graph containing Hard Attention information have eye movement data points, because when the driver spends a lot of time focusing on a target, it can be reasonably inferred that the target is a relevant and important target in the driving environment;

[0054] Among them, F t represents the current detection frame, and t represents the frame number.

[0055] (2) When the target is in F t-5 to F t+5There are at least 5 eye movement data points in one frame of the traffic scene graph containing Hard Attention information. When a target attracts the attention of multiple drivers at the same time, although the target may be noticed for a short time, it is likely to be a sudden target, such as overtaking, which is obviously an important target related to the driving environment.

[0056] When the target meets one of the above two conditions, it is marked and finally a salient target map is obtained.

[0057] like Figure 3 As shown, an overall structure diagram of an improved YOLOV3 that integrates time information in this embodiment. In this embodiment, in step S2, the improved YOLOV3 that integrates time information includes a Hard Attention (hard attention mechanism) information and time information fusion module, a ResNets (ResNets: Residual Networks) feature extraction module and a prediction module that integrates features from different stages.

[0058] In this embodiment, in the improved YOLOV3 that integrates time information, the Hard Attention information and time information fusion module includes a Hard Attention information feature enhancement module and a time information feature extraction module, which are as follows:

[0059] The Hard Attention information feature enhancement module structure includes: a set of 160×160 scale residual blocks and a set of CBAM; the 160×160 scale residual block contains 1 residual unit, and the CBAM contains a channel attention mechanism layer and then a spatial attention mechanism layer with a convolution kernel size of [7, 7] in series to obtain the feature T cb Although the existing feature extraction module can extract feature information from the image, it lacks the saliency processing of the features. CBAM (CBAM: Convolutional Block Attention Module) can further enhance the Hard Attention information in the image by calculating and assigning feature weights in the channel and spatial dimensions.

[0060] The temporal information feature extraction module structure includes: 4 groups of convolution blocks and a convolutional long short-term memory network; each group of convolution blocks specifically includes: a 3×3 convolution, a batch normalization unit, and a leaky ReLU unit; the convolution kernels of the 4 groups of convolution blocks are 8, 16, 32, and 64, respectively, and the 3×3 convolution step size of the last group of convolution blocks is 2; the convolutional long short-term memory network includes 3 hidden layers, and the number of nodes in the hidden layers is 32, 32, and 64, respectively. Compared with FC LSTM, convLSTM can not only reduce a large amount of redundant spatial data, but also the convolution operation can obtain spatial correlation, so it is more suitable for fusing the temporal information of the video.

[0061] Different from the spatial feature extraction of a single frame image, the input of the temporal information feature extraction structure is the current detection frame and its previous 10 frames (one frame is taken every 2 frames, that is, (F t-10 , F t-8 , …, F t-2 , F t )) A total of 6 frames form a time series F. The feature F' of the time series F is extracted through 4 groups of convolution blocks, and then F' is input into the convolutional LSTM to learn the time information of the previous and next frames. In order to obtain the target feature information that needs to be focused on when the time changes are the largest, only the feature T of the last layer of the time series is retained. ls . The feature T ls With T cb After concat, they are fused through a set of convolution blocks and finally input into a 80×80 scale residual block for feature extraction.

[0062] In this embodiment, the Hard Attention information feature enhancement module is specifically as follows:

[0063] The current detection frame is reshaped to 320×320×3, and is subjected to feature extraction by the convolution block and the residual block. It is then input into the channel attention mechanism layer, and finally connected in series to the spatial attention mechanism layer with a convolution kernel size of [7,7]. CBAM calculates and assigns feature weights through the channel attention mechanism layer and the spatial attention mechanism layer to enhance the HardAttention information in the image.

[0064] In this embodiment, in the improved YOLOV3 that integrates time information, the ResNets feature extraction module structure includes 4 groups of convolution blocks and residual blocks of different scales, 80×80, 40×40, 20×20, and 10×10, to extract features, as follows:

[0065] Spatial features are very important for detecting objects and locations in traffic scenes, and the residual network can effectively extract the spatial characteristics of images.

[0066] The residual block of 80×80 scale contains 2 residual units, the residual block of 160×160 scale contains 4 residual units, the residual block of 40×40 scale contains 4 residual units, and the residual block of 20×20 scale contains 2 residual units.

[0067] In this embodiment, in the improved YOLOV3 that integrates time information, the structure of the feature prediction module that integrates different stages includes integrating features of all different stages for target detection, integrating high-level features of four different scales with low-level features to simulate visual perception through information processing of two pathways for object recognition, and constraining position prediction, making the network more stable, as follows:

[0068] The specific sizes of the four stages include: 10×10×21 scale features; 20×20×21 scale features after fusion of 20×20 scale features and 10×10 scale upsampled features; 40×40×21 scale features after fusion of 40×40 scale features and 20×20 scale upsampled features; 80×80×21 scale features after fusion of 80×80 scale features and 40×40 scale upsampled features. Features with larger scales are more accurate in detecting small targets, while features with smaller scales are easier to detect large targets.

[0069] In this embodiment, in step S2, the training process of improving YOLOV3 by integrating time information is as follows:

[0070] S21, the Hard Attention information and time information fusion module structure is used to enhance the Hard Attention information of the current detection frame and the time information features missing from the current detection frame;

[0071] For target detection in traffic scenes, vehicles travel very fast, and the visual information features of salient targets in the scene will change significantly in a very short time, and there may even be targets that appear suddenly, such as overtaking in the left and right lanes. t and the 10 frames before it (take a frame every 2 frames, that is (F t-10 , F t-8 , …, F t-2 , F t )) A total of 6 frames constitute a time series, and the time information features are extracted through convolutional LSTM. Research on visual perception in cognitive neuroscience shows that the pathway for transmitting visual information is that the retina collects information and then gathers it to the primary visual cortex (V1 area), and then the next step of information processing is carried out. The Hard Attention information features enhanced by the current detection frame are first fused with the time information features extracted from the multi-frame time series, and then the next step of target detection processing is carried out.

[0072] Among them, the x frames before the current detection frame are selected every 2 frames, that is, (F t-x , F t-8 , …, F t-2 , F t ), in order to have more distant time information features, through the ablation experiment, x is finally taken as 10 to achieve real-time detection speed.

[0073] S22, the ResNets feature extraction module is used to extract low-level and high-level feature information from the fused features;

[0074] S23, the fusion prediction module of features at different stages can be used to accurately detect objects of different sizes and positions in the salient area of ​​the traffic video;

[0075] Leslie Ungerleider and Mortimer Mishkin showed in their study of visual perception that object recognition processes different types of information through two pathways. One is the ascending longitudinal fascicle, which includes axons terminating in the posterior parietal cortex. This area is related to the location of the object, and the other is the descending longitudinal fascicle, which includes axons terminating in the inferior temporal cortex. This area is related to the category of object recognition. In the process of extracting features from images, the low-level features in the previous stage contain more location information of the target, while the high-level features in the later stage contain more category information of the target. Therefore, the features of different stages in the ResNets feature extraction module are selected for fusion and prediction, which can more accurately determine the location and category of the target.

[0076] S24, divide the data set into training set, validation set, and test set in a ratio of about 8:3:2, and change the size of the input image to 320×320×3 (height H×width W×number of channels C);

[0077] S25, first randomly initialize the parameters of the improved YOLOV3 that integrates time information (the network features are all expressed as H×W×C), and take the training set pictures (320×320×3) and each frame picture and its previous 10 frames (one frame is taken every 2 frames, that is, (F t-10 , F t-8 , …, F t-2 , F t )) A total of 6 frames form a time series, which are input into the Hard Attention information and time information fusion modules respectively. The fused features are input into the ResNets feature extraction module, and the binary cross entropy is used to calculate the loss value between the detected salient targets and the corresponding labels (targets in the standard salient area); the initial learning rate is 10 -2 And set the minimum learning rate to 10 -4 , the momentum value is 0.937, and the decay rate is 5-4 The SGD optimizer is used for training, and the model parameters are saved every 10 epochs and the best model parameters are updated.

[0078] S26. After each epoch of training, the model is verified with the validation set, and steps S21-23 are repeated for iterative training until the calculated mAP and loss values ​​have almost no fluctuations, that is, the parameters in the network are basically stable. The optimal model parameters are obtained and saved for subsequent testing.

[0079] In this embodiment, the method of the present invention further includes step S3, testing and evaluating the improved YOLOV3 effect of integrating time information, as follows:

[0080] S31, import the best model parameters obtained in step S24, and input a random video frame in the test set to test the performance of the improved YOLOV3 that integrates time information.

[0081] S32, conduct qualitative analysis and quantitative index analysis on the test results. Figure 4 shown. Figure 4 (a) is a random original video frame, i.e., the original traffic scene image; Figure 4 (b) is the standard salient target map, i.e., the real salient target map; Figure 4 (c) is the prediction result of the method of the present invention, that is, the detection result diagram of the method of the present invention; Figure 4 (d) is the prediction result of YOLOV3, that is, the detection result diagram of YOLOV3. Figure 4 (c) It can be clearly seen that the targets detected by the improved YOLOV3 with temporal information proposed in this invention are almost the same as the standard salient target map. From top to bottom, Figure 4 From the first row, it can be seen that the detection frame of the proposed method can more accurately frame the detected target than the existing YOLOV3. Figure 4 The second row shows that the existing YOLOV3 has the problem of easily misdetecting targets in the image compared with the method of the present invention; Figure 4 The third, fourth and fifth rows show that compared with the existing YOLOV3, the method of the present invention can more accurately and effectively detect targets important to the driving environment in dark scenes and sudden overtaking scenes because the method of the present invention integrates the time information missing in the current detection frame. This shows that the improved YOLOV3 that integrates time information of the present invention can well simulate the real driving scene and perform significant target detection, and the method of the present invention is reliable and effective.

[0082] Table 1 is a comparison of the evaluation indicators of the method of the present invention and other excellent methods. It is the result of quantitative analysis. The main evaluation indicators used include: mAP value, car AP value, traffic signs AP value, F1-score, FPS (frame rate per second). Comprehensive quantitative indicator analysis also shows that the method of the present invention has superior performance.

[0083] Table 1

[0084] Method mAP car Traffic signs F1-score FPS Faster R-CNN 72.6 77 68.2 0.72 10.5 VFNet 76.7 79.8 73.6 0.50 31.0 YOLOV3 83.5 88.9 78 0.82 35.1 ID-YOLO 85.9 92.8 78.9 0.86 68 This method 88.94 93.95 83.93 0.83 30

[0085] like Figure 5 As shown in FIG. 1 , a comparison diagram of saliency prediction and salient target detection in a traffic scene graph in this embodiment is shown, where: Figure 5 (a) is the significance prediction map, i.e., the prediction result map of the significance model; Figure 5 (b) is a salient target detection diagram, i.e., a detection result diagram of the method of the present invention. Figure 5 From the qualitative results, it can be seen that compared with the existing saliency prediction models, the method of the present invention can not only focus on the salient area, but also accurately detect the objects and positions in the salient area.

[0086] like Figure 6 As shown in FIG. 1 , a comparison diagram of full target detection and salient target detection in a traffic scene graph in this embodiment is shown, where: Figure 6 (a) is the full target detection map, that is, the detection result map of YOLOV3; Figure 6 (b) is a salient target detection diagram, i.e., a detection result diagram of the method of the present invention. Figure 6 From the qualitative results, it can be seen that compared with the existing full-target detection model, the method of the present invention not only focuses on important targets related to driving safety by integrating Hard Attention information features, but also integrates time information features. In dark scenes and sudden overtaking scenes, it can more accurately and effectively detect targets that are important to the driving environment.

[0087] In summary, the method of the present invention is applied to the field of computer vision technology. Through the salient video of the Hard Attention mechanism of experienced drivers, a salient target detection dataset is constructed. The Hard Attention information and time information fusion module first enhances the Hard Attention information features of the current detection frame, then extracts the time information features of multiple frames, and finally fuses the Hard Attention information features with the time information features; the ResNets feature extraction module extracts low-level and high-level features from the fused features; finally, by fusing all the different stage features in the ResNets feature extraction module, the salient targets of the traffic video are accurately detected. Compared with the existing full-target detection model, the improved YOLOV3 that integrates time information not only focuses on important targets related to driving safety by integrating Hard Attention information features, but also can more accurately and effectively detect targets important to the driving environment in dark scenes and sudden overtaking scenes by integrating time information features; compared with the existing salient prediction model, it can not only focus on the salient area, but also accurately detect the targets and positions in the salient area.

[0088] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An improved YOLOV3 traffic video salient object detection method integrating temporal information, the specific steps are as follows: S1. Use the Hard Attention of experienced drivers to obtain salient target maps in traffic scenes; S2. Establish an improved YOLOV3 that integrates time information and perform training optimization. After the training is completed, perform salient target detection. The step S1 is specifically as follows: S11. Obtain pictures containing Hard Attention information through behavioral experiments; The initial eye movement database provides data. The recorded eye movement data can simulate Hard Attention. Each frame of the original traffic scene video is extracted. Then, the eye movement data obtained through behavioral experiments are mapped one by one to the corresponding original traffic scene image. The position of each gaze point corresponds to the spatial position of the original image, and an image containing Hard Attention information is obtained. S12, refer to the picture containing Hard Attention information to mark the original image of the traffic scene, and obtain a salient target map; Use the 5 frames before and after the annotated frame as an annotation reference to ensure that important targets in the driving environment are not missed. There are two principles for annotation: (1) From F t-5 to F t+5 There are at least 5 frames with eye movement data points in the traffic scene graph containing Hard Attention information; in, F t represents the current detection frame, and t represents the frame number; (2) When the target is in F t-5 to F t+5 There are at least 5 eye movement data points in a frame of the traffic scene image containing Hard Attention information; When the target meets one of the above two conditions, it is marked and finally a salient target map is obtained; In the step S2, the improved YOLOV3 integrating time information includes a Hard Attention information and time information fusion module, a ResNets feature extraction module and a feature prediction module integrating different stages; In the improved YOLOV3 that integrates time information, the Hard Attention information and time information fusion module includes a Hard Attention information feature enhancement module and a time information feature extraction module, which are as follows: The Hard Attention information feature enhancement module structure includes: a group of 160×160 scale residual blocks and a group of CBAM modules; the 160×160 scale residual block contains 1 residual unit, and the CBAM contains a channel attention mechanism layer and then a spatial attention mechanism layer with a convolution kernel size of [7, 7] in series to obtain the feature T cb ; The temporal information feature extraction module structure includes: 4 groups of convolution blocks and a convolutional long short-term memory network; each group of convolution blocks specifically includes: a 3×3 convolution, a batch normalization unit, and a Leaky ReLU unit; the convolution kernels of the 4 groups of convolution blocks are 8, 16, 32, and 64, respectively, and the 3×3 convolution step size of the last group of convolution blocks is 2; the convolutional long short-term memory network includes 3 hidden layers, and the number of nodes in the hidden layers is 32, 32, and 64, respectively.

2. According to claim 1, an improved YOLOV3 traffic video salient target detection method integrating time information is characterized in that: The Hard Attention information feature enhancement module is specifically as follows: The current detection frame is reshaped to 320×320×3, and is subjected to feature extraction by the convolution block and the residual block. It is then input into the channel attention mechanism layer, and finally the spatial attention mechanism layer with a convolution kernel size of [7,7] is connected in series. CBAM calculates and assigns feature weights through the channel attention mechanism layer and the spatial attention mechanism layer to enhance the Hard Attention information in the image.

3. According to the improved YOLOV3 traffic video salient target detection method integrating time information according to claim 1, it is characterized in that: In the improved YOLOV3 that integrates time information, the ResNets feature extraction module structure includes 4 groups of convolution blocks and residual blocks of different scales, 80×80, 40×40, 20×20, and 10×10, to extract features, as follows: The residual block of 80×80 scale contains 2 residual units, the residual block of 160×160 scale contains 4 residual units, the residual block of 40×40 scale contains 4 residual units, and the residual block of 20×20 scale contains 2 residual units.

4. According to the improved YOLOV3 traffic video salient target detection method integrating time information according to claim 1, it is characterized in that: In the improved YOLOV3 that integrates temporal information, the structure of the feature prediction module that integrates different stages includes integrating features of all different stages for target detection, integrating high-level features of four different scales with low-level features to simulate visual perception through information processing of two pathways for object recognition, constraining position prediction, making parameterization easier to learn, and making the network more stable, as follows: The specific sizes of the four stages include: features of 10×10×21 scale; features of 20×20 scale are fused with features of 10×10 scale upsampled to 20×20×21 scale; features of 40×40 scale are fused with features of 20×20 scale upsampled to 40×40×21 scale; features of 80×80 scale are fused with features of 40×40 scale upsampled to 80×80×21 scale.

5. According to the improved YOLOV3 traffic video salient target detection method integrating time information according to claim 1, it is characterized in that: In step S2, the training process of improving YOLOV3 by integrating time information is as follows: S21, the Hard Attention information and time information fusion module structure is used to enhance the Hard Attention information of the current detection frame and the time information features missing from the current detection frame; The current detection frame F t Take one frame every 2 frames from the previous 10 frames and compare it with the current detection frame F t A total of 6 frames form a time series, namely (F t-10 , F t-8 , …, F t-2 , F t ), extract the time information features through convolutional LSTM, fuse the HardAttention information features enhanced by the current detection frame with the time information features extracted from the multi-frame time series, and then proceed to the next step of target detection processing; S22, the ResNets feature extraction module is used to extract low-level and high-level feature information from the fused features; S23, the fusion prediction module of features at different stages can be used to accurately detect objects of different sizes and positions in the salient area of ​​the traffic video; S24, divide the data set into training set, validation set, and test set in a ratio of 8:3:

2. In order to shorten the training time, the size of the input image is changed to 320×320×3, that is, height H×width W×number of channels C; S25, first randomly initialize the improved YOLOV3 parameters that integrate time information, and then convert the training set images and time series (F t-10 , F t-8 , …, F t-2 , F t ), respectively input into the Hard Attention information and time information fusion modules, and input the fused features into the ResNets feature extraction module. The binary cross entropy is used to calculate the loss value between the detected salient targets and the corresponding labels. The initial learning rate is 10 -2 And set the minimum learning rate to 10 -4 , the momentum value is 0.937, and the decay rate is 5 -4 The SGD optimizer is used for training, and the model parameters are saved every 10 epochs and the optimal model parameters are updated; Among them, in the improved YOLOV3 parameters, the network features are all represented by H×W×C; the training set image size is 320×320×3; the corresponding label is the target in the standard salient area; S26. After each epoch of training, the model is verified with the validation set, and steps S21-23 are repeated for iterative training until the calculated mAP and loss values ​​have almost no fluctuations, that is, the parameters in the network are basically stable. The optimal model parameters are obtained and saved for subsequent testing.