A traffic fixation target detection method based on saliency prior

Abstract

The present invention provides a method for detecting traffic fixation targets based on saliency prior, which is applied to the field of traffic fixation target detection. Current researchers treat visual saliency and target detection as two independent tasks. In driving scenes, experienced drivers will pay close attention to objects related to driving tasks under the guidance of selective attention mechanisms, and ignore a large amount of information irrelevant to driving tasks, thereby achieving safe driving. The present invention simulates the driver's perception of traffic scenes and focuses on detecting objects that attract the driver's attention. To this end, the present invention establishes a data set that can be used for fixation target detection. Then, the salient area predicted by the salient area guidance module is used to guide the target detection module to pay more attention to the fixation target, thereby improving the detection accuracy of the model. The present invention provides an effective method for traffic fixation target detection and provides another strategy for intelligent driving or advanced driver assistance systems.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a traffic fixation target detection method based on saliency prior. Background Art

[0002] With the development of automobile electronics and intelligence, intelligent driving has become one of the mainstream trends in future automobile development. Visual saliency and object detection, as important components of intelligent driving systems, are important guarantees for safe driving.

[0003] Currently, researchers usually treat visual saliency and object detection in traffic scenes as two independent tasks. That is, visual saliency can predict the area or target that the driver is paying attention to, but does not provide the target category; while object detection tries to detect all the targets in the scene, but cannot predict which objects in the scene attract the driver's attention. These objects that attract the driver's attention are closely related to the current driving task and driving safety. Therefore, there is an urgent need for a method to establish a connection between the two to improve the detection accuracy and detection performance. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a traffic gaze target detection method based on saliency prior, which can simulate the driver's selective attention, combine visual saliency and target detection, focus on the gaze target detection within the driver's gaze area, improve the model detection accuracy, and provide a reference strategy for intelligent driving or advanced driver assistance systems.

[0005] The above technical objectives of the present invention are achieved through the following technical solutions:

[0006] A method for detecting traffic gaze targets based on saliency priors comprises the following steps:

[0007] S1, establishing a gaze target detection dataset based on the driver's eye movements, wherein the gaze target detection dataset includes saliency information for attracting the driver's attention in a traffic scene, and object categories in a salient area that attracts the driver's attention;

[0008] S2, builds a strong baseline detection module based on the YOLOv5-L model and the lightweight attention module CBAM;

[0009] S3, sharing the backbone and top-down paths of the strong baseline detection module established in S2, establishing a salient area guidance module for predicting a salient area that attracts the driver's attention in a traffic scene, wherein the salient area is used to guide the strong baseline detection module to focus on the fixation target in the driver's attention area;

[0010] S4, proposes a saliency guidance strategy by simulating the driver's selective attention in traffic scenes;

[0011] S5, based on the strong baseline detection module established in S2, the salient area guidance module constructed in S3, and the saliency guidance strategy proposed in S4, a fixation target detection model is established;

[0012] S6, scaling the images of the training set to a given input resolution and inputting the images into the gaze target detection model constructed in S5, and training the gaze target detection model, thereby improving the detection performance of the gaze target detection model.

[0013] The present invention is further configured as follows: in S1, when establishing a gaze target detection data set based on the driver's eye movement, the following steps are included:

[0014] Step a, using a driving recorder to collect traffic videos of vehicles in different weather conditions and driving scenarios;

[0015] Step b, allowing the driver to watch the collected traffic video during the simulated driving task, and using an eye tracker to record the driver's eye movement information;

[0016] Step c, mapping the recorded gaze points of all drivers onto the traffic video;

[0017] Step d, marking the gaze target, and finally obtaining the driver's gaze target detection data set;

[0018] Step e: Generate salient region masks based on the labeled fixation target detection training set.

[0019] The present invention is further configured as follows: in step d, when marking the gaze target, one frame is extracted from every five frames of the traffic video to mark the gaze target.

[0020] The present invention is further configured as follows: in S2, a basic module constructed by convolution operation, convolution (k=3, s=1), batch normalization processing and SILU activation is added to the first layer of backbone, and a lightweight attention module CBAM is added to the first and second downsampling layers, so as to establish a strong baseline detection module to detect the specific position and category of the gaze target;

[0021] The strong baseline detection module is optimized using the loss function, as shown in the following formula (1):

[0022] (1)

[0023] In the formula, represents the CIOU loss, which is used to calculate the position deviation between the predicted target position and the marked target position; and is the binary cross entropy loss, It is used to calculate the difference between the predicted target category and the labeled target category; Used to calculate the confidence difference between the predicted target and the labeled target.

[0024] The present invention is further configured as follows: in S3, the composite loss function is used to optimize the salient region guidance module, as shown in the following formula (2):

[0025] (2)

[0026] In the formula, is the salient area mask generated in the above process, The scale is and Represent the width and height of the input image respectively, represents the saliency map predicted by the salient region guidance module, with its scale and The agreement, represents the binary cross entropy loss, which is used to optimize the salient region guidance module from a fine-grained pixel level. It optimizes the IOU loss of the salient region guidance module from the overall region level.

[0027] The present invention is further configured as follows: the salient map generated by the salient region guiding module and the shallow feature fusion of the backbone of the target detection module are fused, and the fusion expression is shown in the following formula (3):

[0028] (3)

[0029] In the formula, represents element-wise multiplication, represents element-by-element addition, is a learnable parameter used to weight the saliency map, represents the saliency map predicted by the salient region guidance module, It is the shallow feature extracted by the target detection module, and its width and height are The agreement, It represents the fused features, which include the global information of the traffic scene and the enhanced features of the gaze target.

[0030] The present invention is further configured as follows: in S6, the fixation target detection model is optimized using a composite loss function, as shown in the following formula (4):

[0031] (4)

[0032] In the formula, is the loss defined in formula (1), is the loss defined in equation (2), is the weight coefficient.

[0033] The present invention is further configured as follows: in S6, the salient area mask generated in step e and formula (2) are used to pre-train the salient area guidance module, the weights obtained by the pre-training are used to initialize the fixation target detection model, the salient area mask generated in step e and the annotated fixation target detection data set and formula (4) are used to train the fixation target detection model, and when the mAP value of the validation set is maximized, the model parameters are retained for testing.

[0034] The invention has the beneficial effects that: the invention simulates the selective attention of the driver and detects the fixation targets that are closely related to the current driving task. The fixation target detection dataset proposed for this purpose provides a benchmark for studying traffic scene perception from the perspective of the driver. In addition, the proposed method establishes a connection between visual saliency and target detection, and uses the predicted salient area as a saliency prior to guide the model to pay more attention to the fixation targets, so that the model can not only predict which targets in the traffic scene are related to the current driving task, but also detect the specific categories and locations of these targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a gaze target detection dataset and a salient area mask sample in an embodiment of the present invention;

[0036] Figure 2 A structural diagram of a gaze target detection model FOD-Net provided in an embodiment of the present invention;

[0037] Figure 3 is a diagram of a salient region and feature fusion structure predicted in an embodiment of the present invention;

[0038] Figure 4 This is a detection effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] The implementation process of an embodiment of the present invention includes the following steps:

[0041] S1, in order to carry out the research on gaze target detection, a driver gaze target detection dataset was established based on the eye movements of 30 drivers, specifically:

[0042] a. Use a driving recorder to collect traffic videos of vehicles in different weather conditions and driving scenarios;

[0043] b. Ask 30 drivers with many years of driving experience to watch the collected traffic videos during a simulated driving task, and use an eye tracker to collect the drivers' eye movement information;

[0044] c, mapping the recorded gaze points of all drivers onto the traffic video;

[0045] d. Extract one frame from every five frames of the video to mark the gaze target. The criteria used are: ① A target in the frame to be marked is viewed by no less than five drivers; ② Referring to the gaze points in the first five frames and the last five frames of the frame to be marked, the driver looks at the same target in at least five frames. If any of the above two criteria is met, the target is marked as the gaze target; the final driver gaze target detection dataset contains 29,053 frames of images, including 20,299 frames for training, 2,917 frames for validation, 5,837 frames for testing, and five target categories: traffic-sign, person, car, motorcycle, and traffic-light.

[0046] e. In order to optimize the salient area guidance module, a salient area mask is generated based on the labeled fixation target detection training set, specifically: ① Calculate the area of ​​the bounding box of the fixation target in each frame in the labeled training set in step (5); ② Set the area value inside the bounding box to 1, and the area value outside the bounding box to 0. It should be noted that for bounding boxes with smaller areas, it is necessary to expand a certain area outward along the four sides of the bounding box when generating the area mask. In this case, for bounding boxes with an area less than 1024 pixels, 4 pixels are chosen to be expanded on all sides. Examples of fixation target detection datasets and salient area masks are as follows: Figure 1 shown.

[0047] S2, in order to effectively fuse the salient prior features, a basic module constructed by convolution (k=3, s=1), batch normalization (batchnorm) and SILU activation is added to the first layer of the backbone. In order to capture the characteristics of the gaze target, a lightweight attention module CBAM is added to the first and second downsampling layers. In order to characterize small-scale targets and reduce model parameters, two detection heads (H2 and H4) corresponding to 4x and 16x downsampling rates are used to replace the H3, H4 and H5 detection heads in YOLOv5-L. At this point, a strong baseline detection module ODM is established to detect the specific location and category of the gaze target, such as Figure 2 As shown to the right of the dotted line in .

[0048] To optimize ODM, a loss function is used as shown in Equation (1):

[0049] (1)

[0050] In the formula, Represents the CIOU loss, which is used to calculate the position deviation between the predicted target position and the labeled target position. and is the binary cross entropy loss, It is used to calculate the difference between the predicted target category and the labeled target category. Used to calculate the confidence difference between the predicted target and the labeled target. Same as the YOLOv5-L model, , and Set them to 0.05, 0.5 and 1.0 respectively.

[0051] S3, sharing the backbone and top-down paths of the ODM established in S2, considers that there may be a large number of small salient regions in traffic scenes, and adds two upsampling layer modules on the basis of the top-down path to construct a salient region guidance module SRGM (such as Figure 2 The salient area that attracts the driver’s attention in the traffic scene can be predicted by using the salient prior to guide the ODM to pay more attention to the fixation targets in the driver’s attention area.

[0052] In order to optimize the salient region guidance module SRGM, a composite loss function is established as shown in the following equation (2):

[0053] (2)

[0054] In the formula, is the salient region mask generated in S1 ( The scale is and represent the width and height of the input image respectively). represents the saliency map predicted by SRGM, its scale and of consistency. represents the binary cross entropy loss, which is used to optimize the SRGM from a fine-grained pixel level. It optimizes the IOU loss of SRGM from the overall regional level. is a hyperparameter, which is set to 0.2 in this case.

[0055] S4, in order to simulate the selective attention of drivers in traffic scenes, a saliency guidance strategy is proposed. First, the salient area predicted by SRGM in S3 is used as a saliency prior to guide the object detection module ODM established in S2 to pay more attention to the gaze target in the driver's gaze area. Specifically, the predicted saliency map and the shallow feature fusion of the backbone of ODM are used, such as Figure 3 The fusion expression is shown in formula (3):

[0056] (3)

[0057] In the formula, represents element-wise multiplication, Represents element-by-element addition. is a learnable parameter used to weight the saliency map. Represents the saliency map predicted by SRGM in S3. is the shallow feature extracted by ODM, and its width and height are of consistency. The fused features contain not only the global information of the traffic scene, but also the enhanced features of the gaze target. The fusion of the saliency prior and the features extracted by ODM is used to simulate the driver's visual behavior and guide the detector to focus on the salient area, thereby extracting the fine features of the gaze target in a targeted manner, and finally improving the accuracy of target detection.

[0058] In addition, given that the target category predicted by the object detection module ODM can provide rich semantic information for the salient region guidance module SRGM, which can refine the salient region, further promoting ODM to focus on the fixation target region more accurately to extract more discriminative fixation target features. Therefore, weight sharing is used to promote the optimization of SRGM and ODM to further improve the detection accuracy of fixation targets.

[0059] S5. Based on the detection module ODM established in S2, the salient region guidance module SRGM constructed in S3, and the saliency guidance strategy proposed in S4, a fixation target detection model FOD-Net is established to detect the fixation target, such as Figure 2 shown.

[0060] In order to optimize FOD-Net, a composite loss function is established as follows (4):

[0061] (4)

[0062] In the formula, is the loss defined in formula (1). is the loss defined in Equation (2). Since fixation target detection is the most important task in this work, salient region prediction is an auxiliary task, which aims to guide ODM to focus on the fixation target and thus improve the detection accuracy. Therefore, a weight coefficient is introduced to facilitate the optimization of the model. In this case, is set to 0.1.

[0063] S6, in order to promote the optimization of the fixation target detection model FOD-Net, the salient region mask generated in S1 and formula (2) in S3 are first used to pre-train the salient region guidance module SRGM. The weights obtained after 100 epochs of training are used to initialize FOD-Net. At this time, the salient region mask generated in S1 and the annotated fixation target detection dataset and formula (4) in S5 are used to train FOD-Net. When the mAP value of the validation set is maximized, the model parameters are retained for testing. The quantitative detection results between different advanced models are shown in Table 1, and the visual detection results are shown in Table 1. Figure 4 As shown in the figure, it should be noted that during the training of SRGM and FOD-Net, the resolution of the image input to the network is set to 608×352 pixels, the data enhancement method uses left-right flipping and scale scaling, and the batch size is set to 12. The hyperparameters used are consistent with the YOLOv5-L model. During the test phase, the batch size is set to 1.

[0064] Table 1

[0065]

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A traffic fixation target detection method based on saliency prior, characterized by: The following steps are involved: S1, establishing a gaze target detection dataset based on the driver's eye movements, wherein the gaze target detection dataset includes saliency information for attracting the driver's attention in a traffic scene, and object categories in a salient area that attracts the driver's attention; S2, builds a strong baseline detection module based on the YOLOv5-L model and the lightweight attention module CBAM; In S2, a basic module consisting of convolution (k=3, s=1), batch normalization and SILU activation is added to the first layer of the backbone, and a lightweight attention module CBAM is added to the first and second downsampling layers to establish a strong baseline detection module ODM to detect the specific location and category of the gaze target; S3, sharing the backbone and top-down paths of the strong baseline detection module established in S2, establishing a salient area guidance module for predicting the salient areas that attract the driver's attention in the traffic scene, and the salient areas are used to guide the strong baseline detection module to focus on the gaze target in the driver's attention area; adding two upsampling layer modules on the basis of the top-down path to construct a salient area guidance module SRGM for predicting the salient areas that attract the driver's attention in the traffic scene; S4, by simulating the selective attention of drivers in traffic scenes, proposes a saliency guidance strategy; the salient area predicted by SRGM in S3 is used as a saliency prior to guide the ODM established in S2 to focus on the gaze target in the driver's gaze area, and the predicted saliency map is fused with the shallow features of the backbone of ODM. The fusion expression is: In the formula, represents element-wise multiplication, represents element-by-element addition, is a learnable parameter used to weight the saliency map, represents the saliency map predicted by SRGM in S3, is the shallow feature extracted by ODM, and its width and height are The agreement, represents the fused features, Contains the global information of the traffic scene and the enhanced features of the gaze target; S5, based on the strong baseline detection module established in S2, the salient area guidance module constructed in S3, and the saliency guidance strategy proposed in S4, a fixation target detection model is established; S6, scaling the images of the training set to a given input resolution and inputting the images into the gaze target detection model constructed in S5, and training the gaze target detection model, thereby improving the detection performance of the gaze target detection model.

2. A method for detecting traffic fixation targets based on saliency priors as claimed in claim 1, characterized in that: In S1, when establishing a gaze target detection dataset based on the driver's eye movements, the following steps are included: Step a, using a driving recorder to collect traffic videos of vehicles in different weather conditions and driving scenarios; Step b, allowing the driver to watch the collected traffic video during the simulated driving task, and using an eye tracker to record the driver's eye movement information; Step c, mapping the recorded gaze points of all drivers onto the traffic video; Step d, marking the gaze target, and finally obtaining the driver's gaze target detection data set; Step e: Generate salient region masks based on the labeled fixation target detection training set.

3. A method for detecting traffic attention targets based on saliency priors as claimed in claim 2, characterized in that: In step d, when marking the gaze target, one frame is extracted from every five frames of the traffic video to mark the gaze target.

4. The method for detecting traffic fixation targets based on saliency prior according to claim 3, characterized in that: In S2, a basic module consisting of convolution (k=3, s=1), batch normalization and SILU activation is added to the first layer of the backbone, and a lightweight attention module CBAM is added to the first and second downsampling layers to establish a strong baseline detection module to detect the specific location and category of the gaze target; The strong baseline detection module is optimized using the loss function, as shown in the following formula (1): （1） In the formula, represents the CIOU loss, which is used to calculate the position deviation between the predicted target position and the marked target position; and is the binary cross entropy loss, It is used to calculate the difference between the predicted target category and the labeled target category; Used to calculate the confidence difference between the predicted target and the labeled target.

5. The method for detecting traffic fixation targets based on saliency prior according to claim 4, characterized in that: In S3, the composite loss function is used to optimize the salient region guidance module, as shown in the following formula (2): （2） In the formula, The salient area mask generated in claim 2, The scale is and Represent the width and height of the input image respectively, represents the saliency map predicted by the salient region guidance module, with its scale and The agreement, represents the binary cross entropy loss, which is used to optimize the salient region guidance module from a fine-grained pixel level. It optimizes the IOU loss of the salient region guidance module from the overall region level.

6. A method for detecting traffic fixation targets based on saliency priors as claimed in claim 5, characterized in that: In S6, the composite loss function II is used to optimize the fixation target detection model, as shown in the following formula (4): （4） In the formula, is the loss defined in formula (1), is the loss defined in equation (2), is the weight coefficient.

7. A method for detecting traffic fixation targets based on saliency priors as claimed in claim 6, characterized in that: In S6, the salient region mask generated in step e and formula (2) are used to pre-train the salient region guidance module. The weights obtained by pre-training are used to initialize the fixation target detection model. The salient region mask generated in step e and the annotated fixation target detection dataset and formula (4) are used to train the fixation target detection model. When the mAP value of the validation set reaches the maximum, the model parameters are retained for testing.

Images