Traffic video gaze target detection method based on bottom-up and top-down approaches

Abstract

The present invention discloses a method for detecting gaze targets in traffic videos based on bottom-up and top-down attention approaches, which is applied to the field of computer vision technology. Existing methods for detecting targets that attract a driver's attention in traffic scenes focus on all targets in the scene and lack effective utilization of the driver's experience. Therefore, directly applying the existing methods to detect gaze targets in traffic videos may cause problems of false detection and missed detection. The present invention is inspired by the driver's combined top-down and bottom-up attention, focuses on detecting gaze targets in traffic videos that are closely related to driving tasks, simulates the driver's perception of traffic scenes by combining top-down and bottom-up attention, and uses low-level saliency and high-level global information closely related to the driving task decoded from a reference frame as a top-down prior to modulate the model's encoding process for the current frame, thereby improving the model's detection performance for video gaze targets.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a target detection technology that attracts the attention of drivers in traffic scenes. Background Art

[0002] Traffic scene perception has a significant impact on driving safety. Inexperienced or distracted drivers often fail to allocate sufficient attention to objects closely related to the driving task, creating potential road hazards. In contrast, experienced drivers, guided by visual selective attention, pay close attention to objects highly relevant to the driving task, thereby achieving safe driving. Therefore, timely and accurate detection of objects closely related to the driving task is crucial for driving safety.

[0003] To detect targets that attract the driver's attention in traffic scenes, existing methods use salient regions as saliency priors to guide the model to selectively focus on traffic gaze targets, thereby improving the model's detection performance. However, this method focuses on static image detection and lacks the use of information between consecutive frames, making it insufficient for traffic video detection. Existing video detection methods focus on all targets in the scene and lack effective utilization of driver experience. Therefore, directly applying existing methods to traffic video gaze target detection can result in false detections and missed detections. In view of this, there is an urgent need for a method that can effectively utilize information between consecutive frames and simulate the driver's perception of traffic scenes, thereby constructing an effective method for detecting traffic video gaze targets. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a traffic video gaze target detection method based on bottom-up and top-down attention, which can simulate the driver's perception of traffic scenes by combining top-down and bottom-up attention, and is used to detect targets that attract the driver's attention in traffic videos. It improves the model's detection ability for targets closely related to driving tasks, and shows potential application or reference value in intelligent driving or assisted driving safety.

[0005] The technical solution adopted by the present invention is: a traffic video gaze target detection method based on bottom-up and top-down methods, comprising:

[0006] S1. Establishing a dataset for traffic video gaze target detection based on a traffic visual saliency prediction database; including: labeling the targets that attract the driver's attention by referring to the driver's gaze areas recorded in the traffic visual saliency prediction database; generating salient region masks based on the labeled traffic video gaze target detection training set;

[0007] S2. Establish a traffic video gaze target detection model VFOD-Net, which includes a top-down modulation module TDMM and a gaze target detection module FODM.

[0008] The top-down modulation module TDMM decodes the shallow saliency and deep global information closely related to the driving task from the reference frame;

[0009] The shallow saliency and deep global information are used as top-down priors to modulate the encoding process of the fixation object detection module FODM for the current frame;

[0010] The fixation object detection module (FODM) encodes the features of the current frame in a bottom-up manner to characterize the specific location and category of the fixation object in the traffic video.

[0011] S3. Use the salient region mask to train the top-down modulation module TDMM, and initialize the traffic video gaze target detection model VFOD-Net according to the trained weights;

[0012] The video gaze object detection dataset with salient region masks and annotations is used to train the initialized traffic video gaze object detection model VFOD-Net;

[0013] S4. Use the trained traffic video gaze target detection model VFOD-Net to perform target detection on each frame of the predicted traffic video.

[0014] Beneficial Effects of the Invention: Inspired by the driver's combined top-down and bottom-up attention, this invention focuses on detecting gaze targets in traffic videos that are closely related to the driving task. The proposed traffic video gaze target detection dataset provides a benchmark for studying traffic scene perception from the driver's perspective. Furthermore, the proposed method simulates the driver's perception of the traffic scene using a combination of top-down and bottom-up attention. By using low-level saliency and high-level global information closely related to the driving task, decoded from the reference frame, as top-down priors, it modulates the model's encoding process for the current frame, thereby improving the model's performance in detecting gaze targets in videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an example of a video gaze target detection dataset constructed in an embodiment of the present invention;

[0016] Figure 2 This is a structural diagram of the video gaze target detection model VFOD-Net provided in an embodiment of the present invention;

[0017] Figure 3 It is a global priori guidance submodule proposed in an embodiment of the present invention;

[0018] Figure 4 This is a detection effect diagram of an embodiment of the present invention;

[0019] Among them, (a) is the real data, (b) is the detection result of YOLOV-X, (c) is the detection result of DINO-4scale, and (d) is the detection result of VFOD-Net of the present invention. DETAILED DESCRIPTION

[0020] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0021] The implementation process of one embodiment of the present invention includes the following steps:

[0022] S1. In order to carry out the research on fixation target detection in traffic videos, a dataset suitable for fixation target detection in traffic videos was established based on the traffic visual saliency prediction database. The traffic visual saliency prediction database is a known technology. For details, please refer to: "T. Deng, H. Yan, L. Qin, T. Ngo, and B. Manjunath, "How do drivers allocate their potential attention? driving fixation prediction via convolutional neural networks," IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 5, 2020.", "H. Tian, ​​T. Deng, and H. Yan, "Driving as well as on a sunny day? predicting driver's fixation in rainy weather conditions via a dual-branch visual model," IEEE / CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1335-1338, 2022."

[0023] a. Referring to the driver's gaze area recorded in the traffic visual saliency prediction database, the targets that attract the driver's attention are marked. The marking criteria used are: ① By referring to F t 11 consecutive frames before and after the frame (F t-5 ~F t+5 ) is the driver’s gaze area. If the driver looks at the same target in at least 5 frames, then F tThe target in the frame is labeled as a traffic fixation target. ② Considering that drivers may be affected by working memory or sensory memory, if the targets that attracted the driver's attention within 60 consecutive frames marked according to rule ① do not disappear, then these targets are also labeled as traffic fixation targets. Finally, a fixation target detection dataset ET-VFOD containing 32 videos was created in the form of VOC-formatted XML files. This dataset contains five target categories: traffic signs, people, cars, two-wheeled vehicles, and traffic lights. Twenty-two videos totaling 101,494 frames were used for training, four videos totaling 16,490 frames were used for validation, and six videos totaling 27,280 frames were used for testing.

[0024] b. To optimize the top-down modulation module, a salient region mask is generated based on the labeled training set of traffic video gaze target detection. Specifically, the value of the area inside the bounding box of the gaze target in each frame of the training set is set to 1, and the value of the area outside the bounding box is set to 0. An example of a video gaze target detection dataset and the corresponding salient region mask sample is shown in the figure. Figure 1 shown.

[0025] S2, in order to reduce model parameters, improve detection speed and obtain high detection accuracy, compared with the ODM module in FOD-Net, such as Figure 2 As shown on the right side of the dotted line in , the number of channels per layer in the proposed fixation object detection module FODM is reduced by 6.25%. Figure 2 The number of CSP modules in FODM is reduced by one third. In addition, during the encoding phase (as shown in the dashed box in Figure 2 The basic module CBS constructed by convolution operation, batch normalization and SILU activation function and the CSP module corresponding to 32 times downsampling rate are omitted. In terms of detection head configuration, the detection head H3 corresponding to the 16 times downsampling rate feature map is used. The proposed FODM performs a bottom-up analysis on the current frame F. t The features of are encoded, and the loss function is shown in the following formula (1):

[0026]

[0027] Where, and It is essentially a binary cross entropy loss, which is used to calculate the confidence and classification error of the predicted gaze target respectively. is the CIOU loss, which is used to regress the predicted fixation target bounding box position. In this case, α1 and α2 are set to 0.05 and 0.5 respectively.

[0028] S3 uses the same encoding and decoding network structure as the gaze target detection module established in S2 (such as Figure 2As shown in the solid and dotted boxes in the figure), a top-down modulation module TDMM is established, as shown in Figure 2 As shown on the left side of the dotted line in . In the encoding path of TDMM, CBS module and CSP module corresponding to 32 times downsampling rate are used to encode richer deep features. In the decoding path, two additional upsampling layers are introduced to decode more fine-grained shallow saliency. In this case, three consecutive frames (F t-2 、F t-1 and F t , i.e. the current frame and the two frames before it) are used as reference frames to concatenate the encoded features to predict F t The loss function for the salient area in the frame that attracts the driver's attention is as follows:

[0029]

[0030] Where S t is the salient region mask corresponding to the t-th frame generated in S1, represents the shallow saliency of two different sizes decoded by the top-down modulation module for the t-th frame, and their sizes are Where h and w represent the height and width of the input image respectively. represents the binary cross entropy loss, Denotes the IOU loss, which is used to optimize the top-down modulation module. In this case, β is set to 0.2.

[0031] Inspired by the driver's ability to combine top-down and bottom-up attention, S4 proposes a top-down modulation strategy. First, the shallow saliency closely related to the driving task decoded by TDMM in S3 is used as a top-down prior to be fused with the shallow features encoded by the fixation object detection module FODM established in S2, as shown in the following formula (3):

[0032]

[0033] Where, represents element-wise addition. ⊙ represents element-wise multiplication. μ is a learnable parameter used to weight the saliency map. Represents the shallow saliency of two different sizes decoded by TDMM, i.e., i∈{1,2}; their sizes are Where h and w represent the height and width of the input image respectively. i is the shallow feature of FODM coding, its scale and RF i represents the fused features, which are further encoded by deeper layers of FODM to enhance the module’s feature representation of the gaze target.

[0034] Then, the deep global information closely related to the driving task decoded by TDMM in S3 is used as a top-down prior and fused with the deep features encoded by FODM established in S2, as shown in Figure 3 As shown, the following steps are included:

[0035] a. Based on the deep global information decoded by TDMM and the deep features encoded by FODM, the query Q, key K and value V are generated, and the calculation is shown in the following formula (4):

[0036]

[0037] Where, is a mathematical symbol used to represent real numbers. Pro represents a linear transformation operation. Rep represents the reconstruction of the scale of a tensor. Represents the deep global information of TDMM decoding. Represents the deep features of FODM encoding.

[0038] b. Calculate the cross attention between query, key and value as shown in Equation (5):

[0039]

[0040] Where, Represents matrix multiplication. is the scaling factor. T represents the transpose operation. Represents the weighted value of the deep global information decoded by TDMM, where the weight is determined by computing the similarity between the query and the key using the softmax function.

[0041] c, the deep global information of TDMM decoding is used as a top-down prior to modulate the FODM encoding process of the current frame, which is calculated as follows:

[0042]

[0043] Where, Represents element-wise addition. Indicates channel splicing. The feature is modulated by the top-down prior. It is input into the subsequent deep encoding module of FODM after reducing the number of channels through 1×1 convolution. Because computing cross-attention requires a lot of computing resources, this case uses channel splicing instead of cross-attention to adjust the third CBS encoding module of FODM.

[0044] S5. Based on the gaze target detection module FODM established in S2, the top-down modulation module TDMM constructed in S3 and the top-down modulation strategy proposed in S4, a traffic video gaze target detection model VFOD-Net is established to detect gaze targets in traffic videos, such as Figure 2 shown.

[0045] In order to optimize VFOD-Net, a composite loss function is established as follows (7):

[0046]

[0047] Where, is the loss for gaze target detection defined in formula (1), is the loss defined in Equation (2) for decoding salient regions. γ is a weighting coefficient, which is set to 0.1 in this example.

[0048] S6: To improve VFOD-Net's representation of traffic gaze targets, we first pre-trained the TDMM using the salient region mask generated in S1 and formula (2) in S3. The weights obtained after 50 epochs of training were used to initialize VFOD-Net. VFOD-Net was then trained using the salient region mask generated in S1 and the annotated video gaze target detection dataset, along with formula (7) in S5. When the mAP value on the validation set is maximized, the model parameters are retained for testing.

[0049] Those skilled in the art should note that when the trained VFOD-Net is actually used to perform target detection on traffic videos to be predicted, since three frames need to be input into TDMM, the input corresponding to the first frame prediction in TDMM is three identical first frames; when predicting the second frame, the input into TDMM is the first frame and two second frames; starting from the third frame prediction, the input into TDMM is the current frame and the two consecutive frames before it.

[0050] The quantitative detection results between different advanced models are shown in Table 1, and the visual detection results are shown in Figure 4As shown in the figure, compared to static image detection models such as FOD-Net, the VFOD-Net proposed in this paper leverages information between consecutive frames and deep global information priors, significantly improving detection accuracy. Compared to video object detection models such as YOLOV-X, the VFOD-Net proposed in this paper leverages the driver's experience, combining top-down and bottom-up attention to encode the current frame, significantly reducing model parameters while achieving higher detection accuracy. The excellent performance of VFOD-Net demonstrates its potential application or reference value in improving the safety of intelligent driving or assisted driving. It should be noted that during the TDMM and VFOD-Net training process, the input network image resolution was set to 608×352 pixels, the data augmentation method used left-right flipping and scaling, and the batch size was set to 8. The hyperparameters used were consistent with the FOD-Net model. During the validation and testing phases, the input image resolution was set to 608×352 pixels, the batch size was set to 1, and no data augmentation method was used.

[0051] Table 1 Quantitative detection results between different advanced models

[0052]

[0053] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A traffic video gaze target detection method based on bottom-up and top-down approaches, characterized in that: include: S1. Establishing a dataset for traffic video gaze target detection based on a traffic visual saliency prediction database; including: labeling the targets that attract the driver's attention by referring to the driver's gaze areas recorded in the traffic visual saliency prediction database; generating salient region masks based on the labeled traffic video gaze target detection training set; S2. Establish a traffic video gaze target detection model VFOD-Net, which includes a top-down modulation module TDMM and a gaze target detection module FODM. The top-down modulation module TDMM decodes the shallow saliency and deep global information closely related to the driving task from the reference frame; The shallow saliency and deep global information are used as top-down priors to modulate the encoding process of the fixation object detection module FODM for the current frame; The fixation object detection module (FODM) encodes the features of the current frame in a bottom-up manner to characterize the specific location and category of the fixation object in the traffic video. S3. Use the salient region mask to train the top-down modulation module TDMM, and initialize the traffic video gaze target detection model VFOD-Net according to the trained weights; The video gaze object detection dataset with salient region masks and annotations is used to train the initialized traffic video gaze object detection model VFOD-Net; S4. Use the trained traffic video gaze target detection model VFOD-Net to perform target detection on each frame of the predicted traffic video.

2. The method for detecting gaze targets in traffic videos based on bottom-up and top-down approaches according to claim 1, characterized in that: In step S1, the target that attracts the driver's attention is marked. The specific process is as follows: A1. By referring to F t 11 consecutive frames before and after the frame, namely F t-5 ~F t+5 The driver's gaze area in the frame. If the driver looks at the same target in at least 5 frames, then F t This target in the frame is labeled as a preliminary traffic fixation target; A2. If the preliminary traffic attention targets marked in step A1 do not disappear within 60 consecutive frames, these targets are marked as final traffic attention targets.

3. The method for detecting gaze targets in traffic videos based on bottom-up and top-down approaches according to claim 2, characterized in that: The top-down modulation module TDMM includes an encoding path and a decoding path; the encoding path includes six encoding layers, of which the first encoding layer is implemented using a CBS module, and the second to sixth encoding layers are implemented using CBS+CSP modules; the decoding path includes six decoding layers, of which the first decoding layer is implemented using a CBS module, the second to fifth decoding layers are implemented using CBS+CSP modules, and the sixth decoding layer is implemented using an SPPF module; The output of each encoding layer in the encoding path is processed by channel concatenation and convolution and then used as the input of the corresponding decoding layer in the decoding path.

4. The method for detecting gaze targets in traffic videos based on bottom-up and top-down approaches according to claim 3, wherein: The fixation object detection module (FODM) consists of an encoding part, a feature fusion part, and a detection head. The encoding part consists of five encoding layers. The first encoding layer is implemented using a CBS module, and the second to fifth encoding layers are implemented using CBS+CSP modules. The feature fusion part is implemented using four CBS+CSP modules and one SPPF module; the four CBS+CSP modules are sequentially recorded as the first CBS+CSP module, the second CBS+CSP module, the third CBS+CSP module, and the fourth CBS+CSP module; The detection head part includes three detection heads, which are marked as detection head H2, detection head H3, and detection head H4; The outputs of the third to fifth coding layers in the coding part serve as the inputs of the first CBS+CSP module, the second CBS+CSP module, and the SPPF module in the feature fusion part respectively; the output of the first CBS+CSP module in the feature fusion part serves as the input of the detection head H2, and the output of the first CBS+CSP module in the feature fusion part also serves as the input of the third CBS+CSP module; the output of the second CBS+CSP module serves as the input of the first CBS+CSP module, and the output of the second CBS+CSP module also serves as the input of the third CBS+CSP module; the output of the third CBS+CSP module serves as the input of the detection head H3, and the output of the third CBS+CSP module also serves as the input of the fourth CBS+CSP module; the output of the SPPF module serves as the input of the second CBS+CSP module, and the output of the SPPF module also serves as the input of the fourth CBS+CSP module; the output of the fourth CBS+CSP module serves as the input of the detection head H4.

5. The method for detecting gaze targets in traffic videos based on bottom-up and top-down approaches according to claim 4, characterized in that: The shallow saliency and deep global information are used as top-down priors to modulate the encoding process of the fixation object detection module FODM for the current frame; specifically: First, the shallow saliency closely related to the driving task decoded by the top-down modulation module TDMM is used as the top-down prior and fused with the shallow features encoded by the fixation target detection module FODM. ​​The calculation is as follows: Where, represents element-by-element addition, ⊙ represents element-by-element multiplication, μ is a learnable parameter, represents the shallow saliency of two different sizes decoded by the top-down modulation module TDMM, i.e., i∈{1,2}; fea i is the shallow feature encoded by the fixation object detection module FODM, and its scale and Consistency; RF i Represents the fused features; Then, the deep global information closely related to the driving task decoded by the top-down modulation module TDMM is used as the top-down prior and fused with the deep features encoded by the fixation object detection module FODM.

6. The method for detecting gaze targets in traffic videos based on bottom-up and top-down approaches according to claim 5, characterized in that: The deep global information decoded by the top-down modulation module TDMM and closely related to the driving task is used as the top-down prior and the deep features encoded by the fixation target detection module FODM for fusion, including the following steps: a. Based on the deep global information decoded by the top-down modulation module TDMM and the deep features encoded by the fixation object detection module FODM, the query Q, key K and value V are generated. The calculation is shown in the following formula: Where Pro represents the linear transformation operation; Repo represents the reconstruction of the scale of the tensor; f dec represents the deep global information decoded by the top-down modulation module TDMM; f enc Represents the deep features encoded by the fixation object detection module FODM; b. Calculate the cross attention between query Q, key K and value V as shown below: Where, represents matrix multiplication, is the scaling factor, T represents the transpose operation, and V wet Represents the weighted value of the deep global information decoded by the top-down modulation module TDMM; c. The deep global information decoded by the top-down modulation module TDMM is used as a top-down prior to modulate the encoding process of the current frame by the fixation target detection module FODM. ​​The calculation is shown in the following formula: Where, represents element-by-element addition, Indicates channel splicing, F mod It is the feature after top-down prior modulation.

7. The method for detecting gaze targets in traffic videos based on bottom-up and top-down approaches according to claim 6, characterized in that: Channel splicing is used instead of cross attention to regulate the third encoding layer in the FODM encoding part of the fixation object detection module.

8. The method for detecting gaze targets in traffic videos based on bottom-up and top-down approaches according to claim 7, characterized in that: The loss function used in the training of the traffic video gaze target detection model VFOD-Net is expressed as: in, is the loss for fixation target detection, and is the binary cross entropy loss, which is used to calculate the confidence and classification error of the predicted gaze target respectively; is the CIOU loss, used to regress the predicted fixation target bounding box position; is the loss for decoding salient regions, α1, α2, β, and γ are weighting coefficients.

Images