Traffic video saliency prediction method and system based on space-time double-branch network feature fusion

Abstract

The invention discloses a traffic video saliency prediction method and system based on space-time double-branch network feature fusion. The method comprises the following steps: acquiring traffic video data to be predicted, selecting a continuous video sequence in a time branch network, selecting a single-frame video image in a space branch network, and respectively inputting the continuous video sequence and the single-frame video image into a trained significance prediction model; the method comprises the following steps: preliminarily encoding a video sequence by adopting an encoder, guiding continuous frame time sequence modeling through a saliency attention transmission mechanism, and then performing decoding operation through a deconvolution decoder to obtain a time saliency map; capturing saliency information of different receptive fields from a single-frame video image through a multi-resolution feature aggregation module which is subjected to joint sampling and layer-by-layer association to obtain a spatial saliency map; and finally, performing fusion decoding on the spatial-temporal characteristics, and calculating a final saliency map. The method can accurately predict the salient region of the traffic video in the driving scene, and has important significance for the development of the auxiliary driving technology.

Description

Technical Field

[0001] The invention relates to a traffic video saliency prediction method and system based on spatiotemporal dual-branch network feature fusion, belonging to the technical field of image communication methods. Background Art

[0002] The prediction of video salient regions is highly consistent with the human visual selection system and plays an important role in the field of perceptual video coding. In the video coding process, the information of salient prediction can effectively eliminate the redundant information of human vision. Traffic video saliency prediction uses the visual selective focusing mechanism of the human eye to quickly screen key salient regions such as vehicles, pedestrians and traffic lights, and further guide the high-quality encoding and fine processing of salient regions, while low-bitrate encoding and rough processing of non-salient regions are performed, or even ignored, thereby optimizing the coding efficiency and reducing the bit rate required for video coding to ensure the accurate transmission of key information, while reducing the bandwidth burden of video image communication. This process ensures that drivers can receive the most critical traffic information under limited network bandwidth. This not only improves communication efficiency, but also ensures the real-time and accuracy of information, providing strong support for the development of assisted driving and unmanned driving technology.

[0003] The research on saliency prediction of traffic videos has attracted widespread attention from scholars at home and abroad, and many important results have emerged. However, the current prediction models still have the following problems: First, the spatial features are not fully utilized and cannot capture key information in the scene, such as pedestrians and vehicles, which significantly affects the accuracy of the predicted attention area; second, since the prediction of driver attention in traffic videos involves a continuous process, it is necessary to track elements such as vehicles and pedestrians in continuous frames. There is a lack of effective guidance and compensation mechanism for the motion information between frames, and the motion information of the salient frames is ignored. It is impossible to accurately judge the dynamic changes of the surrounding environment, which affects the safety and reliability of driving. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a traffic video saliency prediction method and system based on spatiotemporal dual-branch network feature fusion.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion, comprising: S1: Obtain the traffic video data to be predicted, select a continuous video sequence in the time branch network, select a single frame video image in the space branch network, and input them into the trained traffic video saliency prediction model respectively; S2: After the video sequence is initially encoded by the encoder, the saliency attention transmission mechanism guides the temporal modeling of consecutive frames to enhance the saliency dependency between frames, and then the deconvolution decoder performs decoding operation to obtain the temporal saliency map; S3: A multi-resolution feature aggregation module is used to jointly sample and associate a single frame of video images layer by layer to capture the salient information of different receptive fields, so as to enhance the ability to express detailed features and obtain a spatial saliency map. S4: The temporal features and spatial features are fused and decoded by using a concatenation and convolution method, and the final traffic video saliency map is calculated to obtain the prediction result.

[0006] Furthermore, the training steps of the traffic video saliency prediction network include: Collect and preprocess training data into input video frame sequences and label ground truth saliency maps; Construct a traffic video saliency prediction network and set the initial weight value; Input the training set into the prediction network, train the network, and output the predicted saliency map; Use the loss function to calculate the difference between the predicted saliency map and the true saliency map and back propagate; After multiple iterations of training, when the loss function converges, the optimal parameters and structure are saved to obtain a trained traffic video saliency prediction network.

[0007] Furthermore, the saliency attention transmission mechanism is implemented by a Saliency Attention Transmission Module (SAT), which includes a spatiotemporal attention guided fusion submodule and a ConvGRU submodule for temporal modeling to utilize the saliency information of previous frames to enhance the prediction ability of the current frame.

[0008] The spatiotemporal attention guided fusion submodule first calculates the global information of the input feature X in the time dimension, generates attention weights through the fully connected layer, to enhance the model's attention to important time segments in the time series data, and then obtains the adjusted hidden state through multiplication. ; Then, and X are concatenated in the channel dimension and then convolved and Operations such as , etc. normalize it in spatial and temporal dimensions and generate a preliminary attention map. By introducing the spatiotemporal attention mechanism, the model can focus more on the key segments and spatial positions in the video frame, thereby improving the accuracy and efficiency of feature extraction.

[0009] Then, ConvGRU is applied to learn the coherent information before and after the video frames and capture the long-term dependencies of continuous saliency, thereby achieving accurate modeling and prediction of spatiotemporal data.

[0010] Furthermore, the multi-resolution feature aggregation module includes a joint sampling prior block and a layer-by-layer association aggregation block to achieve full interaction of information between multi-resolution features.

[0011] The joint sampling prior block is used to obtain spatial information of different resolutions and enhance the expressiveness of significant features. This module constructs four sets of feature groups with different channels. After each set is subjected to a 3×3 CBR (Conv+BN+ReLU) operation, it is upsampled layer by layer along a top-down path or downsampled layer by layer along a bottom-up path to generate four layers of feature maps with different resolutions. High-resolution images provide more detailed information, while low-resolution images provide more macroscopic structure and context information. By aggregating these spatial information of different resolutions, richer image features can be captured. Feature maps of the same layer have the same resolution. Feature maps of different groups of the same layer are spliced ​​to generate fused feature maps, which serve as prior information for subsequent feature extraction.

[0012] The layer-by-layer association aggregation block is used to aggregate the salient features of different receptive fields and further capture rich image features. This module first changes the number of channels of the fused feature map of each layer through a 1×1 CBR operation, so that it can be summed with the upsampled feature map of the adjacent layer, establish a top-down association path, aggregate the salient information of different receptive fields layer by layer, and realize the interaction of spatial information and semantic information between different layers and different resolutions. Finally, the final spatial salient map is obtained through the sigmoid function.

[0013] In a second aspect, the present invention provides a traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion, comprising: The spatiotemporal data input module obtains the traffic video data to be predicted, selects continuous video sequences in the time branch network, and selects single-frame video images in the space branch network, and inputs them into the traffic video saliency prediction network respectively.

[0014] The temporal feature extraction module uses an encoder to preliminarily encode the input video sequence, captures the dependencies between the previous and next video frames through a saliency attention transmission mechanism, and then performs decoding operations through a deconvolution decoder to extract salient temporal features and obtain a temporal saliency map.

[0015] The spatial feature extraction module obtains the significant information of different receptive fields through the multi-resolution feature aggregation module of joint sampling and layer-by-layer association of the input single-frame image, extracts significant spatial features, and obtains the spatial saliency map.

[0016] The fusion decoding module fuses and decodes the temporal features and spatial features by means of concatenation and convolution, calculates the final traffic video saliency map, and obtains the prediction result.

[0017] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention adopts a method combining a time branch network and a space branch network to train a traffic video saliency prediction network model; uses a saliency attention transmission mechanism to guide the temporal modeling of continuous frames, uses previous frames to predict the current frame, and enhances the saliency dependency between frames; uses a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to capture spatial saliency information of different receptive fields, thereby enhancing the feature expression capability; and further enriches the spatiotemporal features by fusion decoding, effectively improving the prediction accuracy and enhancing the robustness of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0021] Figure 1 It is an overall framework diagram of the traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion provided in the first embodiment of the present invention.

[0022] Figure 2 Schematic diagram of network details of the traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion provided in the first embodiment of the present invention.

[0023] Figure 3 Schematic diagram of the structure of the saliency attention transmission module provided in the first embodiment of the present invention.

[0024] Figure 4 It is a structural diagram of the joint sampling prior block provided in the first embodiment of the present invention.

[0025] Figure 5 It is a structural diagram of the layer-by-layer associated aggregation block provided in the first embodiment of the present invention.

[0026] Figure 6 It is a structural diagram of a traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0029] Embodiment 1 The embodiment of the present invention provides a traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion, comprising: S1: Obtain the traffic video data to be predicted, select a continuous video sequence in the time branch network, select a single frame video image in the space branch network, and input them into the trained traffic video saliency prediction model respectively.

[0030] S2: After the video sequence is initially encoded by the encoder, the saliency attention transfer mechanism is used to guide the temporal modeling of consecutive frames to enhance the saliency dependencies between frames, and then the deconvolution decoder is used to perform decoding operations to obtain a temporal saliency map.

[0031] S3: A single-frame video image is subjected to a multi-resolution feature aggregation module that is jointly sampled and associated layer by layer to capture the salient information of different receptive fields, so as to enhance the ability to express detailed features and obtain a spatial saliency map.

[0032] S4: The temporal features and spatial features are fused and decoded by using a concatenation and convolution method, and the final traffic video saliency map is calculated to obtain the prediction result.

[0033] The present invention provides a preferred embodiment to perform S1. Obtain the video data to be predicted, select a continuous video sequence in the time branch network, select a single frame video image in the space branch network, and input them into the trained traffic video saliency prediction network respectively. The training steps of the prediction network include: The first step is to collect and preprocess the training data, which is the original sequence of traffic videos and their corresponding label true value saliency maps. Two commonly used traffic video saliency prediction datasets are used: DADA-2000 and TDV. DADA-2000 is a large-scale driver attention prediction dataset in driving accident scenes, and the gaze data comes from the eye tracking data of 20 experienced drivers. Currently, 1018 videos are publicly available with a resolution of 1584×660. The dataset is divided into training set, validation set and test set in a ratio of about 3:1:1, that is, 598 videos (about 241,000 frames), 198 videos (about 64,000 frames), and 222 videos (about 70,000 frames) respectively. TDV is a traffic video dataset collected in normal driving scenes, and the gaze information of each video comes from the eye tracking data of 28 experienced drivers. It contains 16 traffic driving videos with a resolution of 1280×1024. The dataset is divided into training set, validation set and test set in the ratio of 10:2:4, that is, 49035 frames in the training stage, 6655 frames in the validation stage and 19135 frames in the test stage.

[0034] The second step is to build a traffic video saliency prediction network. The overall framework is as follows: Figure 1 The specific network details are as shown in Figure 2 As shown, set the initial weight value. Select ADAM optimizer, set the initial learning rate to 0.001, and the momentum and weight decay factors to 0.9 and 0.0001 respectively.

[0035] In the third step, the training set is input into the prediction network, the network is trained, and the predicted saliency map is output. In order to reduce the training time, the resolution of the input video frame is adjusted to 320×192, and random flipping is used for data augmentation. The training batch is set to 5, and each time a single video in the training set is randomly selected. T The continuous sequence of frames is used as input, and the setting is verified by experiments. T The model training platform is a Windows desktop computer with NVIDIA RTX3090Ti.

[0036] In the fourth step, the difference between the predicted saliency map and the true saliency map is calculated using the loss function and back-propagated. This embodiment uses a combination of multiple saliency prediction loss terms as the loss function, including KL divergence (Kullback-LeiblerDivergence) loss, normalized scan path saliency NSS (Normalized Scanpath Saliency) loss, linear correlation coefficient CC (Linear Correlation Coefficient) loss and binary cross entropy BCE (BinaryCrossEntropy) loss. AssumptionsS , B and G denote the predicted saliency map, the annotated binary attention map and the corresponding true saliency map respectively, and the calculation method of each loss term is as follows: , in represents the pixel position, represents the covariance, represents the average value, Represents the standard deviation. The final combined loss function expression is: , Among them, it is verified by experiments that the setting is 0.2, is 0.1.

[0037] The fifth step is to perform multiple iterations of training. When the loss function converges, the optimal parameters and structure are saved to obtain a trained traffic video saliency prediction network.

[0038] The present invention provides a preferred embodiment to perform S2. The video sequence is preliminarily encoded by an encoder to obtain a feature ,in T =6, C =16, H , W is the height and width of the input image. The encoder follows the typical architecture of the convolutional network, consisting of 5 convolutional blocks and 3 2×2 maximum pooling operations. Each convolutional block includes two 3 x 3 convolutions, two batch normalizations (BN) and two rectified linear units (ReLU). Then it is sent to the salient attention transfer mechanism to guide the temporal modeling, further capturing the dependencies between the previous and next frames of the video. The salient attention transfer mechanism is implemented by the salient attention transfer module, which contains the spatiotemporal attention guided fusion submodule and the ConvGRU submodule for temporal modeling. Its structure can be found in Figure 3 .

[0039] The spatiotemporal attention guided fusion submodule first calculates the input features X The global information in the time dimension generates attention weights through the fully connected layer to enhance the model's attention to important time segments in the time series data, and then multiplies them to obtain the adjusted hidden state ; Then, and X Splicing is performed on the channel dimension, and the attention map is generated by preliminary fusion through convolution, tanh function and convolution. , and its calculation formula is as follows: , Next, through The operator calculates the normalized spatial attention map, and its calculation formula is: , in Indicates location The attention map elements, Expressed as a natural constant e Finally, the spatial attention map is combined with the input X Multiply to get the adjusted features , the specific formula is as follows:

[0040] ConvGRU is used to capture the significant dependencies of consecutive frames. The calculation formula of this module is as follows: , in It is used to predict the enhanced features of the salient region of the current frame, and it is also the hidden state passed to the next frame. , ,and Represents the convolution weight.

[0041] The continuous video sequence is passed through N Layer loop calculations are further enhanced by experimental settings N The image is then decoded to 4 by a deconvolution decoder to restore it to the input video frame size, and finally a sigmoid operation is performed to obtain the temporal saliency map of the temporal branch network.

[0042] The present invention provides a preferred embodiment to perform S3. The multi-resolution feature aggregation module includes a joint sampling prior block and a layer-by-layer association aggregation block, and its structure is shown in Figure 2 .

[0043] The joint sampling prior block is used to obtain spatial information of different resolutions and enhance the expressiveness of salient features, see Figure 4 This block has four sets of feature groups with different channels, each set has four layers of feature maps with different resolutions. Each set of calculations starts with a 3×3 CBR operation, and performs downsampling with a step size of 2 along the bottom-up path or upsampling with a scale factor of 2 along the top-down path. Then, the feature maps of the same layer are spliced ​​to obtain four layers of fused features with different resolutions. , which includes rough semantics and fine details. Its expression formula is as follows: , in, represents the original input image features, Indicates Group The feature map of the layer, , Represents the combined operation of 3×3 convolution (Conv), batch normalization (BN), and rectified linear unit (ReLU).

[0044] The layer-by-layer association aggregation block is used to aggregate the salient features of different receptive fields to further capture rich image features. Figure 5 The block first performs a 1×1 CBR operation on the fusion feature maps of each layer to obtain Then the high-level fusion features are upsampled and summed with the adjacent low-level fusion features to obtain This operation is performed from the fourth layer to the first layer, establishing a top-down association path, aggregating the salient information of different receptive fields layer by layer, and enhancing the detail information while making the final aggregated features have the same resolution as the original input. Finally, a sigmoid operation is performed on the aggregated features to obtain the final spatial saliency map The specific formula is as follows: , in, For the sigmoid function operation, Represents the combined operation of 1×1 convolution (Conv), batch normalization (BN), and rectified linear unit (ReLU). It is the spatial saliency map finally output by the spatial branch network.

[0045] The present invention provides a preferred embodiment to perform S4. The temporal features and the spatial features are fused and decoded, the temporal saliency map and the spatial saliency map are first spliced ​​in the channel dimension, and then three 3×3 CBR operations and one 1×1 CBR operation are sequentially performed on the spliced ​​features, the size of the feature map remains unchanged, and finally the final predicted video saliency map is obtained through the sigmoid function.

[0046] In order to verify the effectiveness of the above embodiment 1, the method of the present invention is compared with other advanced methods on two data sets DADA-2000 and TDV, and four commonly used indicators are selected as the measurement: AUC-Judd (AUC-J), standardized scan path significance (NSS), linear correlation coefficient (CC) and Kullback-Leibler Divergence (KLD). The larger the values ​​of the first three indicators and the smaller the KLD indicator value, the better the model effect and the more accurate the predicted significant area. The experimental results are shown in Tables 1 and 2.

[0047] Table 1 Comparison of prediction accuracy on the DADA-2000 dataset

[0048] Table 2 Comparison of prediction accuracy on the TDV dataset

[0049] It can be seen from Table 1 and Table 2 that the method of this embodiment is superior to previous methods in many indicators on the DADA-2000 and TDV datasets, achieves good prediction accuracy, and can more accurately predict the driver's attention in traffic videos, which proves the effectiveness of this embodiment.

[0050] Embodiment 2 This embodiment provides a traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion, see Figure 6 ,include: The spatiotemporal data input module obtains the traffic video data to be predicted, selects continuous video sequences in the time branch network, and selects single-frame video images in the space branch network, and inputs them into the traffic video saliency prediction network respectively; The temporal feature extraction module uses an encoder to preliminarily encode the input video sequence, captures the dependencies between the previous and next video frames through a saliency attention transmission mechanism, and then performs decoding operations through a deconvolution decoder to extract salient temporal features and obtain a temporal saliency map. The spatial feature extraction module obtains the significant information of different receptive fields by jointly sampling and layer-by-layer correlating multi-resolution feature aggregation modules on the input single-frame image, extracts significant spatial features, and obtains a spatial saliency map. The fusion decoding module fuses and decodes the temporal features and spatial features by means of concatenation and convolution, calculates the final traffic video saliency map, and obtains the prediction result.

[0051] It should be noted that the above modules correspond to steps S1 to S4 in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0052] In further embodiments, there is also provided: An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0053] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0054] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0055] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0056] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0057] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0058] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion, characterized in that: The steps include: S1: Obtain the traffic video data to be predicted, select a continuous video sequence in the time branch network, select a single frame video image in the space branch network, and input them into the trained traffic video saliency prediction model respectively; S2: After the video sequence is initially encoded by the encoder, the salient attention transfer mechanism guides the temporal modeling of consecutive frames to enhance the salient dependencies between frames; this mechanism is implemented by the salient attention transfer module, which includes the spatiotemporal attention guidance fusion submodule and the ConvGRU submodule for temporal modeling; The spatiotemporal attention guided fusion submodule first calculates the input features X The global information in the time dimension generates attention weights through the fully connected layer to enhance the model's attention to important time segments in the time series data, and then multiplies them to obtain the adjusted hidden state ; Then, and X Splicing is performed on the channel dimension, and the attention map is generated by preliminary fusion through convolution, tanh function and convolution. , and its calculation formula is as follows: , Next, through The operator calculates the normalized spatial attention map, and its calculation formula is: , in Indicates location The attention map elements, Indicated by natural constant e Finally, the spatial attention map is combined with the input X Multiply to get the adjusted features , the specific formula is as follows: ， ConvGRU is used to capture the significant dependencies of consecutive frames. The calculation formula of this module is as follows: ， in It is used to predict the enhanced features of the salient region of the current frame, and it is also the hidden state passed to the next frame. , ,and represents the convolution weight; The continuous video sequence is further enhanced by multi-layer loop calculation, and then decoded by the deconvolution decoder to restore it to the input video frame size. Finally, the sigmoid operation is performed to obtain the temporal saliency map of the temporal branch network. S3: A single-frame video image is passed through a multi-resolution feature aggregation module to capture significant information of different receptive fields to enhance the ability to express detailed features; this module includes a joint sampling prior block and a layer-by-layer association aggregation block; The joint sampling prior block is used to obtain spatial information of different resolutions and enhance the expressiveness of significant features. This block constructs four sets of feature groups with different channels, each with four layers of feature maps with different resolutions. Each set of calculations starts with a 3×3 CBR operation, i.e., Conv+BN+ReLU, and performs downsampling with a step size of 2 along the bottom-up path or upsampling with a scale factor of 2 along the top-down path. Then, the feature maps of the same layer are spliced ​​to obtain four layers of fused features with different resolutions. , which includes rough semantics and fine details. Its expression formula is as follows: ， in, represents the original input feature map, Indicates Group The feature map of the layer, , Represents the combined operation of 3×3 convolution, batch normalization, and rectified linear unit; The layer-by-layer association aggregation block is used to aggregate the salient features of different receptive fields to further capture rich image features. The block first performs a 1×1 CBR operation on the fusion feature maps of each layer to obtain Then the high-level fusion features are upsampled and summed with the adjacent low-level fusion features to obtain ; This operation is performed from the fourth layer to the first layer in sequence, establishing a top-down association path, aggregating the salient information of different receptive fields layer by layer, while enhancing the detail information, making the final aggregated feature have the same resolution as the original input; finally, a sigmoid operation is performed on the aggregated feature to obtain the final spatial saliency map ; The specific formula is as follows: ， in, For the sigmoid function operation, represents the combined operation of 1×1 convolution, batch normalization, and rectified linear unit, The spatial saliency map obtained by outputting the spatial branch network; S4: The temporal features and spatial features are fused and decoded by using a concatenation and convolution method, and the final traffic video saliency map is calculated to obtain the prediction result.

2. The traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion as claimed in claim 1, characterized in that: The training steps of the trained traffic video saliency prediction network include: Collect and preprocess training data into input video frame sequences and label ground truth saliency maps; Construct a traffic video saliency prediction network and set the initial weight value; Input the training set into the prediction network, train the network, and output the predicted saliency map; Use the loss function to calculate the difference between the predicted saliency map and the true saliency map and back propagate; After multiple iterations of training, when the loss function converges, the optimal parameters and structure are saved to obtain a trained traffic video saliency prediction network.

3. The traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion as claimed in claim 1, characterized in that: The encoder is used to preliminarily encode the video sequence: the encoder follows the typical architecture of a convolutional network, consisting of 5 convolution blocks and 3 2×2 maximum pooling operations, each convolution block includes two 3 x 3 convolutions, two batch normalizations and two rectified linear units.

4. The traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion as claimed in claim 1, characterized in that: The temporal features and the spatial features are fused and decoded. First, the temporal saliency map and the spatial saliency map are spliced ​​in the channel dimension. Then, three 3×3 CBR operations and one 1×1 CBR operation are performed on the spliced ​​features in sequence. The size of the feature map remains unchanged. Finally, the final predicted video saliency map is obtained through the sigmoid function.

5. A traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion, characterized in that: include: The spatiotemporal data input module obtains the traffic video data to be predicted, selects continuous video sequences in the time branch network, and selects single-frame video images in the space branch network, and inputs them into the traffic video saliency prediction network respectively; The temporal feature extraction module uses an encoder to preliminarily encode the input video sequence, and then captures the dependencies between the previous and next video frames through a saliency attention transmission mechanism. Then, a deconvolution decoder performs decoding operations to extract saliency temporal features and obtain a temporal saliency map. The spatial feature extraction module obtains the significant information of different receptive fields by jointly sampling and layer-by-layer correlating multi-resolution feature aggregation modules on the input single-frame image, extracts significant spatial features, and obtains a spatial saliency map. The fusion decoding module fuses and decodes the temporal features and spatial features by means of concatenation and convolution, calculates the final traffic video saliency map, and obtains the prediction result.

6. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 4 is completed.

7. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 4.

Images