Traffic Video Salience Prediction Method and System Based on Feature Fusion of Spatiotemporal Dual-Branch Networks

Abstract

The present invention discloses a traffic video saliency prediction method and system based on spatiotemporal dual-branch network feature fusion. The method comprises: obtaining traffic video data to be predicted, selecting a continuous video sequence in a temporal branch network, and selecting a single-frame video image in a spatial branch network, and inputting each into a trained saliency prediction model; using an encoder to preliminarily encode the video sequence, guiding the continuous frame temporal modeling through a saliency attention transmission mechanism, and then performing decoding operations through a deconvolution decoder to obtain a temporal saliency map; subjecting the single-frame video image to a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to capture salient information of different receptive fields, thereby obtaining a spatial saliency map; and finally fusing and decoding the spatiotemporal features to calculate the final saliency map. The method can accurately predict salient areas in traffic videos of driving scenarios, which is of great significance to the development of assisted driving technology.

Description

Technical Field

[0001] The present invention relates to a traffic video saliency prediction method and system based on spatiotemporal dual-branch network feature fusion, belonging to the technical field of image communication methods. Background Art

[0002] Video saliency region prediction is highly aligned with the human visual selection system and plays a crucial role in perceptual video coding. During video coding, saliency prediction can effectively eliminate redundant information from the human eye. Traffic video saliency prediction leverages the human eye's selective focusing mechanism to quickly filter out key salient regions, such as vehicles, pedestrians, and traffic lights. This allows for high-quality encoding and detailed processing of salient regions, while less salient regions are encoded at a low bitrate and processed coarsely, or even ignored. This optimizes coding efficiency, reduces the required bitrate for video encoding, ensures accurate transmission of critical information, and reduces the bandwidth burden on video image communication. This process ensures that drivers receive the most critical traffic information even with limited network bandwidth. This not only improves communication efficiency but also ensures the real-time and accuracy of information, providing strong support for the development of assisted and autonomous driving technologies.

[0003] Research on saliency prediction in traffic videos has attracted widespread attention from scholars at home and abroad, and many important results have emerged. However, the current prediction models still have the following problems: First, spatial features are insufficiently utilized and key information in the scene, such as pedestrians and vehicles, cannot be captured, which significantly affects the accuracy of predicted attention areas; second, because driver attention prediction in traffic videos involves a continuous process, it is necessary to track elements such as vehicles and pedestrians in consecutive frames. There is a lack of effective guidance and compensation mechanism for motion information between frames, and the motion information of salient frames is ignored, making it impossible to accurately judge the dynamic changes of the surrounding environment, thus affecting driving safety and reliability. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a traffic video saliency prediction method and system based on spatiotemporal dual-branch network feature fusion.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion, comprising:

[0007] S1: Obtain the traffic video data to be predicted, select a continuous video sequence in the time branch network, and select a single frame video image in the spatial branch network, and input them into the trained traffic video saliency prediction model respectively;

[0008] S2: After the video sequence is initially encoded by the encoder, the saliency attention transfer mechanism guides the temporal modeling of consecutive frames to enhance the saliency dependencies between frames. The deconvolution decoder then performs decoding operations to obtain a temporal saliency map.

[0009] S3: A single-frame video image is subjected to a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to capture salient information in different receptive fields, thereby enhancing the ability to express detailed features and obtaining a spatial saliency map.

[0010] S4: Temporal and spatial features are fused and decoded using concatenation and convolution, and the final traffic video saliency map is calculated to obtain the prediction result.

[0011] Furthermore, the training steps of the traffic video saliency prediction network include:

[0012] Collect and preprocess training data into input video frame sequences and label ground truth saliency maps;

[0013] Build a traffic video saliency prediction network and set the initial weight value;

[0014] Input the training set into the prediction network, train the network, and output the predicted saliency map;

[0015] Use the loss function to calculate the difference between the predicted saliency map and the true saliency map and backpropagate;

[0016] After multiple iterations of training, when the loss function converges, the optimal parameters and structure are saved to obtain the trained traffic video saliency prediction network.

[0017] Furthermore, the saliency attention transmission mechanism is implemented by a Saliency Attention Transmission Module (SAT), which includes a spatiotemporal attention guided fusion submodule and a ConvGRU submodule for temporal modeling to utilize the saliency information of previous frames to enhance the prediction ability of the current frame.

[0018] The spatiotemporal attention guided fusion submodule first calculates the global information of the input feature X in the time dimension, generates attention weights through the fully connected layer, and enhances the model's attention to important time segments in the time series data. The adjusted hidden state is then obtained by multiplication. ; Then, and X are concatenated in the channel dimension and then convolved and Operations such as

[15] normalize it in spatial and temporal dimensions and generate a preliminary attention map. By introducing the spatiotemporal attention mechanism, the model can focus more on key segments and spatial locations in the video frame, thereby improving the accuracy and efficiency of feature extraction.

[0019] Then, ConvGRU is applied to learn the coherent information before and after the video frames, capturing the long-term dependencies of continuous saliency, thereby achieving accurate modeling and prediction of spatiotemporal data.

[0020] Furthermore, the multi-resolution feature aggregation module includes a joint sampling prior block and a layer-by-layer association aggregation block to achieve full interaction of information between multi-resolution features.

[0021] The Joint Sampling Prior block is used to obtain spatial information at different resolutions, enhancing the expressiveness of salient features. This module constructs four sets of feature maps with different channels. Each set undergoes a 3×3 CBR (Conv+BN+ReLU) operation, and is then upsampled layer by layer along a top-down path or downsampled layer by layer along a bottom-up path to generate four layers of feature maps at different resolutions. High-resolution images provide more detailed information, while low-resolution images provide more macroscopic structure and context. By aggregating this spatial information at different resolutions, richer image features can be captured. Feature maps at the same layer have the same resolution. Feature maps from different sets of the same layer are concatenated to generate a fused feature map, which serves as prior information for subsequent feature extraction.

[0022] The layer-by-layer association aggregation block aggregates salient features from different receptive fields, further capturing rich image features. This module first uses a 1×1 CBR operation to change the number of channels in each layer's fused feature map, enabling it to be summed with the upsampled feature maps from adjacent layers. This establishes a top-down association path, aggregating salient information from different receptive fields layer by layer, enabling the interaction of spatial and semantic information across different layers and resolutions. Finally, a sigmoid function is applied to the final spatial saliency map.

[0023] In a second aspect, the present invention provides a traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion, comprising:

[0024] The spatiotemporal data input module obtains the traffic video data to be predicted, selects continuous video sequences in the time branch network, and selects single-frame video images in the space branch network, and inputs them into the traffic video saliency prediction network respectively.

[0025] The temporal feature extraction module uses an encoder to preliminarily encode the input video sequence, then captures the dependencies between the previous and next video frames through a saliency attention transmission mechanism, and then performs decoding operations through a deconvolution decoder to extract salient temporal features and obtain a temporal saliency map.

[0026] The spatial feature extraction module processes the input single-frame image through a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to obtain significant information of different receptive fields, extract significant spatial features, and obtain a spatial saliency map.

[0027] The fusion decoding module fuses and decodes temporal and spatial features using a concatenation and convolution approach, calculates the final traffic video saliency map, and obtains the prediction result.

[0028] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention adopts a method combining temporal branching network and spatial branching network to train the traffic video saliency prediction network model; uses the saliency attention transmission mechanism to guide the temporal modeling of continuous frames, uses the previous frame to predict the current frame, and enhances the saliency dependency between frames; uses a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to capture spatial saliency information of different receptive fields, thereby enhancing the feature expression capability; and fusion decoding further enriches the spatiotemporal features, effectively improving the prediction accuracy and enhancing the robustness of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Figure 1 This is an overall framework diagram of the traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion provided in Example 1 of the present invention.

[0034] Figure 2 3 is a schematic diagram of network details of the traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion provided in the first embodiment of the present invention.

[0035] Figure 3 3 is a schematic diagram of the structure of the saliency attention transmission module provided in the first embodiment of the present invention.

[0036] Figure 4 2 is a schematic diagram of the structure of the joint sampling prior block provided in the first embodiment of the present invention.

[0037] Figure 5It is a structural diagram of the layer-by-layer association aggregation block provided in the first embodiment of the present invention.

[0038] Figure 6 3 is a structural diagram of a traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0041] Example 1

[0042] The embodiment of the present invention provides a traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion, comprising:

[0043] S1: Obtain the traffic video data to be predicted, select a continuous video sequence in the time branch network, and select a single frame video image in the spatial branch network, and input them into the trained traffic video saliency prediction model respectively.

[0044] S2: After the video sequence is preliminarily encoded by the encoder, the saliency attention transfer mechanism is used to guide the temporal modeling of consecutive frames to enhance the saliency dependencies between frames. The deconvolution decoder is then used for decoding to obtain a temporal saliency map.

[0045] S3: A single-frame video image is subjected to a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to capture salient information of different receptive fields, thereby enhancing the ability to express detailed features and obtaining a spatial saliency map.

[0046] S4: Temporal and spatial features are fused and decoded using concatenation and convolution, and the final traffic video saliency map is calculated to obtain the prediction result.

[0047] The present invention provides a preferred embodiment for performing S1. The video data to be predicted is obtained, a continuous video sequence is selected in the temporal branch network, and a single frame video image is selected in the spatial branch network, and the two are respectively input into a trained traffic video saliency prediction network. The training steps of the prediction network include:

[0048] The first step is to collect and preprocess training data. The training data consists of raw traffic video sequences and their corresponding ground-truth saliency maps. Two commonly used traffic video saliency prediction datasets are used: DADA-2000 and TDV. DADA-2000 is a large-scale dataset for predicting driver attention in accident scenarios. The gaze data is derived from eye-tracking data of 20 experienced drivers. Currently, 1,018 videos are publicly available with a resolution of 1,584 × 660. The dataset is divided into training, validation, and test sets in a ratio of approximately 3:1:1, consisting of 598 videos (approximately 241,000 frames), 198 videos (approximately 64,000 frames), and 222 videos (approximately 70,000 frames), respectively. TDV is a traffic video dataset collected in normal driving scenarios. The gaze information for each video is derived from eye-tracking data of 28 experienced drivers. It contains 16 traffic driving videos with a resolution of 1,280 × 1,024. The dataset is divided into training set, validation set and test set in the ratio of 10:2:4, that is, 49035 frames in the training stage, 6655 frames in the validation stage and 19135 frames in the test stage.

[0049] The second step is to build a traffic video saliency prediction network. The overall framework is as follows: Figure 1 The specific network details are as shown in Figure 2 As shown, set the initial weight values. Select ADAM optimizer, set the initial learning rate to 0.001, and the momentum and weight decay factors to 0.9 and 0.0001 respectively.

[0050] In the third step, the training set is input into the prediction network, the network is trained, and the predicted saliency map is output. In order to reduce the training time, the resolution of the input video frame is adjusted to 320×192, and random flipping is used for data augmentation. The training batch is set to 5, and each time a single video in the training set is randomly selected. T A continuous sequence of frames is used as input, and the setting is verified by experiments. T The model training platform is a Windows desktop computer with NVIDIA RTX3090Ti.

[0051] In the fourth step, the difference between the predicted saliency map and the true saliency map is calculated using the loss function and back-propagated. This embodiment uses a combination of multiple saliency prediction loss terms as the loss function, including KL divergence (Kullback-LeiblerDivergence) loss, normalized scanpath saliency (NSS) loss, linear correlation coefficient (CC) loss, and binary cross entropy (BCE) loss. Assume S 、 B and G Denote the predicted saliency map, the annotated binary attention map, and the corresponding ground-truth saliency map, respectively. The calculation method of each loss term is as follows:

[0052] ,

[0053] in represents the pixel position, represents the covariance, represents the average value, Represents the standard deviation. The final combined loss function expression is:

[0054] ,

[0055] Among them, it is verified by experiments that is 0.2, is 0.1.

[0056] The fifth step is to iterate the training multiple times. When the loss function converges, the optimal parameters and structure are saved to obtain the trained traffic video saliency prediction network.

[0057] The present invention provides a preferred embodiment to perform S2. The video sequence is preliminarily encoded by an encoder to obtain a feature ,in T =6, C =16, H , W The encoder follows the typical architecture of convolutional networks and consists of 5 convolution blocks and 3 2×2 maximum pooling operations. Each convolution block includes two 3 x 3 convolutions, two batch normalization (BN) and two rectified linear units (ReLU). It is then fed into the saliency attention transfer mechanism to guide temporal modeling and further capture the dependencies between previous and next frames of the video. The saliency attention transfer mechanism is implemented by the saliency attention transfer module, which contains a spatiotemporal attention guidance fusion submodule and a ConvGRU submodule for temporal modeling. Its structure can be found in Figure 3 .

[0058] The spatiotemporal attention guided fusion submodule first calculates the input features X The global information in the time dimension generates attention weights through the fully connected layer to enhance the model's attention to important time segments in the time series data, and then multiplies them to obtain the adjusted hidden state ; Then, and X Splicing is performed on the channel dimension, and the attention map is generated by preliminary fusion of convolution, tanh function and convolution. , which is calculated as follows:

[0059] ,

[0060] Next, through The operator calculates the normalized spatial attention map, and its calculation formula is:

[0061] ,

[0062] in Indicates location Attention map elements, Expressed as a natural constant e Finally, the spatial attention map is combined with the input X Multiply to get the adjusted features , the specific formula is as follows:

[0063]

[0064] ConvGRU is used to capture the significant dependencies between consecutive frames. The calculation formula of this module is as follows:

[0065] ,

[0066] in It is used to predict the enhanced features of the salient region of the current frame and is also the hidden state passed to the next frame. , ,and Represents the convolution weights.

[0067] The continuous video sequence is passed through N Layer loop calculations are further enhanced by experimental settings N The image is then decoded to 4 by a deconvolution decoder to restore it to the input video frame size, and finally a sigmoid operation is performed to obtain the temporal saliency map of the temporal branch network.

[0068] The present invention provides a preferred embodiment to perform S3. The multi-resolution feature aggregation module includes a joint sampling prior block and a layer-by-layer association aggregation block. Figure 2 .

[0069] The joint sampling prior block is used to obtain spatial information of different resolutions and enhance the expressiveness of salient features. Figure 4 The block has four sets of feature groups with different channels, each set has four layers of feature maps with different resolutions. Each set of calculations starts with a 3×3 CBR operation, and performs downsampling with a step size of 2 along the bottom-up path or upsampling with a scale factor of 2 along the top-down path. Then, the feature maps of the same layer are spliced to obtain four layers of fused features with different resolutions. , which includes rough semantics and fine details. Its expression formula is as follows:

[0070] ,

[0071] in, represents the original input image features, Indicates the Group The feature map of the layer, , Represents the combined operation of 3×3 convolution (Conv), batch normalization (BN), and rectified linear unit (ReLU).

[0072] The layer-by-layer association aggregation block is used to aggregate the salient features of different receptive fields to further capture rich image features. Figure 5 The block first performs a 1×1 CBR operation on the fusion feature maps of each layer to obtain Then the high-level fusion features are upsampled and summed with the adjacent low-level fusion features to obtain This operation is performed from the fourth layer to the first layer, establishing a top-down association path, aggregating significant information of different receptive fields layer by layer, while enhancing the detail information, so that the final aggregated features have the same resolution as the original input. Finally, a sigmoid operation is performed on the aggregated features to obtain the final spatial saliency map The specific formula is as follows:

[0073] ,

[0074] in, For the sigmoid function operation, Represents the combined operation of 1×1 convolution (Conv), batch normalization (BN) and rectified linear unit (ReLU), It is the spatial saliency map finally output by the spatial branch network.

[0075] The present invention provides a preferred embodiment for performing S4. Temporal and spatial features are fused and decoded. First, the temporal saliency map and the spatial saliency map are spliced in the channel dimension. Then, three 3×3 CBR operations and one 1×1 CBR operation are sequentially performed on the spliced features, while the size of the feature map remains unchanged. Finally, a sigmoid function is used to obtain the final predicted video saliency map.

[0076] To verify the effectiveness of Example 1, the method presented here was compared with other advanced methods on two datasets, DADA-2000 and TDV. Four commonly used metrics were used for performance evaluation: Area Under the Curve (AUC)-Judd (AUC-J), Normalized Scan Path Significance (NSS), Linear Correlation Coefficient (CC), and Kullback-Leibler Divergence (KLD). Larger values for the first three metrics and smaller KLD values indicate better model performance and more accurate prediction of significant regions. The experimental results are shown in Tables 1 and 2.

[0077] Table 1 Comparison of prediction accuracy on the DADA-2000 dataset

[0078]

[0079] Table 2 Comparison of prediction accuracy on the TDV dataset

[0080]

[0081] As can be seen from Tables 1 and 2, the method of this embodiment outperforms previous methods in many indicators on the DADA-2000 and TDV datasets, achieves good prediction accuracy, and can more accurately predict driver attention in traffic videos, proving the effectiveness of this embodiment.

[0082] Example 2

[0083] This embodiment provides a traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion, see Figure 6 ,include:

[0084] The spatiotemporal data input module obtains the traffic video data to be predicted, selects continuous video sequences in the temporal branch network, and selects single-frame video images in the spatial branch network, and inputs them into the traffic video saliency prediction network respectively;

[0085] The temporal feature extraction module uses an encoder to preliminarily encode the input video sequence, then uses a saliency attention transfer mechanism to capture the dependencies between the previous and next video frames. The module then uses a deconvolution decoder to perform decoding operations, extracting salient temporal features and obtaining a temporal saliency map.

[0086] The spatial feature extraction module processes the input single-frame image through a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to obtain significant information of different receptive fields, extract significant spatial features, and obtain a spatial saliency map.

[0087] The fusion decoding module fuses and decodes temporal and spatial features using a concatenation and convolution approach, calculates the final traffic video saliency map, and obtains the prediction result.

[0088] It should be noted that the above modules correspond to steps S1 to S4 in Example 1. The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0089] In further embodiments, there is also provided:

[0090] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed by the processor. When the computer instructions are executed by the processor, the method described in Example 1 is performed. For the sake of brevity, no further details are given here.

[0091] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0092] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0093] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in embodiment 1 is performed.

[0094] The method in Example 1 can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software module can be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0095] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] The foregoing description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion, characterized by: The steps include: S1: Obtain the traffic video data to be predicted, select a continuous video sequence in the time branch network, and select a single frame video image in the spatial branch network, and input them into the trained traffic video saliency prediction network respectively; S2: After the video sequence is initially encoded by the encoder, the salient attention transfer mechanism guides the temporal modeling of consecutive frames to enhance the saliency dependencies between frames. This mechanism is implemented by the salient attention transfer module, which includes a spatiotemporal attention guidance fusion submodule and a ConvGRU submodule for temporal modeling. The spatiotemporal attention guided fusion submodule first calculates the input features X The global information in the time dimension generates attention weights through the fully connected layer to enhance the model's attention to important time segments in the time series data, and then H t-1 Multiply to get the adjusted hidden state ; Then, and X Splicing is performed on the channel dimension, and the attention map is generated by preliminary fusion of convolution, tanh function and convolution. A t , which is calculated as follows: ， ， Next, through The operator calculates the normalized spatial attention map, and its calculation formula is: ， in express Attention map elements at positions, Expressed as a natural constant e Finally, the spatial attention map is combined with the input X Multiply to get the adjusted features , the specific formula is as follows: ， ConvGRU is used to capture the significant dependencies between consecutive frames. The calculation formula of this module is as follows: ， in, represents the update gate, Represents the reset gate, Represents a new memory unit, It is used to predict the enhanced features of the salient region of the current frame and is also the hidden state passed to the next frame. , and represents the convolution weight; The continuous video sequence is further enhanced by multi-layer loop calculation, and then decoded by the deconvolution decoder to restore it to the input video frame size. Finally, a sigmoid operation is performed to obtain the temporal saliency map of the temporal branch network. S3: A single-frame video image is passed through a multi-resolution feature aggregation module to capture salient information in different receptive fields to enhance the ability to express detailed features. This module includes a joint sampling prior block and a layer-by-layer association aggregation block. The joint sampling prior block is used to obtain spatial information of different resolutions and enhance the expressiveness of significant features. This block constructs four sets of feature groups with different channels, each with four layers of feature maps of different resolutions. Each set of calculations starts with a 3×3 CBR operation, i.e., Conv+BN+ReLU, and performs downsampling with a step size of 2 along the bottom-up path or upsampling with a scale factor of 2 along the top-down path. The feature maps of the same layer are then spliced to obtain four stacks of fused features of different resolutions. , which includes rough semantics and fine details; its expression formula is as follows: ， in, represents the original input feature map, Indicates the Group The feature map of the layer, , Represents the combined operation of 3×3 convolution, batch normalization and rectified linear unit; The layer-by-layer association aggregation block is used to aggregate the salient features of different receptive fields to further capture rich image features. The block first performs a 1×1 CBR operation on the fusion feature maps of each layer to obtain Then the high-level fusion features are upsampled and summed with the adjacent low-level fusion features to obtain This operation is performed from the fourth layer to the first layer, establishing a top-down association path, aggregating significant information of different receptive fields layer by layer, while enhancing the detail information, making the final aggregated feature have the same resolution as the original input; finally, the sigmoid operation is performed on the aggregated feature to obtain the final spatial saliency map ; The specific formula is as follows: ， in, For the sigmoid function operation, represents the combined operation of 1×1 convolution, batch normalization and rectified linear unit, The spatial saliency map output by the spatial branch network; S4: Temporal and spatial features are fused and decoded using concatenation and convolution, and the final traffic video saliency map is calculated to obtain the prediction result.

2. The traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion according to claim 1 is characterized in that: The training steps of the trained traffic video saliency prediction network include: Collect and preprocess training data into input video frame sequences and label ground truth saliency maps; Build a traffic video saliency prediction network and set the initial weight value; Input the training set into the prediction network, train the network, and output the predicted saliency map; Use the loss function to calculate the difference between the predicted saliency map and the true saliency map and backpropagate; After multiple iterative training, when the loss function converges, the optimal parameters and structure are saved to obtain the trained traffic video saliency prediction network.

3. The traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion according to claim 1 is characterized in that: The encoder is used to preliminarily encode the video sequence: the encoder follows the typical architecture of the convolutional network, consisting of 5 convolution blocks and 3 2×2 maximum pooling operations, each convolution block includes two 3 x 3 convolutions, two batch normalizations and two rectified linear units.

4. The traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion according to claim 1, characterized in that: The temporal features and spatial features are fused and decoded. First, the temporal saliency map and the spatial saliency map are spliced in the channel dimension. Then, three 3×3 CBR operations and one 1×1 CBR operation are performed on the spliced features in sequence. The size of the feature map remains unchanged. Finally, the final predicted video saliency map is obtained through the sigmoid function.

5. A traffic video saliency prediction system based on spatiotemporal dual-branch network feature fusion, the system being used to implement the traffic video saliency prediction method based on spatiotemporal dual-branch network feature fusion according to claim 1, characterized in that: include: The spatiotemporal data input module obtains the traffic video data to be predicted, selects continuous video sequences in the temporal branch network, and selects single-frame video images in the spatial branch network, and inputs them into the traffic video saliency prediction network respectively; The temporal feature extraction module uses an encoder to preliminarily encode the input video sequence, then uses a saliency attention transfer mechanism to capture the dependencies between the previous and next video frames. The deconvolution decoder then performs decoding operations to extract salient temporal features and obtain a temporal saliency map. The spatial feature extraction module processes the input single-frame image through a multi-resolution feature aggregation module with joint sampling and layer-by-layer association to obtain significant information of different receptive fields, extract significant spatial features, and obtain a spatial saliency map. The fusion decoding module fuses and decodes temporal and spatial features using a concatenation and convolution approach, calculates the final traffic video saliency map, and obtains the prediction result.

6. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 4 is completed.

7. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the method according to any one of claims 1 to 4.

Images