Attention and graph collaborative enhancement video saliency target detection method

Abstract

The invention discloses a video saliency target detection method based on attention and graph collaborative enhancement. The method comprises the steps of video frame preprocessing, network model establishment and video saliency detection. A collaborative attention graph network is adopted, a collaborative attention graph module AGM and a space edge attention module SEM are respectively used for independently extracting space-time clues and space edge features, inter-frame attention is combined with a space-time graph convolutional network GCN, and then motion information between video frames is captured while a receptive field is expanded to capture a long-distance dependency relationship. The space-time diagram convolutional network GCN connects the features from time to space branches, can collaboratively fuse cross-modal features, and considers global and local space-time features at the same time. The SEM enhances high-level information by extracting spatial and edge features from low-level data using a sobel operator and a spatial attention module. According to the method, the most interested target content of human eyes in a video can be effectively extracted, more attention is paid to the salient region, and background interference is inhibited.

Description

Technical Field

[0001] The present invention relates to a video salient object detection method, in particular to a video salient object detection method with coordinated enhancement of attention and graph, and belongs to the technical field of computer vision. Background Art

[0002] With the popularity of social media platforms such as Tik Tok, Kuaishou, and YouTube Shorts, the production and consumption of short videos have reached unprecedented heights. Short videos are not only diverse in content and frequently updated, but also have increasing user demands for video editing, special effects production, and personalized recommendations. However, analyzing and processing large amounts of video data remains a daunting challenge. As a fundamental task in computer vision, video salient object detection (VSOD) aims to identify and segment the most eye-catching objects in a video; it has many important applications and is beneficial to downstream tasks in real-world scenarios, such as video compression, video editing, video object tracking, and person re-identification. Early VSOD methods typically rely on traditional computer vision features such as color, texture, and motion vectors. These methods typically achieve object detection by designing specific feature extraction and saliency estimation models. These methods have limited performance in scenes with complex backgrounds, dynamic changes, and object occlusion, and are difficult to effectively handle diverse objects and environments. With the success of deep learning in image processing tasks, researchers have begun to apply deep learning models such as convolutional neural networks (CNNs) to VSOD tasks. Although these methods have significantly improved detection performance, they still face problems such as difficulty in processing long temporal videos and capturing complex motion patterns. In addition, such methods are computationally intensive and cannot meet the needs of real-time applications.

[0003] In recent years, attention mechanisms and graph neural networks have been gradually introduced into VSOD to better capture spatiotemporal dependencies and complex target relationships in videos. The self-attention mechanism is used to selectively focus on key spatial regions and time periods in the video, which improves the accurate detection of salient targets. This type of method can better handle long-term dependencies and reduce computational redundancy. The video frames are modeled as graph structures and information is propagated through GCN. Nodes can represent pixels or regions in the video, and edges can represent spatiotemporal relationships. This type of method can effectively integrate spatiotemporal information and is particularly suitable for processing complex motion and multi-target scenes. Summary of the invention

[0004] The purpose of the present invention is to provide a video salient object detection method with coordinated enhancement of attention and graph.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A video salient object detection method with attention and graph collaborative enhancement includes the following steps:

[0007] Step 1: Video frame preprocessing: resize the input frame into a tensor X of a preset size;

[0008] Step 2: Establish a network model: The network model includes an encoder, an intermediate layer, and a decoder; the encoder is composed of Block1, Block2, Block3, and Block4 modules cascaded in sequence, the intermediate layer is composed of spatial edge attention modules SEM1-SEM2 with the same structure and attention and graph collaborative enhancement modules SAGM3-SAGM4 with the same structure in parallel, the spatial edge attention module SEM1 is composed of a spatial attention module and a sobel operator; the attention and graph collaborative enhancement module SAGM3 is composed of an inter-frame attention module IFA and a spatiotemporal graph convolution module GCN;

[0009] The decoder is composed of the fourth to the first feature processing units cascaded in sequence; the fourth to the second feature processing units have the same structure, the fourth feature processing unit is composed of a cascaded upsampling module Upsample4, an adder, and a convolution module Conv4; the first feature processing unit is composed of a cascaded convolution module Conv1 and an upsampling module Upsample1;

[0010] The input terminal of the attention and graph collaborative enhancement module SAGM4 is connected to the output terminal of the Block4 module, and its output terminal is connected to the input terminal of the upsampling module Upsample4 in the fourth feature processing unit; the input terminal of the attention and graph collaborative enhancement module SAGM3 is connected to the output terminal of the Block3 module, and its output terminal is connected to the corresponding input terminal of the adder in the fourth feature processing unit; the input terminal of the spatial edge attention module SEM2 is connected to the output terminal of the Block2 module, and its output terminal is connected to the corresponding input terminal of the adder in the third feature processing unit; the input terminal of the spatial edge attention module SEM1 is connected to the output terminal of the Block1 module, and its output terminal is connected to the corresponding input terminal of the adder in the second feature processing unit;

[0011] Step 3: Video saliency detection: Input each frame in the video into the network model in sequence to detect salient objects; the first to fourth features output by the Block1 module, Block2 module, Block3 module, and Block4 module respectively, the first to fourth features constitute a multi-level feature {F1, F2, F3, F4}, the first to fourth features are respectively input into the input ends of the spatial edge attention module SEM1-SEM2 and the attention and graph collaborative enhancement module SAGM3-SAGM4; the F output by the attention and graph collaborative enhancement module SAGM3-SAGM4 t GCN , t=3,4 and the output of spatial edge attention module SEM1-SEM2t SEM ,t=1,2 input the corresponding input end of the decoder; the decoder outputs the prediction map.

[0012] Furthermore, the spatial edge module SEM1-SEM2 is composed of a spatial attention module and a sobel operator; the spatial attention module performs maximum pooling and average pooling on the input features in the channel direction to obtain two two-dimensional feature maps, and then merges the two two-dimensional feature maps in the channel direction, and obtains a spatial attention map after processing using the convolution layer Conv and the sign function. Sobel operator extracts edge map G X and G y They are the horizontal and vertical edge maps respectively, and the output spatial edge map is

[0013] Furthermore, the attention and graph collaborative enhancement modules SAGM3-SAGM4 consist of inter-frame attention IFA and spatio-temporal graph convolution GCN; the inter-frame attention IFA is used to calculate the attention between frames:

[0014] F t IFA =LN(MHSA(F t ,DOF t )+F t ),t=3,4

[0015] MHSA is a multi-head self-attention calculation mechanism, LN is a regularization layer, and dynamic offset feature DOF t The calculation method is:

[0016]

[0017] For Dynamic Offset Features DOF t For each position P0, P n All positions in R are enumerated, R is, and R is the set of positions around P0.

[0018] MAM t is the attention map between frames, which is calculated as:

[0019] MAM t =Sigmoid(x n+1 -x n )*x n+1 +x n+1 ,t=3,4

[0020] where x n and x n+1 are the nth and n+1th video frames, and Sigmoid is the activation function;

[0021] Spatiotemporal graph convolution for attention between frames F t IFA A graph including a spatial graph and a temporal graph is constructed. Different regions in the image are separated by aggregating the k nearest points in the spatial graph. Then, edge convolution is used to mine the temporal correlation between nodes in the graph. The convolution kernel moves along the time dimension to extract the temporal features in the graph. The spatial graph and the temporal graph are fused to form the output F. t GCN ,t=3,4.

[0022] By adopting the above technical solution, the present invention achieves the following technical effects:

[0023] 1. This paper proposes a novel collaborative attention map network, which uses a collaborative attention map module (SAGM) and a spatial edge attention module (SEM) to independently extract spatiotemporal cues and spatial edge features.

[0024] 2. Combine inter-frame attention IFA with spatiotemporal graph convolutional network GCN to integrate their respective advantages. Inter-frame attention captures motion information between video frames while expanding the receptive field to capture long-distance dependencies. Spatiotemporal GCN models videos as graphs, connects features from time to space branches, and can collaboratively fuse cross-modal features. This synergy can consider global and local spatiotemporal features at the same time.

[0025] 3. SEM enhances high-level information by extracting spatial and edge features from low-level data using the Sobel operator and a spatial attention module. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a framework diagram of the present invention.

[0027] Figure 2 It is a structural diagram of the inter-frame attention IFA of the present invention.

[0028] Figure 3 It is a structural diagram of the spatiotemporal graph convolution GCN of the present invention.

[0029] Figure 4 It is a structural diagram of the space edge module SEM of the present invention.

[0030] Figure 5 It is the input image of embodiment 1 of the present invention.

[0031] Figure 6 It is a saliency map detected by Example 1 of the present invention. DETAILED DESCRIPTION

[0032] The following examples serve to illustrate the invention.

[0033] Example 1

[0034] Reference Figure 1 , a video salient object detection method with attention and graph collaborative enhancement, comprising the following steps:

[0035] A video salient object detection method with attention and graph collaborative enhancement includes the following steps:

[0036] Step 1: Video frame preprocessing: Resize the input frame into a tensor X of a preset size.

[0037] Step 2: Establish a network model: The network model includes an encoder, an intermediate layer, and a decoder; the encoder is composed of Block1, Block2, Block3, and Block4 modules cascaded in sequence, the intermediate layer is composed of spatial edge attention modules SEM1-SEM2 with the same structure and attention and graph collaborative enhancement modules SAGM3-SAGM4 with the same structure in parallel, the spatial edge attention module SEM1 is composed of a spatial attention module and a sobel operator; the attention and graph collaborative enhancement module SAGM1 is composed of an inter-frame attention module IFA and a spatiotemporal graph convolution module GCN;

[0038] The decoder is composed of the fourth to the first feature processing units cascaded in sequence; the fourth to the second feature processing units have the same structure, the fourth feature processing unit is composed of a cascaded upsampling module Upsample4, an adder, and a convolution module Conv4; the first feature processing unit is composed of a cascaded convolution module Conv1 and an upsampling module Upsample1;

[0039] The input end of the attention and graph collaborative enhancement module SAGM4 is connected to the output end of the Block4 module, and its output end is connected to the input end of the upsampling module Upsample4 in the fourth feature processing unit; the input end of the attention and graph collaborative enhancement module SAGM3 is connected to the output end of the Block3 module, and its output end is connected to the corresponding input end of the adder in the fourth feature processing unit; the input end of the spatial edge attention module SEM2 is connected to the output end of the Block2 module, and its output end is connected to the corresponding input end of the adder in the third feature processing unit; the input end of the spatial edge attention module SEM1 is connected to the output end of the Block1 module, and its output end is connected to the corresponding input end of the adder in the second feature processing unit.

[0040] The spatial edge module SEM1-SEM2 consists of a spatial attention module and a sobel operator; the spatial attention module performs maximum pooling and average pooling on the input features in the channel direction to obtain two two-dimensional feature maps, then merges the two two-dimensional feature maps in the channel direction, and obtains a spatial attention map after processing using the convolution layer Conv and the sign function. Sobel operator extracts edge map G X and G y They are the horizontal and vertical edge maps respectively, and the output spatial edge map is

[0041] The attention and graph collaborative enhancement modules SAGM3-SAGM4 consist of inter-frame attention IFA and spatio-temporal graph convolution GCN; the inter-frame attention IFA is used to calculate the attention between frames:

[0042] F t IFA =LN(MHSA(F t ,DOF t )+F t ),t=3,4

[0043] MHSA is a multi-head self-attention calculation mechanism, LN is a regularization layer, and dynamic offset feature DOF t The calculation method is:

[0044]

[0045] For Dynamic Offset Features DOF t For each position P0, P n All positions in R are enumerated, where R is the set of positions around P0.

[0046] MAM t is the attention map between frames, which is calculated as:

[0047] MAM t =Sigmoid(x n+1 -x n )*x n+1 +x n+1 ,t=3,4

[0048] where x n and x n+1 are the nth and n+1th video frames, and Sigmoid is the activation function;

[0049] Spatiotemporal graph convolution for attention between frames F t IFA A graph including a spatial graph and a temporal graph is constructed. Different regions in the image are separated by aggregating the k nearest points in the spatial graph. Then, edge convolution is used to mine the temporal correlation between nodes in the graph. The convolution kernel moves along the time dimension to extract the temporal features in the graph. The spatial graph and the temporal graph are fused to form the output F. t GCN ,t=3,4.

[0050] Step 3: Video saliency detection: Input each frame in the video into the network model in sequence to detect salient objects; the first to fourth features output by the Block1 module, Block2 module, Block3 module, and Block4 module respectively, the first to fourth features constitute a multi-level feature {F1, F2, F3, F4}, the first to fourth features are respectively input into the input ends of the spatial edge attention module SEM1-SEM2 and the attention and graph collaborative enhancement module SAGM3-SAGM4; the F output by the attention and graph collaborative enhancement module SAGM4 t GCN , t=3,4 and the output of spatial edge attention module SEM1-SEM2 t SEM ,t=1,2 input the corresponding input end of the decoder; the decoder outputs the prediction map.

[0051] In this embodiment, the tensor size is 4×3×352×352. After the tensor X is processed by the feature encoders Block1-Block4 in sequence, the first to fourth feature tensors are obtained. The sizes of the first to fourth feature tensors are 4×64×112×112, 4×128×56×56, 4×320×28×28, and 4×512×14×14, respectively. The intermediate layer processes features of different resolutions. The first and second features {F1, F2} have higher resolution and contain more spatial and edge information, so the spatial edge modules SEM1-SEM2 are designed to extract the spatial information of low-level features and the edge information of significant features. {F1, F2} is processed by SEM1 and SEM2 to F1 SEM and The high-level features {F3, F4} have lower resolution and contain more semantic feature information and motion information, so the attention and graph collaborative enhancement modules SAGM3-SAGM4 are designed to extract the semantic information of salient features and the motion information of salient objects in the video. {F3, F4} is obtained by the attention and graph collaborative enhancement modules SAGM3-SAGM4. GCN and

[0052] Step 4: Fusion of multi-level features in the decoding stage: In the feature fusion stage, multi-level features are fused step by step by adding them step by step. The fused features contain more comprehensive information and can better identify the significant features in the video. Specifically, the high-level features are first upsampled to the same resolution as the low-level features, then added to the low-level features and then subjected to 3×3 convolution to enhance the features. The size of the final output prediction image is 4×1×352×352.

[0053] It is particularly noted that, at present, the technical solution of the present invention has been piloted and a user survey has been conducted in a small range. The survey results show that the user satisfaction is high, and preparations for the transformation and promotion of the results have begun. At the same time, a survey on intellectual property risk warning has been carried out.

Claims

1. A video salient object detection method with attention and graph synergistic enhancement, characterized by: The following steps are involved: Step 1: Video frame preprocessing: resize the input frame into a tensor X of a preset size; Step 2: Establish a network model: The network model includes an encoder, an intermediate layer, and a decoder; the encoder is composed of Block1, Block2, Block3, and Block4 modules cascaded in sequence, the intermediate layer is composed of spatial edge attention modules SEM1-SEM2 with the same structure and attention and graph collaborative enhancement modules SAGM3-SAGM4 with the same structure in parallel, the spatial edge attention module SEM1 is composed of a spatial attention module and a sobel operator; the attention and graph collaborative enhancement module SAGM1 is composed of an inter-frame attention module IFA and a spatiotemporal graph convolution module GCN; The decoder is composed of the fourth to the first feature processing units cascaded in sequence; the fourth to the second feature processing units have the same structure, the fourth feature processing unit is composed of a cascaded upsampling module Upsample4, an adder, and a convolution module Conv4; the first feature processing unit is composed of a cascaded convolution module Conv1 and an upsampling module Upsample1; The input terminal of the attention and graph collaborative enhancement module SAGM4 is connected to the output terminal of the Block4 module, and its output terminal is connected to the input terminal of the upsampling module Upsample4 in the fourth feature processing unit; the input terminal of the attention and graph collaborative enhancement module SAGM3 is connected to the output terminal of the Block3 module, and its output terminal is connected to the corresponding input terminal of the adder in the fourth feature processing unit; the input terminal of the spatial edge attention module SEM2 is connected to the output terminal of the Block2 module, and its output terminal is connected to the corresponding input terminal of the adder in the third feature processing unit; the input terminal of the spatial edge attention module SEM1 is connected to the output terminal of the Block1 module, and its output terminal is connected to the corresponding input terminal of the adder in the second feature processing unit; Step 3: Video saliency detection: Input each frame in the video into the network model in sequence to detect salient objects; the first to fourth features output by the Block1 module, Block2 module, Block3 module, and Block4 module respectively, the first to fourth features constitute a multi-level feature {F1, F2, F3, F4}, the first to fourth features are respectively input into the input ends of the spatial edge attention module SEM1-SEM2 and the attention and graph collaborative enhancement module SAGM3-SAGM4; the F output by the attention and graph collaborative enhancement module SSAGM3-SAGM4 t GCN , t=3,4 and the output of spatial edge attention module SEM1-SEM2 t SEM ,t=1,2 input the corresponding input end of the decoder; the decoder outputs the prediction map.

2. The video salient object detection method with attention and graph collaborative enhancement according to claim 1 is characterized in that: The spatial attention module performs maximum pooling and average pooling on the input features in the channel direction to obtain two two-dimensional feature maps. The two two-dimensional feature maps are then merged in the channel direction and processed using the convolution layer Conv and the sign function to obtain the spatial attention map. Sobel operator extracts edge map G X and G y They are the horizontal and vertical edge maps respectively, and the output spatial edge map is 3. The video salient object detection method with attention and graph collaborative enhancement according to claim 1 is characterized in that: The attention and graph collaborative enhancement modules SAGM3-SAGM4 consist of inter-frame attention IFA and spatio-temporal graph convolution GCN; the inter-frame attention IFA is used to calculate the attention between frames: F t IFA =LN(MHSA(F t ,DOF t )+F t ),t=3,4 MHSA is a multi-head self-attention calculation mechanism, LN is a regularization layer, and dynamic offset feature DOF t The calculation method is: For Dynamic Offset Features DOF t For each position P0, P n Exhaustively enumerate all positions in R, where R is the set of positions around P0, MAM t is the attention map between frames, which is calculated as: MAM t =Sigmoid(x n+1 -x n )*x n+1 +x n+1 ,t=3,4 where x n and x n+1 are the nth and n+1th video frames, and Sigmoid is the activation function; Spatiotemporal graph convolution for attention between frames F t IFA A graph including a spatial graph and a temporal graph is constructed. Different regions in the image are separated by aggregating the k nearest points in the spatial graph. Then, edge convolution is used to mine the temporal correlation between nodes in the graph. The convolution kernel moves along the time dimension to extract the temporal features in the graph. The spatial graph and the temporal graph are fused to form the output F. t GCN ,t=3,4.

Images