Attention and graph synergistically enhanced video salient object detection method

Abstract

The application discloses a video saliency object detection method with attention and graph collaborative enhancement, which comprises video frame preprocessing, network model establishment and video saliency detection steps. The collaborative attention graph network is adopted, the spatial edge attention module SEM and the collaborative attention graph module AGM are used respectively to independently extract the space-time clues and the spatial edge features, the inter-frame attention is combined with the space-time graph convolution network GCN, the motion information between the video frames is captured, and the receptive field is expanded to capture the long-distance dependence. The space-time graph convolution network GCN connects the features from the time to the space branch, can collaboratively fuse the cross-modal features, and simultaneously considers the global and local space-time features. The SEM enhances the high-level information by using the sobel operator and the spatial attention module to extract the spatial and edge features from the low-level data. The application can effectively extract the most interesting target content of human eyes in a video, pay more attention to the salient regions, and suppress the background interference.

Description

TECHNICAL FIELD

[0001] The application relates to a video salient object detection method, in particular to a video salient object detection method enhanced by attention and graph collaboration, and belongs to the technical field of computer vision. BACKGROUND

[0002] With the popularity of social media platforms such as TikTok, Kuaishou, YouTube Shorts, etc., the production and consumption of short videos have reached an unprecedented level. Short videos not only have diverse content and frequent updates, but users' demands for video editing, special effect production and personalized recommendation are also increasing. However, analyzing and processing large amounts of video data is still a challenging task. As a basic task of computer vision, video salient object detection (VSOD) aims to identify and segment the most eye-catching objects in a video; it has many important applications and is beneficial to downstream tasks in practical scenarios, such as video compression, video editing, video object tracking and person re-identification. Early VSOD methods usually rely on traditional computer vision features, such as color, texture, motion vector, etc. These methods usually achieve object detection by designing specific feature extraction and saliency estimation models. These methods have limited performance in complex background, dynamic change, target occlusion scenarios, and are difficult to effectively handle diverse targets and environments. With the success of deep learning in image processing tasks, researchers began to apply deep learning models such as convolutional neural networks (CNN) to the VSOD task. Although these methods significantly improve detection performance, they still face challenges such as difficulty in processing long-time video and capturing complex motion patterns. In addition, such methods are computationally intensive and difficult to meet the needs of real-time applications.

[0003] In recent years, attention mechanisms and graph neural networks have been gradually introduced into VSOD to better capture spatiotemporal dependencies and complex target relationships in videos. Through self-attention mechanisms (Self-Attention), selective attention is paid to key spatial regions and time periods in the video, improving the accuracy of salient object detection. This type of method can better handle long-time dependencies and reduce computational redundancy. Video frames are modeled as graph structures, and information is propagated through GCN. Nodes can represent pixels or regions in the video, and edges can represent spatiotemporal relationships. This type of method can effectively integrate spatiotemporal information and is particularly suitable for handling complex motion and multi-target scenarios. SUMMARY

[0004] The purpose of the present application is to provide a video salient object detection method enhanced by attention and graph collaboration.

[0005] To solve the above technical problems, the technical solution adopted by the present application is:

[0006] A video salient object detection method with attention and graph collaborative enhancement, comprising the following steps:

[0007] Step 1: Video frame preprocessing: adjust the size of the input frame to a preset size tensor X;

[0008] Step 2: Establish a network model: the network model includes an encoder, an intermediate layer, and a decoder; the encoder is composed of Block1 module, Block2 module, Block3 module and Block4 module connected in sequence; the intermediate layer is composed of spatial edge attention module SEM1-SEM2 with the same structure and attention and graph collaborative enhancement module SAGM3-SAGM4 with the same structure connected in parallel; the spatial edge attention module SEM1 is composed of a spatial attention module and a sobel operator; the attention and graph collaborative enhancement module SAGM3 is composed of an inter-frame attention module IFA and a spatio-temporal graph convolution module GCN;

[0009] The decoder is composed of fourth to first feature processing units connected in sequence; the fourth to second feature processing units have the same structure, and the fourth feature processing unit is composed of an upsampling module Upsample4, an adder and a convolution module Conv4 connected in sequence; the first feature processing unit is composed of a convolution module Conv1 and an upsampling module Upsample1 connected in sequence;

[0010] The input end of the attention and graph collaborative enhancement module SAGM4 is connected to the output end of the Block4 module, and the output end thereof is connected to the input end of the upsampling module Upsample4 in the fourth feature processing unit; the input end of the attention and graph collaborative enhancement module SAGM3 is connected to the output end of the Block3 module, and the output end thereof is connected to the corresponding input end of the adder in the fourth feature processing unit; the input end of the spatial edge attention module SEM2 is connected to the output end of the Block2 module, and the output end thereof is connected to the corresponding input end of the adder in the third feature processing unit; the input end of the spatial edge attention module SEM1 is connected to the output end of the Block1 module, and the output end thereof is connected to the corresponding input end of the adder in the second feature processing unit;

[0011] Step 3: Video saliency detection: input each frame in the video into the network model in sequence to detect the salient object; the first to fourth features output by the Block1 module, the Block2 module, the Block3 module and the Block4 module respectively form a multi-level feature {F1, F2, F3, F4}, and the first to fourth features are input into the input ends of the spatial edge attention module SEM1-SEM2 and the attention and graph collaborative enhancement module SAGM3-SAGM4 respectively; the F t GCN , t = 3, 4 output by the attention and graph collaborative enhancement module SAGM3-SAGM4 and the Ft SEM t = 1,2 input the corresponding input end of the decoder; the decoder output prediction map.

[0012] Further, the spatial edge module SEM1-SEM2 is composed of a spatial attention module and a sobel operator; the spatial attention module respectively passes the input features through channel direction maximum pooling and average pooling to obtain two two-dimensional feature maps, then merges the two two-dimensional feature maps in the channel direction, and obtains a spatial attention map after processing by using a convolution layer Conv and a sign function The sobel operator extracts an edge map G X and G y are horizontal and vertical direction edge maps respectively, and the output spatial edge map is

[0013] Further, the attention and graph collaborative enhancement module SAGM3-SAGM4 is composed of an inter-frame attention IFA and a spatio-temporal graph convolution GCN; the inter-frame attention IFA is used to calculate the inter-frame attention:

[0014] F t IFA = LN(MHSA(F t , DOF t )+F t ), t = 3,4

[0015] Wherein MHSA is a multi-head self-attention computing mechanism, and LN is a regularization layer; the calculation method of the dynamic offset feature DOF t is as follows:

[0016]

[0017] For each position P0 on the dynamic offset feature DOF t , P n enumerates all positions in R, and R is a position set around P0.

[0018] MAM t is the attention map between frames, and the calculation method is as follows:

[0019] MAM t = Sigmoid(x n+1 -x n )*x n+1 +x n+1 , t = 3,4

[0020] Wherein x n and x n+1 are the nth and the n+1 video frames, and Sigmoid is an activation function.

[0021] spatio-temporal graph convolution for inter-frame attention F t IFA A graph including a spatial graph and a temporal graph is constructed, different regions in the image are divided by aggregating the k nearest points in the spatial graph, and then an edge convolution is used to mine the temporal correlation between nodes in the graph, the convolution kernel moves along the time dimension, and the temporal features in the graph are extracted, and the spatial graph and the temporal graph are fused to output F t GCN t=3,4.

[0022] By adopting the technical scheme, the present application has the following technical effects:

[0023] 1. The present application proposes a novel collaborative attention graph network, which independently extracts spatio-temporal clues and spatial edge features using a collaborative attention graph module (SAGM) and a spatial edge attention module (SEM).

[0024] 2. The inter-frame attention (IFA) is combined with the spatio-temporal graph convolution network (GCN) to integrate their respective advantages. The inter-frame attention captures the motion information between video frames, while expanding the receptive field to capture long-distance dependencies. The spatio-temporal GCN models the video as a graph, connecting the features from the time to the spatial branch, and can collaboratively integrate cross-modal features. This synergy can consider both global and local spatio-temporal features.

[0025] 3. The SEM enhances high-level information by extracting spatial and edge features from low-level data using a Sobel operator and a spatial attention module. DETAILED DESCRIPTION

[0026] Figure 1 is the framework diagram of the present application.

[0027] Figure 2 is the structure diagram of the inter-frame attention (IFA) of the present application.

[0028] Figure 3 is the structure diagram of the spatio-temporal graph convolution (GCN) of the present application.

[0029] Figure 4 is the structure diagram of the spatial edge module (SEM) of the present application.

[0030] Figure 5 is the input image of embodiment 1 of the present application.

[0031] Figure 6 is the saliency map detected by embodiment 1 of the present application. DETAILED DESCRIPTION

[0032] The following examples are used to illustrate the present application.

[0033] Example 1

[0034] Referring to Figure 1 A video saliency object detection method enhanced by attention and graph collaboration, comprising the following steps:

[0035] A video saliency object detection method enhanced by attention and graph collaboration, comprising the following steps:

[0036] Step 1: Video frame preprocessing: adjust the size of the input frame to a preset size tensor X.

[0037] Step 2: Establish a network model: the network model includes an encoder, an intermediate layer, and a decoder; the encoder is composed of Block1 module, Block2 module, Block3 module and Block4 module connected in sequence; the intermediate layer is composed of spatial edge attention module SEM1-SEM2 with the same structure and attention and graph collaborative enhancement module SAGM3-SAGM4 with the same structure connected in parallel; the spatial edge attention module SEM1 is composed of a spatial attention module and a sobel operator; the attention and graph collaborative enhancement module SAGM1 is composed of an inter-frame attention module IFA and a spatio-temporal graph convolution module GCN;

[0038] The decoder is composed of fourth to first feature processing units connected in sequence; the fourth to second feature processing units have the same structure, and the fourth feature processing unit is composed of an upsampling module Upsample4, an adder and a convolution module Conv4 connected in sequence; the first feature processing unit is composed of a convolution module Conv1 and an upsampling module Upsample1 connected in sequence;

[0039] The input end of the attention and graph collaborative enhancement module SAGM4 is connected to the output end of the Block4 module, and the output end thereof is connected to the input end of the upsampling module Upsample4 in the fourth feature processing unit; the input end of the attention and graph collaborative enhancement module SAGM3 is connected to the output end of the Block3 module, and the output end thereof is connected to the corresponding input end of the adder in the fourth feature processing unit; the input end of the spatial edge attention module SEM2 is connected to the output end of the Block2 module, and the output end thereof is connected to the corresponding input end of the adder in the third feature processing unit; the input end of the spatial edge attention module SEM1 is connected to the output end of the Block1 module, and the output end thereof is connected to the corresponding input end of the adder in the second feature processing unit.

[0040] The spatial edge module SEM1-SEM2 is composed of a spatial attention module and a sobel operator; the spatial attention module processes the input features through channel direction maximum pooling and average pooling to obtain two two-dimensional feature maps, and then merges the two two-dimensional feature maps in the channel direction to obtain a spatial attention map after processing by a convolution layer Conv and a sign function The sobel operator extracts an edge map G X and G y are horizontal and vertical direction edge maps respectively, the output spatial edge map is

[0041] The attention and graph collaborative enhancement module SAGM3-SAGM4 is composed of inter-frame attention IFA and spatio-temporal graph convolution GCN; the inter-frame attention IFA is used to calculate the inter-frame attention:

[0042] F t IFA = LN(MHSA(F t , DOF t )+F t ), t = 3, 4

[0043] wherein MHSA is a multi-head self-attention computing mechanism, and LN is a regularization layer; the calculation method of the dynamic offset feature DOF t is as follows:

[0044]

[0045] For each position P0, P t in the dynamic offset feature DOF n , all positions in R are exhausted, R being a position set around P0.

[0046] MAM t is an inter-frame attention map, and the calculation method thereof is as follows:

[0047] MAM t = Sigmoid(x n+1 -x n )*x n+1 +x n+1 , t = 3, 4

[0048] wherein x n and x n+1 are the nth and the n+1 video frames, and Sigmoid is an activation function;

[0049] The spatio-temporal graph convolution is used for the inter-frame attention F t IFA A graph including a spatial graph and a temporal graph is constructed, different regions in the image are divided by aggregating the k nearest points in the spatial graph, and then an edge convolution is used to mine the time correlation between nodes in the graph, the convolution kernel moves along the time dimension to extract the time features in the graph, and the spatial graph and the temporal graph are fused to obtain the output F t GCN , t = 3, 4.

[0050] Step 3: Video saliency detection: input each frame of the video into the network model in turn to detect the salient object; the first to fourth features output by the Block1 module, the Block2 module, the Block3 module, and the Block4 module respectively form a multi-level feature {F1, F2, F3, F4}, and the first to fourth features are input into the input ends of the spatial edge attention modules SEM1-SEM2 and the attention and graph collaborative enhancement modules SAGM3-SAGM4 respectively; the F t GCN output by the attention and graph collaborative enhancement module SAGM4 is input into the corresponding input ends of the decoder together with the F t SEM output by the spatial edge attention modules SEM1-SEM2; the decoder outputs a prediction map.

[0051] In this embodiment, the tensor size is 4x3x352x352. After the tensor X is processed by the feature encoders Block1-Block4 in turn, the first to fourth feature tensors are obtained, and the sizes of the first to fourth feature tensors are 4x64x112x112, 4x128x56x56, 4x320x28x28, and 4x512x14x14 respectively. The intermediate layers process features of different resolutions. The first and second features {F1, F2} have higher resolutions and contain more spatial and edge information, so the spatial edge modules SEM1-SEM2 are designed to extract the spatial information of low-level features and the edge information of salient features. After the {F1, F2} are processed by the SEM1 and SEM2, the F1 SEM and The high-level features {F3, F4} have lower resolutions and contain more semantic feature information and motion information, so the attention and graph collaborative enhancement modules SAGM3-SAGM4 are designed to extract the semantic information of salient features and the motion information of salient objects in the video. After the {F3, F4} are processed by the attention and graph collaborative enhancement modules SAGM3-SAGM4, the F3 GCN and

[0052] Step 4: Fusion of multi-level features in the decoding stage: in the feature fusion stage, the multi-level features are fused in a step-by-step addition manner, and the fused features contain more comprehensive information, which can better identify the salient features in the video. Specifically, first, the high-level features are upsampled to the same resolution as the low-level features, and then added to the low-level features and then subjected to 3x3 convolution to strengthen the features. Finally, the size of the output prediction image is 4x1x352x352.

[0053] It is particularly pointed out that at present, the technical scheme of the application has been carried out in a pilot test, and user use research has been carried out in a small range. The research results show that the user satisfaction is high, and the achievement transformation popularization is prepared. At the same time, the intellectual property risk early warning research is carried out.

Claims

1. A video salient object detection method with collaborative enhancement of attention and graph, characterized by: The following steps are involved: Step 1: Video frame preprocessing: resize the input frame into a tensor X of a preset size; Step 2: Establish a network model: The network model includes an encoder, an intermediate layer, and a decoder. The encoder consists of a cascade of Block1, Block2, Block3, and Block4 modules. The intermediate layer consists of spatial edge attention modules SEM1-SEM2 with the same structure and attention and graph collaborative enhancement modules SAGM3-SAGM4 with the same structure in parallel. The spatial edge attention module SEM1 consists of a spatial attention module and a sobel operator. The attention and graph collaborative enhancement module SAGM1 consists of an inter-frame attention module IFA and a spatiotemporal graph convolution module GCN. The decoder consists of the fourth to the first feature processing units cascaded in sequence; the fourth and second feature processing units have the same structure, and the fourth feature processing unit consists of a cascaded upsampling module Upsample4, an adder, and a convolution module Conv4; the first feature processing unit consists of a cascaded convolution module Conv1 and an upsampling module Upsample1; The input terminal of the attention and graph collaborative enhancement module SAGM4 is connected to the output terminal of the Block4 module, and its output terminal is connected to the input terminal of the upsampling module Upsample4 in the fourth feature processing unit; the input terminal of the attention and graph collaborative enhancement module SAGM3 is connected to the output terminal of the Block3 module, and its output terminal is connected to the corresponding input terminal of the adder in the fourth feature processing unit; the input terminal of the spatial edge attention module SEM2 is connected to the output terminal of the Block2 module, and its output terminal is connected to the corresponding input terminal of the adder in the third feature processing unit; the input terminal of the spatial edge attention module SEM1 is connected to the output terminal of the Block1 module, and its output terminal is connected to the corresponding input terminal of the adder in the second feature processing unit; Step 3: Video saliency detection: Each frame in the video is input into the network model in sequence to detect salient objects; the first to fourth features output by the Block1 module, Block2 module, Block3 module, and Block4 module respectively, the first to fourth features form a multi-level feature {F1, F2, F3, F4}, the first to fourth features are input into the input ends of the spatial edge attention module SEM1-SEM2 and the attention and graph collaborative enhancement module SAGM3-SAGM4 respectively; the F output by the attention and graph collaborative enhancement module SSAGM3-SAGM4 t GCN , t=3,4 and the output of spatial edge attention module SEM1-SEM2 t SEM ,t=1,2 input the corresponding input end of the decoder; the decoder outputs the prediction map.

2. The video salient object detection method with coordinated attention and graph enhancement according to claim 1 is characterized by: The spatial attention module performs maximum pooling and average pooling on the input features in the channel direction to obtain two two-dimensional feature maps. The two two-dimensional feature maps are then merged in the channel direction and processed using the convolution layer Conv and the sign function to obtain the spatial attention map. Sobel operator extracts edge map G X and G y They are horizontal and vertical edge maps respectively, and the output spatial edge map is 3. The video salient object detection method with coordinated attention and graph enhancement according to claim 1 is characterized by: The attention and graph collaborative enhancement modules SAGM3-SAGM4 consist of inter-frame attention IFA and spatiotemporal graph convolution GCN; inter-frame attention IFA is used to calculate the attention between frames: F t IFA =LN(MHSA(F t ,DOF t )+F t ),t=3,4 MHSA is a multi-head self-attention calculation mechanism, LN is a regularization layer; dynamic offset feature DOF t The calculation method is: For Dynamic Offset Feature DOF t Each position P0, P n Exhaust all positions in R, R is the set of positions around P0, MAM t is the attention map between frames, which is calculated as: MAM t =Sigmoid(x n+1 -x n )*x n+1 +x n+1 ,t=3,4 where x n and x n+1 are the nth and n+1th video frames, and Sigmoid is the activation function; Spatiotemporal graph convolution for attention between frames F t IFA A graph consisting of a spatial graph and a temporal graph is constructed. Different regions in the image are separated by aggregating the k closest points in the spatial graph. Edge convolution is then used to mine the temporal correlation between nodes in the graph. The convolution kernel moves along the time dimension to extract the temporal features in the graph. The spatial graph and the temporal graph are fused to form the output F. t GCN ,t=3,4。

Images