A video salient object detection method based on a graph model

Abstract

The application discloses a video saliency object detection method based on a graph model. The application comprises the following steps: (1) a video saliency object detection model based on a graph convolution network is proposed; (2) a cross-level interaction module based on a graph convolution is designed, time-space features of different levels are taken as graph nodes, edges are constructed according to distance information between cross-modal features and cross-level features, features of the graph nodes are updated through graph convolution, and the cross-modal features and the cross-level features are fused in a channel dimension, so that time-space deep features are generated; (3) a feature self-correction module based on a graph convolution is designed, the time-space deep features are mapped in a space dimension, a plurality of graph nodes correspond to a plurality of space regions, and features of the graph nodes are updated through graph convolution. The application fully mines space information and time information, introduces a graph convolution network as a link for extracting features, infers saliency regions in a feature map, and realizes mutual enhancement between a plurality of saliency objects.

Description

Technical Field

[0001] This invention belongs to the field of computer vision. Video salient object detection aims to locate and segment the most attention-grabbing objects by utilizing spatial and temporal cues hidden in video sequences. This task originates from human visual attention behavior in cognitive research, namely, the rapid transfer of attention to the most informative region in a visual scene. Background Technology

[0002] Humans capture important object and scene information in the objective world through their visual perception system, such as attributes like depth, appearance, color, and shape. Whether in a simple scene or a complex environment, humans can quickly locate regions of interest, grasp key information, and process and integrate information rapidly and effectively. To enable computer systems to also quickly locate important targets and perceive important scene information, the task of visual salient detection has emerged. Salient regions of a scene typically contain important targets of human interest or the content that best represents the image; they are areas that can attract visual attention in a short time. Salient object detection is the process of identifying these targets or regions of interest. As an effective preprocessing technique, salient object detection has been widely applied to numerous computer vision tasks such as retrieval, recognition, segmentation, redirection, enhancement, person detection, evaluation, and compression.

[0003] With the advent of the big data era, data formats have undergone tremendous changes. Traditional image data is no longer sufficient to meet people's ever-growing sensory needs, and video data has experienced explosive growth. How to accurately and consistently extract salient targets from video data has become a pressing new challenge. Given the good scalability of video salient target detection technology, it has been widely applied in video target detection, video summarization, and content-based video retrieval. Unlike image salient target detection, video salient target detection requires combining temporal and spatial information to continuously locate motion-related salient targets in video sequences. Compared to collaborative salient target detection, video salient target detection also needs to consider motion information and temporal characteristics, and has the prior knowledge of "high correlation between adjacent video frames." Therefore, how to fully mine the motion information and temporal relationships of video sequences has become the key to video salient target detection research. Due to the large volume of video data, significant scene changes, and inconsistent target sizes, video salient target detection research is quite difficult, and the overall algorithm performance is relatively low. In recent years, the field of video salient target detection has developed rapidly, with new algorithms emerging one after another, and algorithm performance constantly being improved.

[0004] Based on whether or not the model needs to be trained, existing video salient object detection methods can be divided into two main categories: methods based on underlying cues and learning-based methods. Underlying cues-based methods include those based on transform analysis, information theory, sparse representation, and visual priors; learning-based methods include those based on traditional learning and those based on deep learning.

[0005] Based on traditional learning methods

[0006] Liu et al. [1] A video saliency detection model based on superpixel segmentation is proposed, which perceives motion information by calculating motion and color histograms. (Wang et al.) [2] An unsupervised, geodesic-distance-based video salient object segmentation model is proposed, which extracts superpixel region boundaries and inter-frame motion boundaries, and uses geodesic distance to obtain a saliency map. Liu et al. [3] A video saliency detection model based on superpixel-level graphs and spatiotemporal propagation is proposed. (Lee et al.) [4] This paper proposes a method for video saliency detection that combines low-level and high-level visual features with machine learning algorithms. (Ren et al.) [5] A video saliency detection model based on sparse representation is proposed.

[0007] Deep learning-based methods

[0008] In recent years, with the rapid development of deep learning, deep learning-based video salient object detection algorithms have shown great potential. Deep learning-based video salient object detection requires fully mining the spatiotemporal information of video sequences and simultaneously fusing two types of cross-modal information in a complementary manner without introducing noise to obtain accurate and consistent video salient objects. Summarizing existing methods, based on their framework structure, they can be divided into those based on 3D convolutional fusion of spatiotemporal features, those using Long Short-Term Memory networks to mine temporal features, those introducing optical flow maps to form a two-stream structure, and those employing other structures.

[0009] Le et al. [6] An end-to-end 3D fully convolutional network is proposed, employing UNet. [7] The topological structure enhances contextual information and enriches salient features at each level through skip connections; Cheng et al. [8] A decoder based on 3D convolution temporal units is proposed to improve the ability to perceive motion information. The design of this structural unit maintains lightweight design without reducing the ability to perceive motion information.

[0010] Fan et al. [9]A Long Short-Term Memory (SSAV) Convolutional Network oriented towards saliency transfer is proposed. It innovatively combines the traditional SSAV Convolutional Network (ConvLSTM) with a saliency transfer-aware attention mechanism, simultaneously considering temporal variations and saliency transfer to achieve accurate salient object detection in videos. (Song et al.)

[10] We propose a fast video salient object detection model based on a recursive network structure, which introduces pyramid dilated convolution and bidirectional ConvLSTM modules to learn multi-scale spatiotemporal information.

[0011] Li et al.

[11] A motion-guided video salient object detection network is proposed. FlowNet2 is used to generate optical flow images, and a branch is designed to extract motion features from the optical flow images. Therefore, the quality of the optical flow image will inevitably affect the results. (Xu et al.)

[12] A multi-branch attention network based on graph convolution is proposed. Graph models are constructed at both static and dynamic branches, and features are extracted using graph convolution. However, using supervoxels as nodes for prediction inevitably leads to a blocky effect in the final prediction graph. Ji et al.

[13] A simple yet efficient guidance and instruction network architecture is proposed, which allows motion features to implicitly and explicitly guide the learning of appearance features at the encoding and decoding layers, respectively.

[0012] In addition, Gu et al.

[14] A video salient object detection network based on a self-attention mechanism is proposed. A constrained self-attention operation is designed to capture local motion cues, thereby preventing the learning of motion-irrelevant contextual information and reducing computational and storage costs. Zhang et al.

[15] A dynamic context-sensitive filtering network is proposed, in which the context-sensitive filtering module dynamically generates context-sensitive convolutional kernels by estimating location-related affinity weights, thereby improving the model's adaptability to constantly changing scenes. Summary of the Invention

[0013] The purpose of this invention is to provide a video salient object detection method based on a graph model. This method involves a video salient object detection model based on a graph convolutional network.

[0014] The technical solution adopted by this invention to solve its technical problem is as follows:

[0015] (1) A video salient target detection model based on graph convolutional network is proposed to fully explore spatial and temporal information and consider the fusion and complementarity between cross-modal and cross-level features.

[0016] (2) Design a layer-by-layer interaction module based on graph convolution. The spatiotemporal features of different layers are used as graph nodes. Edges are constructed based on the distance information between cross-modal features and cross-layer features. The features of graph nodes are updated through graph convolution. Cross-modal features and cross-layer features are fused in the channel dimension to generate spatiotemporal depth features.

[0017] (3) Design a feature self-correction module based on graph convolution to map spatiotemporal depth features in the spatial dimension. Corresponding graph nodes represent the corresponding spatial regions. The features of graph nodes are updated through graph convolution, thereby modeling the semantic relationship in space, strengthening the connection between salient regions, and fully highlighting the salient target.

[0018] Furthermore, step (1) is implemented as follows:

[0019] The basic framework of the video salient object detection model is as follows: Figure 1 As shown, the structure follows an encoder-decoder architecture. The input to the encoding part consists of three images: the RGB image of the current frame, the RGB image of the next frame, and the optical flow map corresponding to the current frame. The encoding part is composed of three backbone networks, which extract features corresponding to the three images respectively. Two ResNet-50 backbone networks are used to extract features of the RGB image of the current frame and the RGB image of the next frame, respectively, and a ResNet-34 backbone network is used to extract features of the optical flow map corresponding to the current frame. This setup is because some information has already been extracted during the generation of the optical flow map, and a simple backbone network is sufficient to extract information related to saliency while reducing the complexity of the model. In addition, unlike the two-stream structure, the input here uses three branches. Considering that when the quality of the optical flow map is poor, the model can only rely on static images to judge salient targets, which can easily cause the model to focus on salient targets in the image while ignoring motion information in the video, the adjacent next frame RGB image is added as a supplement for motion information. To reduce computational cost, features at each level, after initial extraction by the backbone network, pass through a dimensionality reduction module consisting of a 3×3 convolutional layer, a Batch Normalization (BN) layer, and a ReLU activation layer, reducing the number of feature channels to 64. For the depth features of the fifth layer (the last layer of the backbone network), they are connected to the ASPP module (composed of multiple parallel dilated convolutional layers with different sampling rates, used to acquire multi-scale object information), thereby fusing rich multi-scale features. Thus, the encoding part obtains the features F corresponding to the current frame's RGB image, the next frame's RGB image, and the optical flow map corresponding to the current frame. i A , Where i is the number of layers in the backbone network, and its value ranges from 1 to 5.

[0020] Furthermore, step (2) is implemented as follows:

[0021] A graph convolution-based inter-layer interaction module fuses the spatiotemporal features extracted from adjacent layers of the backbone network to generate spatiotemporal depth features. Specifically, the input is F. i A , The output is

[0022] For the features of the first layer: since there are no adjacent shallow layer features, the first layer features, i.e., F1A, are repeatedly input when inputting Inter GCNs. F1 A ,

[0023] For the features of the second layer: the input is F1. A , Similarly, for the features of the third layer and the features of the fourth layer

[0024] For the features of the fifth layer: since it has no adjacent deep features, the features of the fifth layer are repeatedly input when inputting Inter GCNs, i.e.

[0025] Then, considering the relational dependencies between or within different spatial regions of salient targets, a graph convolution-based feature self-correction module is employed to enhance spatiotemporal depth features, enabling features to mutually reinforce and complement each other in the spatial dimension, thus achieving self-correction. Finally, a simple layer-by-layer decoding structure is used to obtain the final salient prediction map.

[0026] The beneficial effects of this invention are as follows:

[0027] On one hand, this invention employs a graph convolution-based inter-level interaction module, using spatiotemporal features from different levels as graph nodes. Edges are constructed based on distance information between cross-modal and cross-level features. Node features are updated via a graph convolutional network to generate spatiotemporal depth features, thus fusing cross-modal and cross-level features in the channel dimension. On the other hand, a graph convolution-based feature self-correction module maps spatiotemporal depth features spatially, with corresponding nodes representing specific spatial regions. Node features are updated via a graph convolutional network to model semantic relationships in space, strengthening connections between salient regions and fully highlighting salient targets.

[0028] The video salient object detection model based on graph convolutional networks (GCNs) introduces GCNs as part of the feature extraction process, inferring salient regions in the feature maps and enabling mutual enhancement among multiple salient objects. This highlights salient regions, suppresses background regions, and improves model performance. Therefore, in video salient object detection, GCNs are introduced into the feature fusion and feature enhancement steps. Specifically, on the one hand, Inter GCNs promote the fusion of features between different levels and different modalities (temporal and spatial domains), highlighting salient objects jointly emphasized by spatiotemporal feature maps and suppressing commonly weakened backgrounds and occluders. On the other hand, Intra GCNs refine and enhance the features themselves, correcting the localization information of salient objects for deep decoded features and refining the edge features of salient objects for shallow decoded features. Attached Figure Description

[0029] Figure 1 This is a diagram of the overall structure.

[0030] Figure 2 Here is a structural diagram of Inter GCNs;

[0031] Figure 3 Here is a structural diagram of Intra GCNs;

[0032] Figure 4 To compare experimental data. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] like Figure 1-3 As shown, a video salient object detection method based on graph models includes the following key modules:

[0035] (1) Inter-level interaction modules (InterGCNs) based on graph convolution

[0036] To facilitate the interactive fusion of spatiotemporal features across different levels, a layer-by-layer interaction module based on graph convolution was designed, such as... Figure 2 As shown. The design aims to introduce graph convolution to build relationships between features at different levels and modalities.

[0037] Specifically, Inter GCNs take the spatiotemporal features of adjacent layers as input. Output spatiotemporal depth features

[0038] First, features are processed using Global Average Pooling (GAP). Compress and transform in spatial dimension to As shown in Equation (1), each compressed feature is regarded as a graph node (9 in total), and these graph nodes contain rich channel information.

[0039] Next, graph convolution inference is performed. Edge information is established by calculating the cosine distance between nodes, and an adjacency matrix is ​​constructed using the cosine distance, as shown in Equation (2). The graph nodes are iteratively updated according to the general formula of graph convolution. Here, the number of graph convolution layers is set to 2, as shown in Equation (3).

[0040] Next, for the updated graph node, it is multiplied with the original spatiotemporal features of the corresponding input, so that the original spatiotemporal features are strengthened or weakened in some channels. Then, after splicing and convolution operations, the spatiotemporal features are generated after preliminary fusion.

[0041] Finally, to strengthen the central features, i.e., the intermediate spatial features in the Inter GCNs input features, denoted as F... i A The residual structure is adopted to add the central feature to the spatiotemporal feature after preliminary fusion to generate the spatiotemporal depth feature, as shown in Equation (4).

[0042]

[0043]

[0044]

[0045]

[0046] Among them, F i A Indicates input features, This represents the processed graph node features. `avepool(·)` represents the global average pooling operation. 1×1 (·) denotes a convolution operation with a 1×1 kernel, ||·|| denotes the magnitude of the eigenvector, and A ij Let σ(·) represent the adjacency matrix, and let σ(·) represent the nonlinear activation function. N represents the degree matrix generated by the adjacency matrix. (l) W represents the node features of the l-th graph convolutional layer. (l) This represents the weight matrix, Cat(·) denotes the concatenation operation, and F i ST It represents the spatiotemporal depth feature.

[0047] (2) Feature self-correction module based on graph convolution (Intra interaction module, IntraGCNs)

[0048] This invention designs a feature self-correction module based on graph convolution, such as... Figure 3 As shown. For spatiotemporal depth features, graph convolution operations in the spatial dimension not only fully highlight the overall salient target, but also, through the semantic associations between objects in different spatial locations, enable the inference of all salient targets even when multiple targets are far apart in spatial location.

[0049] Specifically, the first step is to construct the image, focusing on spatiotemporal depth features. Projecting the spatiotemporal depth features Projecting from coordinate space to graph space, using a convolutional layer with a 1×1 kernel... Convert to low-dimensional features Use f p The function is used to convert low-dimensional features into graph node representations. f p Functions can be derived from matrices Parametric representation, each column w of W k For each graph node v k A learnable cluster center is specified. Here, it is established through soft assignment. Each feature vector f in i With each column w in W k Contact As shown in equation (5).

[0050]

[0051] Where / represents element-wise division, σ k represents the sum of column vectors, exp(·) represents the exponential function with base e, and ||·||2 represents the formula for calculating the L2 norm.

[0052] The projected graph node v is calculated using equation (6-7). k .

[0053]

[0054]

[0055] in, The eigenvector f obtained from equation (5) above represents the eigenvector f. i with w k soft allocation, v′ k Represents the eigenvector f i with w k The weighted average of the residuals, v k express The k-th graph node in the graph.

[0056] Next, the graph node features are updated through two layers of graph convolution, as shown in equation (3).

[0057] Finally, the graph nodes are connected via f Rp The function is back-projected into the coordinate space to obtain the updated features. In form, f Rp A function can be parameterized as The updated features are added to the original spatiotemporal depth features to generate the internally self-corrected features F. i As shown in equation (8).

[0058]

[0059] Example 1:

[0060] We implemented our method in PyTorch. We used pre-trained ResNet-50 and ResNet-34 on ImageNet as our initial backbone. We used RAFT to generate optical flow images. First, we pre-trained the RGB branches using the training set of the DUTS dataset, and then pre-trained the optical flow branches using the optical flow maps from the DAVIS and DAVSOD training sets as input. After the backbone network pre-training was complete, we loaded the trained parameters and trained the entire network again using the training sets from the DAVIS and DAVSOD datasets. We augmented the training data by randomly flipping and rotating the images to prevent overfitting. We used the Adam optimizer and trained the model with an initial learning rate of 1e-5 until convergence.

[0061] Experimental data

[0062] This paper compares the proposed model with other state-of-the-art models, namely SGSP, STBP, SFLR, SCNN, SCOM, FGRNE, MBNM, PDB, SSAV, MGA, PCSA, MAGCN, STFA, GTNet, DCFNet, and CAG-DDE. Quantitative experimental results on four datasets—DAVIS, DAVSOD, ViSal, and SegV2—are presented, including MAE, max F-measure, and S-measure. Figure 4As shown. In the field of video salient object detection, the above four datasets are publicly available standard datasets, and the three metrics mentioned above can comprehensively evaluate the model's performance. It is worth noting that we highlight the top three data points on each dataset: dark gray indicates the first-ranked model, light gray indicates the second-ranked model, and black indicates the third-ranked model. The model outperforms existing similar models on the DAVIS and DAVSOD datasets. On DAVIS, the max F-measure is 1 percentage point ahead of the second-place model, and on DAVSOD, the max F-measure is 2.6 percentage points ahead of the second-place model. Furthermore, on the SegV2 dataset, the model ranks second in all metrics with a small gap from the first-place model. On the ViSal dataset, it also achieves comparable performance to similar models. In summary, this demonstrates the model's advantages in complex and dynamic scenes.

Claims

1. A video salient object detection method based on a graph model, characterized in that... Includes the following steps: (1) A video salient object detection model based on graph convolutional networks is proposed to fully exploit spatial and temporal information; (2) Design a layer-by-layer interaction module based on graph convolution, take the spatiotemporal features of different layers as graph nodes, construct edges based on the distance information between cross-modal features and cross-layer features, update the features of graph nodes through graph convolution, and fuse cross-modal features and cross-layer features in the channel dimension to generate spatiotemporal depth features; (3) Design a feature self-correction module based on graph convolution to map the spatiotemporal depth features in the spatial dimension. Corresponding graph nodes represent the corresponding spatial regions. The features of the graph nodes are updated through graph convolution, thereby modeling the semantic relationship in space, strengthening the connection between salient regions, and fully highlighting the salient target.

2. The video salient object detection method based on a graph model according to claim 1, characterized in that... Step (1) is implemented as follows: The video salient object detection model described above adopts an encoder-decoder structure; The input to the encoder section consists of three images: the RGB image of the current frame, the RGB image of the next frame, and the optical flow map corresponding to the current frame. The encoder consists of three backbone networks that extract features corresponding to the three images respectively. Among them, two ResNet-50 backbone networks are used to extract the RGB image features of the current frame and the RGB image features of the next frame, respectively, and a ResNet-34 backbone network is used to extract the optical flow map features corresponding to the current frame. The adjacent next frame RGB image is added as a supplement to the motion information. After initial extraction by the backbone network, the features at each level are passed through a dimensionality reduction module consisting of a 3×3 convolutional layer, a BN layer, and a ReLU activation layer, reducing the number of feature channels to 64. For the deep features of the fifth layer of the backbone network, they are connected to the ASPP module to fuse rich multi-scale features. The encoder part obtains the spatiotemporal features of the current frame RGB image, the next frame RGB image, and the optical flow map corresponding to the current frame, respectively. , , ,in This is the number of layers in the backbone network, with a value ranging from 1 to 5.

3. The video salient object detection method based on a graph model according to claim 2, characterized in that... Step (2) is implemented as follows: A graph convolution-based inter-layer interaction module fuses the spatiotemporal features extracted from adjacent layers of the backbone network to generate spatiotemporal depth features; specifically, the input is... , , , , , , , , The output is .

4. The video salient object detection method based on a graph model according to claim 3, characterized in that... Step (2) is implemented in detail as follows: For the features of the first layer: since it has no adjacent shallow layer features, the features of the first layer are repeatedly input when inputting the graph convolution-based inter-layer interaction module, i.e. , , , , , , , , ; For the features of the second layer: the input is , , , , , , , , ; Similarly, the features of the third and fourth layers are consistent with the features of the second layer; For the features of the fifth layer: since it has no adjacent deep features, the features of the fifth layer are repeatedly input when inputting the graph convolution-based inter-layer interaction module. , , , , , , , , .

5. The video salient object detection method based on a graph model according to claim 3, characterized in that... The specific steps of the graph convolution-based inter-layer interaction module, which takes the spatiotemporal features of adjacent layers as input and outputs spatiotemporal depth features, are as follows: First, features are processed using global average pooling. Compress and transform in spatial dimension to As shown in equation (1), each compressed feature is treated as a graph node; Secondly, graph convolution inference is performed. The information of the edges is established by calculating the cosine distance between graph nodes, and the adjacency matrix is ​​constructed using the cosine distance, as shown in Equation (2). The graph nodes are iteratively updated according to the general formula of graph convolution, and the number of graph convolution layers is set to 2, as shown in Equation (3). Next, for the updated graph node, it is multiplied with the original spatiotemporal features of the corresponding input, so that the original spatiotemporal features are strengthened or weakened in some channels. Then, after splicing and convolution operations, the spatiotemporal features are generated after preliminary fusion. Finally, to strengthen the central features, namely the intermediate spatial features in the input features of the inter-level interaction module based on graph convolution, it is represented as: The residual structure is adopted to add the central feature to the spatiotemporal feature after preliminary fusion to generate the spatiotemporal depth feature, as shown in Equation (4); (1) (2) (3) (4) in, Indicates input features, This represents the processed graph node features. This represents a global average pooling operation. This is represented as a convolution operation with a 1×1 kernel. The magnitude of the eigenvector is represented by its eigenvector. Represents the adjacency matrix. Represents a non-linear activation function. This represents the degree matrix generated from the adjacency matrix. This represents the node features of the l-th graph convolutional layer. Represents the weight matrix. This represents a splicing operation. It represents the spatiotemporal depth feature.

6. A video salient object detection method based on a graph model according to claim 1, 2, 3, 4, or 5, characterized in that... Step (3) is implemented as follows: First, the composition is established, focusing on spatiotemporal depth features. Projecting the spatiotemporal depth features Projecting from coordinate space to graph space, using a convolutional layer with a 1×1 kernel... Convert to low-dimensional features ,use The function is used to convert low-dimensional features into graph node representations. , The function can be derived from the matrix Parametric representation, Each column For each graph node A learnable cluster center was specified; clustering was established through soft assignment. Each feature vector in and Each column Contact As shown in equation (5); (5) in, This indicates that corresponding elements are divided. Represents the sum of column vectors. This represents an exponential function with base e. The formula for calculating the L2 norm; Calculate the projected graph nodes using equations (6)-(7). ; (6) (7) in, The eigenvector obtained from equation (5) above represents the eigenvector. and soft allocation, Representing the eigenvector and The weighted average of the residuals express The k-th graph node in the graph; Next, the graph node features are updated through two layers of graph convolution; Finally, the graph nodes are passed through The function is projected back onto the coordinate space to obtain the updated features; formally, A function can be parameterized as The updated features are added to the original spatiotemporal depth features to generate internally self-corrected features. As shown in equation (8): (8)。

Images