A video salient object detection method and system based on bimodal cross fusion and multi-graph interactive learning

Abstract

The application provides a video salient object detection method and system based on bimodal cross fusion and multi-graph interactive learning. The method can realize effective fusion of space-time cross-modal features by mutual constraint of spatial features and motion features, and effectively learn and infer the complex logical relationship between sub-regions by using the advantages of graph neural networks, capture rich space-time context structure information, and finally improve the overall performance of video salient object detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a video saliency object detection method and system based on bimodal cross-fusion and multi-graph interactive learning. Background Technology

[0002] Video is an important medium for recording information, conveying ideas, and expressing emotions, providing us with a realistic and objective representation of the world by recording real, dynamic scenes. However, the large amount of redundant information contained in video data makes it characterized by low value density, which poses a significant challenge to intelligent video analysis and places a heavy burden on the computing resources upon which the analysis depends.

[0003] Video salient object detection, as an effective way to address the aforementioned challenges, is a fundamental and cutting-edge problem in the field of intelligent video analytics. Its essence lies in using algorithms to analyze the semantic relationships between objects in dynamic visual scenes based on human visual attention mechanisms, distinguishing the importance of different objects, and thus locating the most distinctive and attention-grabbing objects and separating them from the background. As an effective foundational step in video processing, video salient object detection not only improves the performance of subsequent high-level visual understanding tasks but also guides the rational allocation of computing resources, allocating limited computing resources to complex computational tasks, thereby efficiently completing video information processing tasks in application domains. For example, in video target tracking tasks, changes in target pose, appearance, and sudden changes in position can significantly impact tracker performance. Class-independent video salient object detection can provide target knowledge, helping the tracker quickly and accurately recover targets that have failed to be tracked. Similarly, in video understanding tasks, different objects in a scene contribute differently to understanding the video content. Video salient object detection can distinguish the importance of different objects, thereby improving the ability to abstractly understand the video content.

[0004] Therefore, it can be seen that video salient object detection, as one of the common basic research problems in the field of video processing, can be widely applied to many cutting-edge fields of national economic development such as video surveillance, target detection and tracking, multimedia information understanding, and social media, and plays an important role in promoting the information intelligence process of society.

[0005] Video salient object detection aims to identify and segment the most interesting and motion-related object regions in dynamic visual scenes. Unlike image salient object detection, video salient object detection requires not only capturing spatial appearance information but also fully extracting temporal motion cues. In recent years, the wave of deep learning has driven in-depth research in computer vision, leading to the emergence of many deep learning-based video salient object detection methods. Even so, fully extracting and integrating information from spatial and motion cues remains a significant challenge for current video salient object detection methods. Therefore, how to extract salient cues from spatial and motion modalities has been a highly focused research topic in video salient object detection. However, existing video salient object detection methods typically only use motion features to guide the learning process of spatial features, or blindly fuse spatial and motion features. Due to the incomplete consideration of learning strategies, these methods perform poorly in complex scenes. In addition, most video saliency detection models are built on convolutional neural networks. Convolutional neural networks expand their receptive field by stacking multiple pooling layers and convolutional layers. This continuous downsampling operation inevitably loses a lot of high-frequency information such as edges and structures, resulting in problems such as missing parts of objects in the saliency detection results.

[0006] Therefore, designing a video salient object detection model based on bimodal cross-modal feature fusion and multi-graph interactive learning is of great research significance in addressing the aforementioned problems of effective fusion of spatiotemporal cross-modal features and sufficient capture of salient structural information. Summary of the Invention

[0007] The purpose of this invention is to solve the problems in the prior art and to propose a video saliency object detection method and system based on dual-modal cross-fusion and multi-graph interactive learning.

[0008] This invention is achieved through the following technical solution: This invention proposes a video saliency object detection method based on dual-modal cross-fusion and multi-graph interactive learning, the method specifically being:

[0009] Step 1: Take the RGB image of frame t and its corresponding optical flow image as input, and extract spatial and motion features through two independent ResNet50 backbone networks respectively.

[0010] Step 2: Utilize the cross-attention module to effectively fuse the spatial and motion features output from the K feature extraction layers in the backbone network across spatiotemporal modalities;

[0011] Step 3: Use a multi-scale feature fusion module to capture multi-scale feature cues from the spatial feature extraction branch, the temporal feature extraction branch, and the dual-modal cross-fusion branch to improve the overall robustness of the method.

[0012] Step 4: Perform graph projection operations on the multi-scale spatial features, multi-scale motion features, and multi-scale fusion features respectively to transform them into non-Euclidean space to obtain the corresponding map structure data, namely spatial map, motion map, and fusion map.

[0013] Step 5: Input the spatial graph, motion graph and fused graph into the multi-graph interactive learning network for processing. The multi-relation dynamic attention graph convolution operator captures richer spatiotemporal context structure information, and the graph interaction module maximizes the interaction of spatiotemporal information.

[0014] Step 6: Finally, the fused graph, which is the final output of the node embedding information mining process, is subjected to graph reprojection to restore it to the original Euclidean space. Then, after processing by a fully connected layer, the final saliency detection result is generated.

[0015] Furthermore, in step 2, for the k-th layer, global average pooling is first used to extract features from each spatial feature. and motion characteristics Extracting channel-level vectors and Then, through two layers of 1×1 convolution operations, i.e. and θ t (x;W θ ),in and W θ These represent the learnable parameters of the two convolutional layers, generating two discriminative global descriptors, which are then applied using the Sigmoid function. Transform the element values ​​of these two descriptors to the [0,1] interval, and use them as channel attention weights; then, for and Through outer product operation Generate candidate features

[0016]

[0017] Similarly, for and Through outer product operation Generate candidate features

[0018]

[0019] Finally, in order to extract deep fusion features, the two candidate features are... and and the output features of the previous layer cross-attention module The fusion is performed in the k-th feature extraction layer R of ResNet50. k Element-wise addition is performed in [x]. Obtain fusion features that include spatiotemporal correlations.

[0020]

[0021] in, This represents a 0 tensor, with K set to 4.

[0022] Furthermore, in step 3, the output features of the last four feature extraction layers of the ResNet50 backbone network are first used as input, and they are represented from low to high as follows: Then, the high-level features are propagated down to the low-level features step by step, and gradually fused with the low-level features. The fusion process can be represented as:

[0023]

[0024] in, Conv represents the fusion feature output of the (i-1)th stage. 3×3 (·) represents a 3×3 convolution operation, and Up(·) represents an upsampling operation; finally, the fused feature outputs from the three stages are fused through a concatenation operation to obtain the final multi-scale fused feature output.

[0025]

[0026] Concat(·) represents a concatenation operation.

[0027] Furthermore, in step 4, a 1×1 convolution operation is first used to reduce the dimensionality of each feature, transforming them into low-dimensional features. as well as Then these three features are respectively subjected to graph projection operation f proj (·) is transformed into the corresponding spatial graph node. Motion graph nodes and fusion graph nodes Right now:

[0028]

[0029] Through the above methods, spatial maps motion graph and fusion diagram Construction complete, including as well as These represent the set of nodes in the spatial graph, the set of nodes in the motion graph, and the set of nodes in the fused graph, respectively. as well as These represent the set of edges in the spatial graph, the set of edges in the motion graph, and the set of edges in the fused graph, respectively.

[0030] Furthermore, in step 5, the spatial map and motion graph In both static and motion feature spaces, a multi-relation dynamic attention graph convolution operator is used to mine potential semantic relationships and local / global graph structure information between paired graph nodes, updating the embedding information of the graph nodes. Then, the outputs of the spatial graph node embedding information mining process and the corresponding motion graph node embedding information mining process are simultaneously fed into the fusion graph node embedding information mining process enhanced by the graph interaction module. Spatial and motion graph nodes are adaptively selected to fuse and encode spatiotemporal semantic features.

[0031] Furthermore, in step 6, the spatial graph nodes and motion graph nodes obtained in step 5 are processed by two layers of multi-relation dynamic attention graph convolution and fused by the corresponding map interaction module in the same manner, and then a graph reprojection operation is performed. Embedding the nodes of the fused graph into the fused graph output from the information mining process Mapping back to the original Euclidean space, i.e.:

[0032]

[0033] The above method achieves the mapping from non-Euclidean space graph structure data to Euclidean space pixel-level feature maps, thus preparing for the output of the final pixel-level saliency prediction map.

[0034] Furthermore, in step 6, after processing by the fully connected layer, the final saliency prediction result is generated. Right now:

[0035]

[0036] This invention proposes a video saliency object detection system based on dual-modal cross-fusion and multi-graph interactive learning, the system comprising:

[0037] Cross-attention module: The RGB image of frame t and its corresponding optical flow image are taken as input, and spatial and motion features are initially extracted through two independent ResNet50 backbone networks. The cross-attention module is used to effectively fuse the spatial and motion features output from K feature extraction layers in the backbone network across spatiotemporal modalities.

[0038] Multi-scale feature fusion module: A multi-scale feature fusion module is used to capture multi-scale feature cues from the spatial feature extraction branch, the temporal feature extraction branch, and the dual-modal cross-fusion branch, respectively, to improve the overall robustness of the method;

[0039] The graph interaction module performs graph projection operations on multi-scale spatial features, multi-scale motion features, and multi-scale fusion features respectively, transforming them to non-Euclidean space to obtain corresponding map structure data, namely spatial graph, motion graph, and fusion graph. The spatial graph, motion graph, and fusion graph are simultaneously input into a multi-graph interactive learning network for processing. The multi-relation dynamic attention graph convolution operator captures richer spatiotemporal contextual structure information, while the graph interaction module maximizes the interaction of spatiotemporal information. Finally, the fusion graph, which is the final output of the node embedding information mining process in the fusion graph, undergoes graph reprojection to restore it to the original Euclidean space. After processing by a fully connected layer, the final saliency detection result is generated.

[0040] The beneficial effects of this invention are as follows:

[0041] The present invention proposes a video salient object detection method based on bimodal cross-fusion and multi-graph interactive learning. By mutually constraining spatial and motion features, it can achieve effective fusion of spatiotemporal cross-modal features and leverage the advantages of graph neural networks to effectively learn and infer the complex logical relationships between sub-regions, capture rich spatiotemporal contextual structural information, and ultimately improve the overall performance of video salient object detection. Attached Figure Description

[0042] Figure 1 The method proposed in this invention compares with 11 existing advanced methods on four standard datasets. α Histograms of maxF and MAE indices.

[0043] Figure 2 The graph shows the PR curves of the proposed method and 11 existing advanced methods on four standard datasets.

[0044] Figure 3 This is a visual comparison example of the method proposed in this invention and existing advanced methods.

[0045] Figure 4 Here is a flowchart of the method.

[0046] Figure 5 This is a schematic diagram of the cross-attention module structure.

[0047] Figure 6 This is a schematic diagram of the multi-scale feature fusion module structure.

[0048] Figure 7This is a schematic diagram of the structure of a multi-relational dynamic attention graph convolution operator.

[0049] Figure 8 This is a schematic diagram of the interactive module structure. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention proposes a video saliency object detection method based on dual-modal cross-fusion and multi-graph interactive learning. The method specifically comprises:

[0052] Step 1: First, extract the RGB image I of frame t. t and its corresponding optical flow image O t (Obtained from the RGB image of frame t and the RGB image of frame t+1 via FlowNet2.0) are fed into two independent ResNet50 backbone networks for spatial and motion feature extraction;

[0053] Step 2: Extract spatial features from the K feature extraction layers in the backbone network. and motion characteristics Simultaneously, the data is fed into the Cross-Attention Module (CAM) to achieve spatiotemporal cross-modal feature fusion. Specifically, for the k-th layer, Global Average Pooling (GAP) is first used to extract features from each spatial feature. and motion characteristics Extracting channel-level vectors and Then, through two layers of 1×1 convolution operations, i.e. and θ t (x;W θ ),in and W θ These represent the learnable parameters of the two convolutional layers, generating two discriminative global descriptors, which are then applied using the Sigmoid function. The element values ​​of these two descriptors are transformed to the [0,1] interval and used as channel attention weights. Then, for and Through outer product operation Generate candidate features

[0054]

[0055] Similarly, for and Through outer product operation Generate candidate features

[0056]

[0057] Finally, in order to extract deep fusion features, the two candidate features are... and and the output features of the previous layer cross-attention module Fusion is performed. In the k-th feature extraction layer R of ResNet50... k Element-wise addition is performed in [x]. Obtain fusion features that include spatiotemporal correlations.

[0058]

[0059] It is worth noting that, This represents the 0 tensor. As a rule of thumb, K is set to 4.

[0060] Step 3: A multi-scale feature fusion module (MFFM) is used to capture multi-scale feature cues from the spatial feature extraction branch, the temporal feature extraction branch, and the bimodal cross-fusion branch, thereby improving the overall robustness of the method. Specifically, the output features of the last four feature extraction layers of the ResNet50 backbone network are first used as input, and they are represented from low to high as follows: Then, the high-level features are propagated down to the low-level features step by step, and gradually fused with the low-level features. The fusion process can be represented as:

[0061]

[0062] in, Conv represents the fusion feature output of the (i-1)th stage. 3×3 (·) represents a 3×3 convolution operation, and Up(·) represents an upsampling operation. Finally, the fused feature outputs from the three stages are fused through a concatenation operation to obtain the final multi-scale fused feature output.

[0063]

[0064] Concat(·) represents a concatenation operation.

[0065] Step 4: Multi-scale spatial features Multiscale motion characteristics and multi-scale fusion features respectively through graph projection operation f proj (·) Model the data as a corresponding graph structure. Specifically, firstly, use 1×1 convolution operations to reduce the dimensionality of each data point, transforming them into low-dimensional features. as well as Then these three features are respectively subjected to graph projection operation f proj (·) is transformed into the corresponding spatial graph node. Motion graph nodes and fusion graph nodes Right now:

[0066]

[0067] Through the above methods, spatial maps motion graph and fusion diagram Construction complete, including as well as These represent the set of nodes in the spatial graph, the set of nodes in the motion graph, and the set of nodes in the fused graph, respectively. as well as These represent the set of edges in the spatial graph, the set of edges in the motion graph, and the set of edges in the fused graph, respectively.

[0068] Step 5: Simultaneously input the constructed three graph structures into the Multi-Graph Mutual Learning Network (MGMLN) for processing. Specifically, the spatial graph... and motion graph In both the static and motion feature spaces, the Multi-Relations Dynamic Attention Graph Convolution (MDAGC) operator is used to mine potential semantic relationships and local / global graph structure information between paired graph nodes, updating the embedding information of the graph nodes. Then, the outputs of the spatial graph node embedding information mining process and the corresponding motion graph node embedding information mining process are simultaneously fed into the fused graph node embedding information mining process enhanced by the Graph Interaction Module (GIM). Spatial and motion graph nodes are adaptively selected to fuse and encode spatiotemporal semantic features.

[0069] Step 6: After processing the spatial graph nodes and motion graph nodes obtained in Step 5 through two layers of multi-relation dynamic attention graph convolution and the fusion processing of the corresponding map interaction module, a graph reprojection operation is performed. Embedding the nodes of the fused graph into the fused graph output from the information mining process Mapping back to the original Euclidean space, i.e.:

[0070]

[0071] The above method maps non-Euclidean graph structure data to Euclidean pixel-level feature maps, preparing for the output of the final pixel-level saliency prediction map. Finally, after processing by a fully connected layer (FCL), the final saliency prediction result is generated. Right now:

[0072]

[0073] Performance tests were conducted on four publicly available, representative, and challenging test datasets: DAVIS, ViSal, SegTrack-V2, and DAVSOD. To ensure fairness in comparisons with existing video salient object detection methods, all competing methods were generated from published source code with default parameters or obtained from publicly available results. The 11 video salient object detection methods based on deep convolutional neural networks included in the comparison are SCNN, SCOM, PDB, SSAV, LSTI, PCSA, STVS, DCFNet, FSNet, ALGRF, and SCANet. α The comparison results of the three evaluation indicators, maxF and MAE, are as follows: Figure 1 As shown.

[0074] from Figure 1 It is clear that the method proposed in this invention outperforms other state-of-the-art methods on all performance metrics of the DAVIS and SegTrack-V2 datasets, and ranks among the top three on the ViSal and DAVSOD datasets. Specifically, on the SegTrack-V2 dataset, compared to the second-ranked comparison, the evaluation metric S... α The maximum efficiency, maximum efficiency, and maximum efficiency (MAE) increased by 1.5%, 0.8%, and 13.3%, respectively.

[0075] Figure 2 For comparison results based on PR curves, from Figure 2As can be seen from the PR curves, the method proposed in this invention performs comparably to other video salient object detection methods on the ViSal dataset, SegTrack-V2 dataset, and DAVSOD dataset, and achieves better performance results than 11 other state-of-the-art comparison methods on the DAVIS dataset.

[0076] In addition to the quantitative analysis experiments described above, this invention also selected some representative test data for visualization comparison experiments to more intuitively demonstrate the performance comparison between the method proposed in this invention and existing advanced methods. Figure 3 This paper presents visual comparison examples of the proposed method and 11 advanced comparative methods in some challenging and complex scenarios, demonstrating the performance comparison of the proposed method and the comparative methods in a more intuitive way. These include some highly challenging dynamic visual scenarios, such as low foreground-background contrast (columns 1-2), background motion (columns 3-4), object deformation (columns 5-8), and the presence of multiple salient objects (columns 9-10). Figure 3 As shown in the visual examples of saliency maps in these complex scenes, it can be seen that, compared with the method proposed in this invention, most existing state-of-the-art video salient object detection methods have failed to correctly segment salient object regions, exhibiting problems such as blurred edges of salient objects, poor detection integrity, and incorrect highlighting of complex backgrounds. However, the proposed method can accurately predict the most salient objects and coherent details in complex dynamic visual scenes. These visual comparison results further demonstrate the excellent detection capability of the proposed method.

[0077] This invention proposes a video saliency object detection method based on bimodal cross-fusion and multi-graph interactive learning, the overall design of which is as follows: Figure 4 As shown, the detailed process is as follows:

[0078] (1) First, the RGB image of frame t and its corresponding optical flow image are used as input, and spatial features and motion features are initially extracted through two independent ResNet50 backbone networks respectively.

[0079] (2) The cross-attention module is used to effectively fuse the spatial and motion features output from the K feature extraction layers in the backbone network across spatiotemporal modal features.

[0080] (3) Furthermore, a multi-scale feature fusion module is used to capture multi-scale feature cues from the spatial feature extraction branch, the temporal feature extraction branch, and the dual-modal cross-fusion branch, respectively, to improve the overall robustness of the method;

[0081] (4) Perform graph projection operations on the multi-scale spatial features, multi-scale motion features and multi-scale fusion features respectively to transform them into non-Euclidean space in order to obtain the corresponding map structure data, namely spatial map, motion map and fusion map.

[0082] (5) Input the spatial graph, motion graph and fusion graph into the multi-graph interactive learning network for processing. Capture richer spatiotemporal context structure information through the multi-relation dynamic attention graph convolution operator, and maximize the spatiotemporal information interaction by using the graph interaction module.

[0083] (6) Finally, the fusion graph, which is the final output of the fusion graph node embedding information mining process, is restored to the original Euclidean space by graph reprojection operation. Then, after being processed by a fully connected layer, the final saliency detection result is generated.

[0084] This invention proposes a video saliency object detection system based on dual-modal cross-fusion and multi-graph interactive learning, the system comprising:

[0085] Cross-attention module: The RGB image of frame t and its corresponding optical flow image are taken as input, and spatial and motion features are initially extracted through two independent ResNet50 backbone networks. The cross-attention module is used to effectively fuse the spatial and motion features output from K feature extraction layers in the backbone network across spatiotemporal modalities.

[0086] Multi-scale feature fusion module: A multi-scale feature fusion module is used to capture multi-scale feature cues from the spatial feature extraction branch, the temporal feature extraction branch, and the dual-modal cross-fusion branch, respectively, to improve the overall robustness of the method;

[0087] The graph interaction module performs graph projection operations on multi-scale spatial features, multi-scale motion features, and multi-scale fusion features respectively, transforming them to non-Euclidean space to obtain corresponding map structure data, namely spatial graph, motion graph, and fusion graph. The spatial graph, motion graph, and fusion graph are simultaneously input into a multi-graph interactive learning network for processing. The multi-relation dynamic attention graph convolution operator captures richer spatiotemporal contextual structure information, while the graph interaction module maximizes the interaction of spatiotemporal information. Finally, the fusion graph, which is the final output of the node embedding information mining process in the fusion graph, undergoes graph reprojection to restore it to the original Euclidean space. After processing by a fully connected layer, the final saliency detection result is generated.

[0088] The functions of each module are as follows:

[0089] Cross-attention module: To fully utilize the mutual constraints between spatial and motion features, a cross-attention module based on the channel attention mechanism is designed, with the specific structure as follows: Figure 5As shown, this module extracts valid cues from two different modalities, and then uses the features of the other modality to constrain the current feature.

[0090] The specific workflow is as follows:

[0091] (1) Spatial features obtained from two independent ResNet50 backbone networks and motion characteristics As input. Taking the k-th layer as an example, firstly, Global Average Pooling (GAP) is used to extract each spatial feature S. k and motion characteristics M k Extracting the channel-level vector V k S and V k M ;

[0092] (2) Through two layers of 1×1 convolution operations, i.e. and θ(x; W) θ ),in and W θ These represent the learnable parameters of the two convolutional layers, generating two discriminative global descriptors, which are then applied using the Sigmoid function. The element values ​​of these two descriptors are transformed to the [0,1] interval and used as channel attention weights.

[0093] (3) Then, for S k and Through outer product operation Generate candidate features

[0094]

[0095] Similarly, for M k and Through outer product operation Generate candidate features

[0096]

[0097] (4) Finally, in order to extract deep fusion features, we examine the two candidate features. and and the output features of the previous layer cross-attention module Fusion is performed. In the k-th feature extraction layer R of ResNet50... k Element-wise addition is performed in [x]. Obtain fusion features that include spatiotemporal correlations.

[0098]

[0099] It is worth noting that, This represents the 0 tensor. Based on experience, K is set to 4.

[0100] Multi-scale feature fusion module: Considering that high-level features of the network mainly contain abstract global semantic information, which is beneficial for locating salient object regions, while low-level features mainly contain more local detail information, which is beneficial for sharpening the edges of salient objects, fusing multi-level features is essential for generating accurate saliency prediction maps. This invention designs a multi-scale feature fusion module to fully fuse features from multiple scales, with the specific structure as follows: Figure 6 As shown.

[0101] The specific workflow is as follows:

[0102] (1) The output features of the last four feature extraction layers of the ResNet50 backbone network are used as input, and they are represented from low to high as B1, B2, B3, and B4, respectively. The high-level features are propagated to the low-level features step by step and gradually fused with the low-level features. The fusion process can be represented as follows:

[0103] B i ' -1 =Conv 3×3 (Up(Conv 3×3 (B i ))+Up(Conv 3×3 (B i-1 )),i∈{2,3,4}

[0104] Among them, B i ' -1 ,i∈{2,3,4} represents the fusion feature output of the (i-1)th stage, Conv 3×3 (·) represents a 3×3 convolution operation, and Up(·) represents an upsampling operation.

[0105] (2) The fusion feature outputs of the three stages are fused through a concatenation operation to obtain the final multi-scale fusion feature output F. M :

[0106] F M =Concat(B1',B2,B3')

[0107] Concat(·) represents a concatenation operation.

[0108] Multi-Relation Dynamic Attention Graph Convolution Operator: To fully and effectively mine the embedded information of graph nodes, this invention designs a novel multi-relation dynamic attention graph convolution operator. By considering the various implicit semantic relationships between different graph nodes, it enhances the information transmission between them. Simultaneously, under a certain semantic relationship, an attention mechanism is used to adaptively assign weights to different neighboring graph nodes, achieving effective aggregation of neighboring graph node information. The specific structure is as follows: Figure 7 As shown.

[0109] The specific workflow is as follows:

[0110] (1) Calculate the k nearest neighbor graph nodes for each graph node using Euclidean distance;

[0111] (2) Learning functions by side To learn the current graph node v i Its neighboring graph nodes The size of the edge weights between To fully explore the implicit semantic relationships between the current graph node and its neighboring graph nodes, and enhance information interaction between different graph nodes, the specific calculation process is as follows:

[0112]

[0113] Where, r learn R represents the relation type index, and R represents the total number of relations to be learned. Indicates the current graph node v i The set of neighboring graph nodes, learning function A multilayer perceptron (MLP) can be used for fitting and learning, defined as:

[0114]

[0115] (3) at the rth learn Calculate neighbor graph nodes under various semantic relations. For the current graph node v i Importance weights

[0116]

[0117] (4) Finally, by weighted aggregation of the embedding information of neighboring graph nodes under different semantic relations, the embedding information of the current graph nodes is updated, that is:

[0118]

[0119] Graph Interaction Module: Embeds graph nodes from the l-th layer output during the spatial graph node embedding information mining process. The graph nodes output from layer l during the motion graph node embedding information mining process Effective fusion can provide complementary clues for fully mining salient spatiotemporal information. For example, introducing salient spatial clues can compensate for the missing static parts of salient moving objects; similarly, when the contrast between the foreground and background is low, introducing salient motion clues can suppress the interference of the static background. Therefore, this invention designs a graph interaction module, the specific structure of which is as follows: Figure 8 As shown.

[0120] The specific workflow is as follows:

[0121] (1) First, a multilayer perceptron (MLP) is used. S (·) and MLP S (·) Map spatial graph node embeddings and motion graph node embeddings to a more abstract graph embedding space:

[0122]

[0123]

[0124] (2) Then, measure the affinities between nodes in each pair of graphs:

[0125] W att =softmax(S lT M l )

[0126] Among them, W att ∈R N×N It is an affinity matrix that encodes pairwise relationships in graph embeddings. The softmax(·) function modulates the matrix S. lT M l The results are normalized column-wise, with each element representing the correlation coefficient from one channel to other channels. Since the above formula encodes the kinship between graph nodes by evaluating the relationships between embedded channels, this module fully utilizes saliency spatial cues and saliency motion cues from a more global perspective, at the cost of occupying some parameter space.

[0127] (3) After this, salient motion cues are passed to the spatial graph nodes:

[0128]

[0129] To improve the stability of features within graph nodes, a residual structure is used to combine the features of the original spatial graph nodes. and

[0130]

[0131] Among them, α control Features of nodes in the original spatial graph Based on experience, α is set to 0.3 to account for the impact of [the effect].

[0132] (4) Similarly, saliency spatial cues are passed to the nodes of the motion graph:

[0133]

[0134] To improve the stability of the internal features of graph nodes, a residual structure is used to combine the features of the original moving graph nodes. and

[0135]

[0136] Among them, β control Features of nodes in the original motion graph Based on experience, β is set to 0.3 to account for the impact of [the effect].

[0137] (5) Finally, the spatial graph nodes containing salient motion cues and motion graph nodes containing salient spatial cues The data are then added together and fused, and fed into the l-th layer of the fusion graph node embedding information mining process to further encode salient spatiotemporal semantic information.

[0138] The method proposed in this invention is based on the PyTorch deep learning framework and implemented on an NVIDIA TITAN RTX (24G) using the PyTorch Geometric (PyG) machine learning library. The proposed method model is trained in stages using image saliency object detection datasets and video saliency object detection datasets. In the first stage, the spatial feature extraction branch is trained using the image saliency object detection training set DUTS-TR and the video saliency object detection training sets DAVIS (containing 30 video sequences) and FBMS (containing 29 video sequences) to avoid overfitting. The training iterations in this stage are set to 50 and the batch size to 8. In the second stage, the motion feature extraction branch is trained using the optical flow images corresponding to the video saliency object detection training sets DAVIS and FBMS, where the optical flow images are generated by FlowNet2.0. The training iterations in this stage are set to 50 and the batch size to 8. In the third stage, the weights pre-trained in the first and second stages are loaded into the spatial feature extraction branch and the motion feature extraction branch. Then, the entire network model is trained end-to-end using the video saliency object detection training sets DAVIS and FBMS, including the original RGB images and the corresponding optical flow images. The training iterations in this stage are set to 20 and the batch size to 8. In each stage, the entire network model is optimized using the Stochastic Gradient Descent (SGD) algorithm, with a momentum of 0.9, a learning rate of 2e-3, and a weight decay of 5e-4. The learning rate decreases by 10% every 20 training iterations. All input images are resized to 352×352. To enhance the stability and generalization of the proposed learning method, a multi-scale training strategy ({0.75, 1, 1.25}) is employed during the training phase. Quantitative and qualitative experiments are conducted on four publicly available benchmark datasets: DAVIS, ViSal, SegTrack-V2, and DAVSOD. To ensure fairness in comparison with existing video salient object detection methods, all competing methods are generated from published source code with default parameters or obtained from publicly available results.

[0139] The foregoing has provided a detailed description of the video saliency object detection method and system based on bimodal cross-fusion and multi-graph interactive learning proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A video salient object detection method based on bimodal cross-fusion and multi-graph interactive learning, characterized in that: The method is specifically as follows: Step 1: Take the RGB image of frame t and its corresponding optical flow image as input, and extract spatial and motion features through two independent ResNet50 backbone networks respectively. Step 2: Utilize the cross-attention module to effectively fuse the spatial and motion features output from the K feature extraction layers in the backbone network across spatiotemporal modalities; Step 3: Use a multi-scale feature fusion module to capture multi-scale feature cues from the spatial feature extraction branch, the temporal feature extraction branch, and the dual-modal cross-fusion branch to improve the overall robustness of the method. Step 4: Perform graph projection operations on the multi-scale spatial features, multi-scale motion features, and multi-scale fusion features respectively to transform them into non-Euclidean space to obtain the corresponding map structure data, namely spatial map, motion map, and fusion map. Step 5: Input the spatial graph, motion graph and fused graph into the multi-graph interactive learning network for processing. The multi-relation dynamic attention graph convolution operator captures richer spatiotemporal context structure information, and the graph interaction module maximizes the interaction of spatiotemporal information. Step 6: Finally, the fused graph, which is the final output of the node embedding information mining process, is subjected to graph reprojection to restore it to the original Euclidean space. Then, after processing by a fully connected layer, the final saliency detection result is generated.

2. The method according to claim 1, characterized in that, In step 2, for the k-th layer, global average pooling is first used to extract features from each spatial feature. and motion characteristics Extracting channel-level vectors and Then, through two layers of 1×1 convolution operations, i.e. and θ t (x;W θ ),in and W θ These represent the learnable parameters of the two convolutional layers, generating two discriminative global descriptors, which are then applied using the Sigmoid function. Transform the element values ​​of these two descriptors to the [0,1] interval, and use them as channel attention weights; then, for and Through outer product operation Generate candidate features Similarly, for and Through outer product operation Generate candidate features Finally, in order to extract deep fusion features, the two candidate features are... and and the output features of the previous layer cross-attention module The fusion is performed in the k-th feature extraction layer R of ResNet50. k Element-wise addition is performed in [x]. Obtain fusion features that include spatiotemporal correlations. in, This represents a 0 tensor, with K set to 4.

3. The method according to claim 2, characterized in that, In step 3, the output features of the last four feature extraction layers of the ResNet50 backbone network are first used as input, and they are represented from low to high as follows: Then, the high-level features are propagated down to the low-level features step by step, and gradually fused with the low-level features. The fusion process can be represented as: in, Conv represents the fusion feature output of the (i-1)th stage. 3×3 (·) represents a 3×3 convolution operation, and Up(·) represents an upsampling operation; finally, the fused feature outputs from the three stages are fused through a concatenation operation to obtain the final multi-scale fused feature output. Concat(·) represents a concatenation operation.

4. The method according to claim 3, characterized in that, In step 4, a 1×1 convolution operation is first used to reduce the dimensionality of each feature, transforming them into low-dimensional features. as well as Then these three features are respectively subjected to graph projection operation f proj (·) is transformed into the corresponding spatial graph node. Motion graph nodes and fusion graph nodes Right now: Through the above methods, spatial maps motion graph and fusion diagram Construction complete, including as well as These represent the set of nodes in the spatial graph, the set of nodes in the motion graph, and the set of nodes in the fused graph, respectively. as well as These represent the set of edges in the spatial graph, the set of edges in the motion graph, and the set of edges in the fused graph, respectively.

5. The method according to claim 4, characterized in that, In step 5, the spatial diagram and motion graph We mine potential semantic relationships and local / global graph structure information between paired graph nodes using multi-relation dynamic attention graph convolution operators in both static and motion feature spaces, and update the embedding information of graph nodes. Next, the output of the spatial graph node embedding information mining process and the output of the corresponding motion graph node embedding information mining process are simultaneously fed into the fusion graph node embedding information mining process enhanced by the graph interaction module. The spatiotemporal semantic features are fused and encoded by adaptively selecting spatial graph nodes and motion graph nodes.

6. The method according to claim 5, characterized in that, In step 6, the spatial graph nodes and motion graph nodes obtained in step 5 are processed by two layers of multi-relation dynamic attention graph convolution and fused by the corresponding map interaction module in the same way, and then graph reprojection is performed. Embedding the nodes of the fused graph into the fused graph output from the information mining process Mapping back to the original Euclidean space, i.e.: The above method achieves the mapping from non-Euclidean space graph structure data to Euclidean space pixel-level feature maps, thus preparing for the output of the final pixel-level saliency prediction map.

7. The method according to claim 6, characterized in that, In step 6, after processing by the fully connected layer, the final saliency prediction result is generated. Right now:

8. A video saliency object detection system based on bimodal cross-fusion and multi-graph interactive learning, characterized in that: The system includes: Cross-attention module: The RGB image of frame t and its corresponding optical flow image are taken as input, and spatial and motion features are initially extracted through two independent ResNet50 backbone networks. The cross-attention module is used to effectively fuse the spatial and motion features output from K feature extraction layers in the backbone network across spatiotemporal modalities. Multi-scale feature fusion module: A multi-scale feature fusion module is used to capture multi-scale feature cues from the spatial feature extraction branch, the temporal feature extraction branch, and the dual-modal cross-fusion branch, respectively, to improve the overall robustness of the method; The graph interaction module performs graph projection operations on multi-scale spatial features, multi-scale motion features, and multi-scale fusion features respectively, transforming them to non-Euclidean space to obtain corresponding map structure data, namely spatial graph, motion graph, and fusion graph. The spatial graph, motion graph, and fusion graph are simultaneously input into a multi-graph interactive learning network for processing. The multi-relation dynamic attention graph convolution operator captures richer spatiotemporal contextual structure information, while the graph interaction module maximizes the interaction of spatiotemporal information. Finally, the fusion graph, which is the final output of the node embedding information mining process in the fusion graph, undergoes graph reprojection to restore it to the original Euclidean space. After processing by a fully connected layer, the final saliency detection result is generated.

Images