A video salient object detection method and system based on spatio-temporal context scene relationship propagation

Abstract

The application provides a video salient object detection method based on spatio-temporal context scene relationship propagation, and relates to the technical field of video salient object detection. First, scene analysis is performed on each frame of video in a video frame sequence to obtain an instance-level object corresponding to each frame of video. Then, global instance-level features, local instance-level features and intra-frame low-level features of each frame of video and the corresponding instance-level object are extracted. Then, a matching frame of any frame of video is randomly sampled from the same video frame sequence, global instance-level features of the matching frame are extracted, the global instance-level features are integrated into global instance-level features of the matching frame, and time features between video frames are obtained. The dense spatial attention mechanism is used to integrate the local instance-level features into the global instance-level features to obtain spatial features. The time features and the spatial features are spliced to obtain spatio-temporal features. The spatio-temporal features and the global instance-level features are input into a convolutional neural network based on a gated recurrent unit for updating to obtain high-level spatio-temporal features. Finally, the intra-frame low-level features and the high-level spatio-temporal features are fused and decoded to generate a video salient object mask detection result. The application utilizes rich inter-frame and intra-frame scene relationship information in the video, and improves the accuracy of video salient object detection in complex scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video salient object detection, and more particularly to a video salient object detection method and system based on spatiotemporal context scene relationship propagation. BACKGROUND

[0002] Salient object detection is a key task in computer vision, aiming to simulate the attention mechanism of the human visual system to quickly locate and identify the most attention-grabbing objects in the scene, and perceive the entire scene based on visually salient objects and local regions. In practical applications, a salient object detector usually generates a saliency mask to facilitate the positioning of the most attention-grabbing regions and objects in the image. Therefore, it can provide certain preprocessing results for many object-oriented visual tasks. However, video salient object detection not only needs to handle a large amount of data, but also faces the problem of temporal consistency, because the location and shape of salient objects in different frames will change, and the focus of attention will also shift between different objects or regions. At the same time, dynamic lighting conditions and cluttered backgrounds make it more challenging to accurately detect salient objects.

[0003] To better cope with these challenges, existing video salient detection methods propose cross-attention mechanisms that can capture important spatiotemporal regions in videos and establish associations between objects in different frames. However, existing video salient detection methods ignore the analysis of complex scenes that are helpful for downstream tasks in real-world application scenarios. These complex scenes may contain multiple objects, complex backgrounds, and interactions between objects. At the same time, inter-frame and intra-frame scene relationships as strong saliency priors play a crucial role in video salient detection. A video is composed of a series of consecutive frames, and inter-frame relationships reflect the continuity and variability of objects over time, while intra-frame relationships embody the layout and interaction of objects in space. The insufficient analysis of complex scenes and the insufficient capture of inter-frame and intra-frame scene relationships in existing technologies lead to the inability to accurately distinguish salient objects from cluttered backgrounds, affecting the accuracy of video salient object detection. SUMMARY

[0004] To solve the problem of insufficient analysis of complex scenes and insufficient capture of inter-frame and intra-frame scene relationships in current video salient object detection methods, which leads to the inability to accurately distinguish salient objects from cluttered backgrounds, the present application provides a video salient object detection method based on spatiotemporal context scene relationship propagation, which improves the accuracy of video salient object detection.

[0005] To achieve the above technical effects, the technical solutions of the present application are as follows:

[0006] S1: scene parsing is performed on each frame of video in a video frame sequence to obtain an instance-level object corresponding to each frame of video;

[0007] S2: features of each frame of video and the instance-level object corresponding thereto are extracted, the features including global instance-level features, local instance-level features and intra-frame low-level features;

[0008] S3: a matching frame of any frame of video is randomly sampled from the same video frame sequence, global instance-level features of the matching frame are extracted, the global instance-level features are integrated into global instance-level features of the matching frame to obtain temporal features between video frames;

[0009] S4: the local instance-level features are integrated into the global instance-level features using a dense spatial attention mechanism to obtain spatial features;

[0010] S5: the temporal features described in step S3 and the spatial features described in step S4 are spliced to obtain spatio-temporal features;

[0011] S6: the spatio-temporal features and the global instance-level features are input into a convolutional neural network based on a gated recurrent unit for updating to obtain high-level spatio-temporal features;

[0012] S7: the intra-frame low-level features and the high-level spatio-temporal features are fused and decoded to generate a video saliency object mask detection result.

[0013] Further, the process of obtaining the instance-level object corresponding to each frame of video satisfies:

[0014] An instance-level scene parsing module is used to perform scene parsing on each frame of video, to identify and segment the instance-level object corresponding to each frame of video, and to generate a mask corresponding to each instance-level object, the expression of which is:

[0015] [S i,1 ,S i,2 ,···,S i,m ]=P SAMSeg (I i )

[0016] In the formula, P SAMSeg represents the instance-level scene parsing module, S i,m represents a mask of an mth instance-level object of an ith frame, and I i represents an ith frame in a video.

[0017] Further, the step S1 further includes:

[0018] A mask corresponding to each instance-level object is generated, a mask operation is performed on an original frame I i , a region of the instance is extracted, a mask region is retained, and a non-mask region is filled with 0, the expression of which is:

[0019] [I i,1 ,I i,2 ,···,I i,m ]=I i ·[S i,1 ,S i,2 ,···,S i,m ]

[0020] wherein I i,m denotes the m-th instance-level object of the i-th frame.

[0021] Further, the process of extracting the feature of each frame video and its corresponding instance-level object satisfies:

[0022] [x i ; z i,j ] = ε DeepLabv3 ([I i ; I i,j ])

[0023] [x l ] = ε DeepLabv3 ([I i ])

[0024] wherein j = 1, 2, ···, m denotes the index of the instance-level object extracted from the i-th frame, ε DeepLabv3 denotes the encoder of the pre-trained DeepLabv3 model, x i denotes the global instance-level feature, z i,j denotes the local instance-level feature, x l denotes the intra-frame low-level feature, I i denotes the i-th frame in the video.

[0025] According to the above technical means, the feature of each frame video and its corresponding instance-level object is extracted, the global instance-level feature extracted captures the global information of the entire instance-level object in the video frame, the local instance-level feature captures the detailed feature of the instance-level object, and the intra-frame low-level feature contains the basic visual feature in the video frame, which is used in the subsequent feature fusion to improve the accuracy of video salient object detection

[0026] Further, the step S3 comprises:

[0027] Firstly, the global instance-level feature of the current frame is calculated by linear projection transformation, and then the matrix multiplication operation is performed with the global instance-level feature of the matching frame to obtain the importance score describing the input features, and the process satisfies:

[0028]

[0029] where Linear denotes a linear projection transformation, denotes the transpose of x i , denotes global instance-level features of the matching frame, • denotes a matrix multiplication operation, and i denotes a description of the importance score between input features;

[0030] Next, importance filtering and optimization of the global instance-level features are performed to retain information of the salient part, resulting in attention-optimized features, which satisfy:

[0031] β i = x i · softmax(α i )

[0032] where softmax denotes an activation function, and i denotes attention-optimized features;

[0033] Finally, the attention-optimized features are extracted through a convolution operation to obtain temporal features, which satisfy:

[0034]

[0035] where σ denotes a Sigmoid activation function, denotes temporal features.

[0036] Further, the step S4 includes:

[0037] First, attention weight scores are calculated through a linear projection transformation, which satisfies:

[0038]

[0039] where α i←j denotes attention weight scores, and Linear denotes a linear projection transformation;

[0040] Next, the local instance-level features are weighted to highlight salient features, resulting in attention-optimized features, which satisfy:

[0041] β i←j = z i,j · softmax(α i←j )

[0042] where β i←j denotes weighted features;

[0043] Then, local features are extracted through a nonlinear transformation and a convolution operation, which satisfy:

[0044] z i←j= σ(CONV(β i←j )*β i←j

[0045] In the formula, z i←j represents the j-th instance feature is propagated to the local feature of the current i-th frame;

[0046] Finally, the plurality of local features are feature spliced and convolutional operation is performed to obtain the spatial feature, and the process satisfies:

[0047]

[0048] In the formula, represents the spatial feature, and CAT represents the feature splicing operation.

[0049] Further, the calculation expression of the spatio-temporal feature is:

[0050]

[0051] In the formula, ρ i represents the spatio-temporal feature, CAT represents the feature splicing operation, and CONV represents the convolutional operation, represents the spatial feature, represents the temporal feature.

[0052] According to the above technical means, the temporal feature can capture the time continuity between video frames and the change trend of the instance-level object, so as to more accurately capture the salient object of the video. The spatial feature can capture the complex semantic relationship between the instance-level objects in the video frame. The splicing of the spatio-temporal feature integrates the advantages of the temporal feature and the spatial feature, has more rich information, and can more accurately capture the salient object in the video in a complex environment.

[0053] Further, the calculation process of the high-level spatio-temporal feature is

[0054]

[0055] In the formula, represents the k-th updated high-level spatio-temporal feature, and ConvGRU represents the convolutional neural network based on the gating cycle unit.

[0056] According to the above technical means, by inputting the spatio-temporal feature and the global instance-level feature into the convolutional neural network based on the gating cycle unit for updating, the fusion of the spatio-temporal context information is enhanced and the rich spatial details are retained in the relationship propagation.

[0057] Further, the process of feature fusion of the intra-frame low-level feature and the high-level spatio-temporal feature to generate the saliency mask prediction result satisfies:

[0058]

[0059] In the formula, indicates a decoder, y i indicates a prediction result.

[0060] The application also provides a video salient object detection system based on spatio-temporal context scene relationship propagation, comprising:

[0061] An instance-level object acquisition module is configured to perform scene analysis on each frame of video in a video frame sequence to obtain an instance-level object corresponding to each frame of video.

[0062] A feature extraction module is configured to extract features of each frame of video and the instance-level object corresponding thereto, wherein the features include global instance-level features, local instance-level features and intra-frame low-level features.

[0063] A temporal feature extraction module is configured to randomly sample a matching frame of any frame of video from the same video frame sequence, extract global instance-level features of the matching frame, integrate the global instance-level features into the global instance-level features of the matching frame, and obtain temporal features between video frames.

[0064] A spatial feature extraction module is configured to integrate local instance-level features into global instance-level features using a dense spatial attention mechanism to obtain spatial features.

[0065] A feature splicing module is configured to splice the temporal features and the spatial features to obtain spatio-temporal features.

[0066] A high-level spatio-temporal feature generation module is configured to input the spatio-temporal features and the global instance-level features into a convolutional neural network based on a gated recurrent unit for updating to obtain high-level spatio-temporal features.

[0067] A detection result generation module is configured to decode the intra-frame low-level features and the high-level spatio-temporal features after feature fusion to generate a video salient object mask detection result.

[0068] Compared with the prior art, the method has the beneficial effects that:

[0069] The application provides a video salient object detection method and system based on spatio-temporal context scene relationship propagation. Firstly, instance-level objects corresponding to each frame of video are obtained. Features of each frame of video and the corresponding instance-level objects are extracted, which capture global information, detailed information and basic visual information of the embodiment objects in the video frame. Then, time features containing time continuity between video frames and change trend of instance-level objects and spatial features of complex semantic relationship between instance-level objects in the video frame are obtained, which are spliced into spatio-temporal features. Next, the spatio-temporal features and global instance-level features are input into a convolutional neural network of a gated recurrent unit for updating, enhancing the fusion of spatio-temporal context information and preserving rich spatial details in relationship propagation. Finally, after feature fusion of the intra-frame low-level features and high-level spatio-temporal features, a video salient object mask detection result is generated. The application utilizes rich spatio-temporal context information in the video, and improves the accuracy of video salient object detection in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 A flowchart of a video salient object detection method based on spatio-temporal context scene relationship propagation proposed in an embodiment of the application is shown.

[0071] Figure 2 A processing framework flowchart of a video salient object detection method based on spatio-temporal context scene relationship propagation proposed in an embodiment of the application is shown.

[0072] Figure 3 A structural diagram of a video salient object detection system based on spatio-temporal context scene relationship propagation proposed in an embodiment of the application is shown. DETAILED DESCRIPTION

[0073] The drawings are only used for illustrative description and cannot be understood as a limitation of the patent;

[0074] In order to better illustrate the embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0075] For those skilled in the art, it is understandable that some well-known content in the drawings may be omitted.

[0076] The technical solutions of the application will be further described below in combination with the drawings and embodiments.

[0077] The position relationship described in the drawings is only used for illustrative description and cannot be understood as a limitation of the patent;

[0078] Embodiment 1

[0079] The embodiment proposes a video salient object detection method based on spatio-temporal context scene relationship propagation, as shown in Figure 1As shown in the flowchart of the method, the method proposed in this embodiment includes the following steps in whole:

[0080] S1: scene parsing is performed on each frame of video in a video frame sequence to obtain an instance-level object corresponding to each frame of video;

[0081] S2: features of each frame of video and the corresponding instance-level object are extracted, and the features include global instance-level features, local instance-level features and intra-frame low-level features;

[0082] S3: a matching frame of any frame of video is randomly sampled from the same video frame sequence, the global instance-level features are integrated into the matching frame features to obtain temporal features between video frames;

[0083] S4: the local instance-level features are integrated into the global instance-level features using a dense spatial attention mechanism to obtain spatial features;

[0084] S5: the temporal features in step S3 and the spatial features in step S4 are spliced,

[0085] to obtain spatiotemporal features;

[0086] S6: the spatiotemporal features and the global instance-level features are input into a convolutional neural network based on a gated recurrent unit for updating to obtain high-level spatiotemporal features;

[0087] S7: the intra-frame low-level features and the high-level spatiotemporal features are fused and decoded to generate a video saliency object mask detection result.

[0088] In this embodiment, as shown in the processing framework flowchart of the video saliency object detection method. Figure 2 An instance-level scene parsing module is used to perform scene parsing on each frame of video. The instance-level scene parsing module is SAM (Segment Anything Model). SAM solves a series of new downstream segmentation problems on a series of new data distributions through prompting engineering. It includes three main parts: an image encoder, a prompt encoder and a mask decoder. The image encoder calculates image embedding, the prompt encoder calculates prompt embedding, and the mask decoder generates segmentation masks according to the embedding.

[0089] An instance-level scene parsing module is used to perform scene parsing on each frame of video. The instance-level scene parsing module is SAM (Segment Anything Model). SAM solves a series of new downstream segmentation problems on a series of new data distributions through prompting engineering. It includes three main parts: an image encoder, a prompt encoder and a mask decoder. The image encoder calculates image embedding, the prompt encoder calculates prompt embedding, and the mask decoder generates segmentation masks according to the embedding.

[0090] [S i,1 ,S i,2 ,···,S i,m ]=P SAMSeg (I i )

[0091] wherein P SAMSeg denotes an example-level scene parsing module, S i,m denotes a mask of the mth example-level object of the ith frame, I i denotes the ith frame in the video.

[0092] In addition, a corresponding mask is generated for each example-level object, and the mask is obtained by performing a mask operation on the original frame I i The mask operation extracts the region of the example, retains the mask region, and fills the non-mask region with 0, and the expression is as follows:

[0093] [I i,1 ,I i,2 ,···,I i,m ]=I i ·[S i,1 ,S i,2 ,···,S i,m ]

[0094] wherein I i,m denotes the mth example-level object of the ith frame.

[0095] The mask operation performs a bitwise multiplication on the original frame and the mask of the video, and the result only contains the content of the region corresponding to the mask, and the remaining part is filled with 0.

[0096] In this embodiment, in order to extract a fixed number of example objects, the confidence is calculated according to the intersection over union IoU, and the top five example objects with the highest confidence are selected as the example objects for feature extraction.

[0097] In this embodiment, the DeepLabv3 model is used to extract features from each frame and the corresponding example-level object, to obtain features with spatial information and high-level semantic information. At the same time, low-level semantic information is extracted for subsequent feature fusion. The DeepLabV3 model is a deep learning model for image semantic segmentation. The model captures multi-scale context information by using a multi-hole convolution and an ASPP module (Atrous Spatial Pyramid Pooling), thereby improving the accuracy of feature extraction.

[0098] The process of extracting features of each frame of video and the corresponding example-level object satisfies:

[0099] [x i ;z i,j ]=ε DeepLabv3 ([I i ;I i,j ])

[0100] [x l ]=ε DeepLabv3 ([Ii ])

[0101] where j = 1, 2, ···, m represents the index of the instance-level object extracted from the i-th frame, ε DeepLabv3 represents the encoder of the pre-trained DeepLabv3 model, x i represents the global instance-level feature, z i,j represents the local instance-level feature, x l represents the intra-frame low-level feature, I i represents the i-th frame in the video.

[0102] In this embodiment, the CAT operation is used for feature splicing, and then the CONV convolution operation is performed on the spliced features to further fuse the spliced features. The spliced spatio-temporal features fuse the advantages of time features and space features.

[0103] The calculation expression of the spatio-temporal feature is:

[0104]

[0105] where ρ i represents the spatio-temporal feature, CAT represents the feature splicing operation, CONV represents the convolution operation, represents the spatial feature, represents the temporal feature.

[0106] In this embodiment, the global instance-level feature and the spatio-temporal feature are input into the convolutional neural network based on the gating cycle unit for updating, enhancing the fusion of spatio-temporal context information and preserving rich spatial details in relationship propagation. Convolutional neural network (CNN) and gating cycle unit (GRU) are two kinds of deep learning models. Convolutional neural network CNN is a neural network model for processing image data, which extracts features in the image through convolution operation and pooling operation, and classifies or regresses prediction through multi-layer neural network. GRU is a gated recurrent neural network that can effectively process sequence data and has strong memory ability.

[0107] The calculation process of the high-level spatio-temporal feature is

[0108]

[0109] where, represents the high-level spatio-temporal feature after k updates, and ConvGRU represents the convolutional neural network based on the gating cycle unit. The first input of the convolutional neural network based on the gating cycle unit is h0=x i .

[0110] Finally, in order to maintain the simplicity and scalability of the present application, an encoder is used to output the final saliency mask prediction result, which combines a convolutional layer and a prediction head with a sigmoid activation function. The decoder of the output mask is composed of a convolutional layer CONV, an up-sampling layer Upsample and a sigmoid activation function σ. At the same time, in order to generate more accurate saliency masks, the present application extracts low-level features within the frame and fuses them with high-level spatio-temporal features, thereby recovering more details of the saliency mask prediction map.

[0111] The process of feature fusion of the intra-frame low-level features and the high-level spatio-temporal features to generate the saliency mask prediction result satisfies:

[0112]

[0113] In the formula, represents the decoder, y i represents the prediction result.

[0114] Embodiment 2

[0115] This embodiment is directed to the steps of obtaining the temporal features and spatial features in Embodiment 1.

[0116] In this embodiment, the matching frame of any frame video is randomly sampled from the same video frame sequence, the global instance-level features of the matching frame are extracted, the global instance-level features are integrated into the global instance-level features of the matching frame, and the steps of obtaining the temporal features between the video frames include:

[0117] First, the global instance-level features of the current frame are calculated by linear projection transformation, and then matrix multiplication operation is performed with the global instance-level features of the matching frame to obtain the importance score describing the input features. The process satisfies:

[0118]

[0119] In the formula, Linear represents linear projection transformation, represents the transpose of x i , represents the global instance-level features of the matching frame, and · represents matrix multiplication operation, α i represents the importance score describing the input features.

[0120] Next, the importance filtering and optimization of the global instance-level features are performed to retain the information of the significant part and obtain the attention-optimized features. The process satisfies:

[0121] β i = x i · softmax(α i )

[0122] where softmax denotes the activation function, β i Attention optimized features. The softmax activation function normalizes the output values to the range of 0 to 1. The softmax activation function assigns a high probability value to the part with a high importance score between features, and finally, the multiplication with the global instance-level feature highlights the features with high correlation and retains the information of the significant part.

[0123] Finally, the attention-optimized features are extracted through convolution operation to obtain the temporal features, which satisfy:

[0124]

[0125] where σ denotes the Sigmoid activation function, denotes the temporal feature.

[0126] In this embodiment, random sampling of any frame of the matching frame of the video can break the strong time dependence between two adjacent frames to enhance the diversity of the training sample pair. Due to the large training capacity and large computational complexity, only one matching frame is selected for inter-frame temporal feature coding, and the DeepLabv3 model is used to extract the global instance-level feature of the matching frame.

[0127] In this embodiment, the attention mechanism is used, and the current frame is used as Value (V) and Key (K), and the matching frame is used as Query (Q). This mechanism can capture the temporal correlation between frames and explicitly model the inter-frame interaction relationship, thereby improving the detection ability of the salient object in the dynamic scene. Query (Q) is a query vector that measures the relevance with information at all other positions. Key (K) is a key vector that represents the identification feature of each position. Value (V) is a value vector that represents the information to be weighted and summed according to the matching degree of Query and Key. The current frame as Value (V) and Key (K): provides the information Value that needs to be weighted, and the index (Key) used to calculate the attention weight. Such a design enables the model to simultaneously focus on the information of the current frame itself and its association with other frames. The matching frame as Query (Q) represents the query vector that seeks information related to the current frame.

[0128] In this embodiment, the dense spatial attention mechanism is used to integrate the local instance-level feature into the global instance-level feature, and the steps of obtaining the spatial feature include:

[0129] First, the attention weight score is calculated through linear projection transformation, which satisfies:

[0130]

[0131] where αi←j This represents the attention weight score, and Linear represents the linear projection transformation.

[0132] Next, the local instance-level features are weighted to highlight salient features, resulting in attention-optimized features. This process satisfies the following:

[0133] β i←j =z i,j ·softmax(α i←j )

[0134] In the formula, β i←j The first part represents the weighted features, and the second part represents the activation function that normalizes the output values ​​to the range of 0 to 1. The softmax activation function assigns high probability values ​​to the parts with high attention weight scores. Finally, it is multiplied with the local instance-level features, which highlights the features with high relevance and retains the information of the significant parts.

[0135] Then, local features are extracted through nonlinear transformations and convolution operations, a process that satisfies:

[0136] z i←j =σ(CONV(β) i←j ))*β i←j

[0137] In the formula, z i←j This indicates that the feature of the j-th instance is propagated to the local feature of the current i-th frame;

[0138] Finally, multiple local features are concatenated and convolved to obtain spatial features. This process satisfies the following:

[0139]

[0140] In the formula, CAT represents spatial features and signifies feature splicing operations.

[0141] Example 3

[0142] like Figure 3 The structural diagram shown in this embodiment also provides a video salient object detection system based on spatiotemporal context scene relationship propagation. This system includes:

[0143] The instance-level object acquisition module is used to perform scene parsing on each frame of the video frame sequence and obtain the instance-level object corresponding to each frame of the video.

[0144] The feature extraction module is used to extract features of each video frame and its corresponding instance-level object. The features include: global instance-level features, local instance-level features, and intra-frame low-level features.

[0145] a time feature extraction module, configured to sample a matching frame of any frame video from the same video frame sequence, extract a global instance level feature of the matching frame, integrate the global instance level feature into a global instance level feature of the matching frame, and obtain a time feature between video frames;

[0146] a space feature extraction module, configured to integrate the local instance level feature into the global instance level feature using a dense space attention mechanism, and obtain a space feature;

[0147] a feature splicing module, configured to splice the time feature and the space feature, and obtain a space-time feature;

[0148] a high-level space-time feature generation module, configured to input the space-time feature and the global instance level feature into a convolutional neural network based on a gated recurrent unit for updating, and obtain a high-level space-time feature;

[0149] a detection result generation module, configured to decode the feature fusion of the intra-frame low-level feature and the high-level space-time feature, and generate a video saliency object mask detection result.

[0150] The embodiments are merely examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. All the embodiments do not need to be exhausted. Any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A video salient object detection method based on spatio-temporal context scene relationship propagation, characterized in that, The method comprises the following steps: S1: scene parsing is performed on each frame of video in a video frame sequence to obtain an instance-level object corresponding to each frame of video; S2: features of each frame of video and the corresponding instance-level object are extracted, and the features comprise global instance-level features, local instance-level features and intra-frame low-level features; S3: a matching frame of any frame of video is randomly sampled from the same video frame sequence, global instance-level features of the matching frame are extracted, the global instance-level features are integrated into the global instance-level features of the matching frame, and time features between video frames are obtained; The step S3 comprises: The global instance-level features of the current frame are subjected to linear projection transformation calculation, and then matrix multiplication operation is performed on the global instance-level features of the matching frame to obtain an importance score describing the input features: where Linear denotes a linear projection transformation, denotes the transpose of x i denotes the transpose of x i denotes global instance-level features, denotes global instance-level features of the matching frame, denotes a matrix multiplication operation, and i denotes a significance score describing the importance between input features; The importance of the global instance-level features is filtered and optimized, and significant information is retained to obtain features after attention optimization: β i = x i softmax(a i ) where softmax denotes an activation function, and β i attention-optimized features; The features after attention optimization are subjected to convolution operation to extract time features: In the formula, σ represents a Sigmoid activation function, denotes a time characteristic; S4: a dense spatial attention mechanism is used to integrate the local instance-level features into the global instance-level features to obtain spatial features; The step S4 comprises: Attention weight scores are calculated by performing linear projection transformation calculation on the local instance-level features: wherein α i←j denotes the attention weight score, Linear denotes a linear projection transformation, z i,j denotes the local instance-level feature, j = 1, 2, ···, m denotes the index of the instance-level object extracted from the i-th frame; The local instance-level features are weighted to highlight significant features to obtain features after attention optimization: β i←j = z i,j softmax(a i←j ) In the formula, β i←j denotes the weighted feature; Local features are extracted through nonlinear transformation and convolution operation: z i←j = σ(CONV(β i←j ))*β i←j wherein z i←j denotes the jth instance feature is propagated to the local feature of the current ith frame; The multiple local features are subjected to feature splicing and convolution operation to obtain spatial features: In the formula, represents a spatial feature, and CAT represents a feature concatenation operation. S5: the time features described in the step S3 and the spatial features described in the step S4 are subjected to feature splicing to obtain spatio-temporal features; S6: the spatio-temporal features and the global instance-level features are input into a convolutional neural network based on a gated recurrent unit for updating to obtain high-level spatio-temporal features; S7: the intra-frame low-level features and the high-level spatio-temporal features are subjected to feature fusion and then decoding to generate a video saliency object mask detection result.

2. The method of claim 1, wherein, The process of obtaining the instance-level object corresponding to each frame of video satisfies: An instance-level scene parsing module is used to perform scene parsing on each frame of video, to identify and segment the instance-level object corresponding to each frame of video, and to generate a mask corresponding to each instance-level object, and the expression is: [S i,1 ,S i,2 ,···,S i,m ]=P SAMSeg (I i ) where P SAMSeg represents an instance-level scene parsing module, S i,m represents a mask of the mth instance-level object of the ith frame, I i represents the ith frame in a video.

3. The method of claim 2, wherein, The step S1 further comprises: A corresponding mask is generated for each instance-level object, through the original frame I i A mask operation is performed to extract the region of the instance, the mask region is retained, and the non-mask region is filled with 0, and the expression is as follows: [I i,1 ,I i,2 ,···,I i,m ] = I i · [S i,1 , S i,2 , ···, S i,m ] where I i,m denotes the mth instance-level object of the ith frame.

4. The method of claim 1, wherein, The process of extracting the features of each frame of video and the corresponding instance-level object satisfies: [x i ; z i,j ] = ε DeepLabv3 ([I i ; I i,j ]), [x l ] = ε DeepLabv3 ([I i ]). where j = 1, 2, · · ·, m denotes the index of the instance-level object extracted from the i-th frame, ε DeepLabv3 denotes the encoder of the pre-trained DeepLabv3 model, x i denotes the global instance-level feature, z i,j denotes the local instance-level feature, x l denotes the intra-frame low-level feature, I i denotes the i-th frame in the video.

5. The method of claim 1, wherein, The calculation expression of the spatio-temporal features is: In the formula, p i represents the spatio-temporal feature, CAT represents the feature concatenation operation, CONV represents the convolution operation, represents the spatial feature, represents the temporal feature.

6. The method of claim 5, wherein, The calculation process of the high-level spatio-temporal features is In the formula, denotes the high-level spatio-temporal feature after k updates, and ConvGRU denotes a convolutional neural network based on a gated recurrent unit.

7. The method of claim 6, wherein, The process of generating a saliency mask prediction result by fusing the intra-frame low-level features and the high-level spatio-temporal features satisfies: In the formula, represents a decoder, y i represents a prediction result.

8. A system for video salient object detection based on spatio-temporal context scene relationship propagation, for implementing the method for video salient object detection based on spatio-temporal context scene relationship propagation according to any one of claims 1-7, characterized in that, It comprises: An instance-level object acquisition module is configured to perform scene parsing on each frame of video in a video frame sequence to obtain an instance-level object corresponding to each frame of video; A feature extraction module is configured to extract features of each frame of video and the corresponding instance-level object, and the features comprise global instance-level features, local instance-level features and intra-frame low-level features; A time feature extraction module is configured to randomly sample a matching frame of any frame of video from the same video frame sequence, extract global instance-level features of the matching frame, integrate the global instance-level features into the global instance-level features of the matching frame, and obtain time features between video frames. The spatial feature extraction module is configured to integrate local instance-level features into global instance-level features using a dense spatial attention mechanism to obtain spatial features. The feature splicing module is configured to splice the temporal features and the spatial features to obtain spatio-temporal features. The advanced spatio-temporal feature generation module is configured to input the spatio-temporal features and the global instance-level features into a convolutional neural network based on a gated recurrent unit for updating to obtain advanced spatio-temporal features. The detection result generation module is configured to decode the feature fusion of the intra-frame low-level features and the advanced spatio-temporal features to generate a video saliency object mask detection result.

Images