Video saliency object detection method and system based on spatio-temporal context scene relation propagation

Abstract

The invention provides a video saliency object detection method based on spatio-temporal context scene relation propagation, and relates to the technical field of video saliency object detection. The method comprises the following steps: firstly, performing scene analysis on each frame of video in a video frame sequence to obtain an instance-level object corresponding to each frame of video; then, extracting global instance-level features, local instance-level features and intra-frame low-level features of each frame of video and the corresponding instance-level object; then, randomly sampling a matching frame of any frame of video from the same video frame sequence, extracting global instance-level features of the matching frame, and integrating the global instance-level features into the global instance-level features of the matching frame to obtain time features between the video frames; and integrating the local instance-level features into the global instance-level features by using a dense spatial attention mechanism to obtain spatial features. And carrying out feature splicing on the time features and the space features to obtain space-time features. And inputting the spatial-temporal features and the global instance-level features into a convolutional neural network based on a gated cycle unit for updating to obtain advanced spatial-temporal features. And finally, performing feature fusion on the intra-frame low-level features and the high-level spatial-temporal features, and then decoding to generate a video saliency object mask detection result. According to the method, rich inter-frame and intra-frame scene relation information in the video is utilized, and the accuracy of video saliency object detection in a complex scene is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video salient object detection, and more specifically relates to a video salient object detection method and system based on spatiotemporal context scene relationship propagation. Background Art

[0002] Salient object detection is a key task in computer vision, which aims to simulate the attention mechanism of the human visual system to quickly locate and identify the most attention-grabbing objects in a scene, and perceive the entire scene based on visually salient objects and local regions. In practical applications, salient object detectors usually generate a saliency mask to facilitate locating the most attention-grabbing regions and objects in an image. Therefore, it can provide certain preprocessing results for many object-oriented vision tasks. However, video salient object detection not only requires processing a large amount of data, but also faces the problem of temporal consistency, because the position and shape of salient objects in different frames will change, and the focus of attention will also shift between different objects or regions. At the same time, dynamic lighting conditions and cluttered backgrounds make it more challenging to accurately detect salient objects.

[0003] In order to better cope with these challenges, existing video saliency detection methods have proposed a cross-attention mechanism, which can capture important spatiotemporal regions in the video and establish the association between objects in different frames. However, existing video saliency detection methods ignore the analysis of complex scenes that are helpful for downstream tasks in practical application scenarios. These complex scenes may contain multiple objects, complex backgrounds, and interactions between objects. At the same time, the scene relationships between frames and within frames play a vital role in video saliency detection as strong saliency priors. Videos are composed of a series of continuous frames. The relationship between frames reflects the continuity and variability of objects in time, while the relationship within frames reflects the layout and interaction of objects in space. The insufficient analysis of complex scenes and the insufficient capture of the scene relationships between frames and within frames in the existing technology make it impossible to accurately distinguish salient objects from cluttered backgrounds, affecting the accuracy of video salient object detection. Summary of the invention

[0004] In order to solve the problem that current video salient object detection methods are insufficient in analyzing complex scenes and insufficient in capturing inter-frame and intra-frame scene relationships, resulting in the inability to accurately distinguish salient objects from cluttered backgrounds, the present invention provides a video salient object detection method based on spatiotemporal context scene relationship propagation to improve the accuracy of video salient object detection.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] S1: Perform scene analysis on each frame in the video frame sequence to obtain the instance-level object corresponding to each frame;

[0007] S2: extracting features of each frame of video and its corresponding instance-level object, wherein the features include: global instance-level features, local instance-level features, and intra-frame low-level features;

[0008] S3: Randomly sample the matching frame of any frame of the video from the same video frame sequence, extract the global instance-level features of the matching frame, integrate the global instance-level features into the global instance-level features of the matching frame, and obtain the temporal features between video frames;

[0009] S4: Use dense spatial attention mechanism to integrate local instance-level features into global instance-level features to obtain spatial features;

[0010] S5: performing feature splicing on the time feature described in step S3 and the space feature described in step S4 to obtain a spatiotemporal feature;

[0011] S6: Input the spatiotemporal features and global instance-level features into the convolutional neural network based on the gated recurrent unit for updating to obtain high-level spatiotemporal features;

[0012] S7: After feature fusion, low-level features and high-level spatiotemporal features within the frame are decoded to generate video salient object mask detection results.

[0013] Furthermore, the process of obtaining the instance-level object corresponding to each frame of video satisfies:

[0014] The instance-level scene parsing module is used to perform scene parsing on each frame of the video, identify and segment the instance-level objects corresponding to each frame of the video, and generate a mask corresponding to each instance-level object. The expression is:

[0015] [S i,1 ,S i,2 ,···,S i,m ]=P SAMSeg (I i )

[0016] Where P SAMSeg represents the instance-level scene parsing module, S i,m represents the mask of the mth instance-level object in the i-th frame, I i Represents the i-th frame in the video.

[0017] Furthermore, the step S1 further includes:

[0018] Generate a corresponding mask for each instance-level object, through the original frame I i Perform mask operation, extract the instance area, retain the mask area, and fill the non-mask area with 0. The expression is:

[0019] [I i,1 ,I i,2 ,···,I i,m ]=I i ·[S i,1 ,S i,2 ,···,S i,m ]

[0020] Among them, I i,m Represents the mth instance-level object in the i-th frame.

[0021] Furthermore, the process of extracting features of each frame of video and its corresponding instance-level object satisfies:

[0022] [x i ;z i,j ]=ε DeepLabv3 ([I i ;I i,j ])

[0023] [x l ]=ε DeepLabv3 ([I i ])

[0024] where j = 1, 2, ···, m represents the index of the instance-level object extracted from the i-th frame, ε DeepLabv3 represents the encoder of the pre-trained DeepLabv3 model, x i represents the global instance-level feature, z i,j represents the local instance-level feature, x l Represents the low-level features within the frame, I i Represents the i-th frame in the video.

[0025] According to the above technical means, the features of each frame of video and its corresponding instance-level object are extracted. The extracted global instance-level features capture the global information of the entire instance-level object in the video frame, and the local instance-level features capture the detailed features of the instance-level object. The low-level features extracted in the frame include the basic visual features in the video frame, which are used in the subsequent feature fusion to improve the accuracy of video salient object detection.

[0026] Furthermore, the step S3 comprises:

[0027] First, the global instance-level features of the current frame are linearly projected and transformed, and then matrix multiplied with the global instance-level features of the matching frame to obtain the importance score describing the input features. This process satisfies:

[0028]

[0029] In the formula, Linear represents linear projection transformation, Represents x i The transpose of represents the global instance-level features of the matching frame, represents the matrix multiplication operation, and α i Represents the importance score between the input features;

[0030] Next, the importance of the global instance-level features is filtered and optimized, and the significant part of the information is retained to obtain the attention-optimized features. This process satisfies:

[0031] β i =x i ·softmax(α i )

[0032] In the formula, softmax represents the activation function, β i Features after attention optimization;

[0033] Finally, the attention-optimized features are used to extract temporal features through convolution operations. This process satisfies:

[0034]

[0035] In the formula, σ represents the Sigmoid activation function, Represents time characteristics.

[0036] Furthermore, the step S4 comprises:

[0037] First, the attention weight score is calculated by linear projection transformation, which satisfies:

[0038]

[0039] In the formula, α i←j Represents the attention weight score, and Linear represents the linear projection transformation;

[0040] Next, the local instance-level features are weighted to highlight the significant features and obtain the attention-optimized features. This process satisfies:

[0041] β i←j =z i,j ·softmax(α i←j )

[0042] In the formula, β i←j represents the weighted features;

[0043] Then, local features are extracted through nonlinear transformation and convolution operations. This process satisfies:

[0044] z i←j=σ(CONV(β i←j ))*β i←j

[0045] In the formula, z i←j Indicates that the j-th instance feature is propagated to the local feature of the current i-th frame;

[0046] Finally, multiple local features are concatenated and convolved to obtain spatial features. This process satisfies:

[0047]

[0048] In the formula, represents spatial features, and CAT represents feature concatenation operations.

[0049] Furthermore, the calculation expression of the spatiotemporal feature is:

[0050]

[0051] In the formula, ρ i represents spatiotemporal features, CAT represents feature concatenation operations, and CONV represents convolution operations. Represents spatial features, Represents time characteristics.

[0052] Based on the above technical means, temporal features can capture the temporal coherence between video frames and the changing trend of instance-level objects, thereby more accurately capturing the salient objects in the video. Spatial features can capture the complex semantic relationship between instance-level objects in video frames. The splicing of temporal and spatial features combines the advantages of temporal and spatial features, has richer information, and can more accurately capture salient objects in videos in complex environments.

[0053] Furthermore, the calculation process of the advanced spatiotemporal features is:

[0054]

[0055] In the formula, represents the high-level spatiotemporal features after k updates, and ConvGRU represents a convolutional neural network based on a gated recurrent unit.

[0056] According to the above technical means, by inputting spatiotemporal features and global instance-level features into the convolutional neural network of the gated recurrent unit for updating, the fusion of spatiotemporal contextual information is enhanced and rich spatial details are retained in relation propagation.

[0057] Furthermore, the process of fusing the low-level features and high-level spatiotemporal features within the frame to generate a saliency mask prediction result satisfies:

[0058]

[0059] In the formula, represents the decoder, y i Represents the prediction result.

[0060] The present invention also provides a video salient object detection system based on spatiotemporal context scene relationship propagation, comprising:

[0061] The instance-level object acquisition module is used to perform scene analysis on each frame of video in the video frame sequence to obtain the instance-level object corresponding to each frame of video;

[0062] A feature extraction module is used to extract features of each frame of video and its corresponding instance-level object, wherein the features include: global instance-level features, local instance-level features, and intra-frame low-level features;

[0063] The temporal feature extraction module is used to randomly sample the matching frame of any frame of video from the same video frame sequence, extract the global instance-level features of the matching frame, integrate the global instance-level features into the global instance-level features of the matching frame, and obtain the temporal features between video frames;

[0064] A spatial feature extraction module is used to integrate local instance-level features into global instance-level features using a dense spatial attention mechanism to obtain spatial features;

[0065] A feature splicing module, used for splicing the time feature and the space feature to obtain a spatiotemporal feature;

[0066] A high-level spatiotemporal feature generation module is used to input the spatiotemporal features and global instance-level features into a convolutional neural network based on a gated recurrent unit for updating to obtain high-level spatiotemporal features;

[0067] The detection result generation module is used to fuse the low-level features and high-level spatiotemporal features within the frame and decode them to generate the mask detection results of the video salient objects.

[0068] Compared with the prior art, the beneficial effects of this method are:

[0069] The present invention provides a method and system for detecting video salient objects based on spatiotemporal context scene relation propagation. First, the instance-level object corresponding to each frame of video is obtained. The features of each frame of video and its corresponding instance-level object are extracted, and these features capture the global information, detail information and basic visual information of the embodiment object in the video frame. Then, the temporal features including the temporal coherence between video frames and the change trend of the instance-level object and the spatial features of the complex semantic relationship between the instance-level objects in the video frame are obtained, and they are spliced ​​into spatiotemporal features. Then, the spatiotemporal features and the global instance-level features are input into the convolutional neural network of the gated recurrent unit for updating, so as to enhance the fusion of spatiotemporal context information and retain rich spatial details in relation propagation. Finally, the low-level features and high-level spatiotemporal features in the frame are decoded after feature fusion to generate a video salient object mask detection result. The present invention utilizes the rich spatiotemporal context information in the video to improve the accuracy of video salient object detection in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flowchart showing a video salient object detection method based on spatiotemporal contextual scene relationship propagation proposed in an embodiment of the present invention;

[0071] Figure 2 A flowchart showing a processing framework of a video salient object detection method based on spatiotemporal context scene relationship propagation proposed in an embodiment of the present invention;

[0072] Figure 3 A structural diagram showing a video salient object detection system based on spatiotemporal contextual scene relationship propagation proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0074] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0075] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0076] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0077] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent;

[0078] Example 1

[0079] This embodiment proposes a video salient object detection method based on spatiotemporal context scene relationship propagation, such as Figure 1The method proposed in this embodiment generally includes the following steps:

[0080] S1: Perform scene analysis on each frame in the video frame sequence to obtain the instance-level object corresponding to each frame;

[0081] S2: extracting features of each frame of video and its corresponding instance-level object, wherein the features include: global instance-level features, local instance-level features, and intra-frame low-level features;

[0082] S3: Randomly sample the matching frame of any frame from the same video frame sequence, integrate the global instance-level features into the matching frame features, and obtain the temporal features between video frames;

[0083] S4: Use dense spatial attention mechanism to integrate local instance-level features into global instance-level features to obtain spatial features;

[0084] S5: performing feature splicing on the time feature described in step S3 and the space feature described in step S4,

[0085] Get the spatiotemporal characteristics;

[0086] S6: Input the spatiotemporal features and global instance-level features into the convolutional neural network based on the gated recurrent unit for updating to obtain high-level spatiotemporal features;

[0087] S7: After feature fusion, low-level features and high-level spatiotemporal features within the frame are decoded to generate video salient object mask detection results.

[0088] In this embodiment, if Figure 2 The processing framework flow chart of the video salient object detection method is shown in Figure 1. The instance-level scene parsing module is used to perform scene parsing on each frame of the video. The instance-level scene parsing module is SAM (SegmentAnything Model), which solves a series of downstream segmentation problems on new data distributions through prompt engineering. It consists of three main parts: image encoder, prompt encoder, and mask decoder. The image encoder calculates image embeddings, the prompt encoder calculates prompt embeddings, and the mask decoder generates segmentation masks based on these embeddings.

[0089] The instance-level scene parsing module is used to perform scene parsing on each frame of the video, identify and segment the instance-level objects corresponding to each frame of the video, and generate a mask corresponding to each instance-level object. The expression is:

[0090] [S i,1 ,S i,2 ,···,S i,m ]=P SAMSeg (I i )

[0091] Where P SAMSeg represents the instance-level scene parsing module, S i,m represents the mask of the mth instance-level object in the i-th frame, I i Represents the i-th frame in the video.

[0092] In addition, a corresponding mask is generated for each instance-level object, through the original frame I i Perform mask operation, extract the instance area, retain the mask area, and fill the non-mask area with 0. The expression is:

[0093] [I i,1 ,I i,2 ,···,I i,m ]=I i ·[S i,1 ,S i,2 ,···,S i,m ]

[0094] Among them, I i,m Represents the mth instance-level object in the i-th frame.

[0095] The mask operation performs bitwise multiplication of the original frame of the video and the mask. The result only contains the content of the area corresponding to the mask, and the rest is filled with 0.

[0096] In this embodiment, in order to extract a fixed number of instance objects, the confidence is calculated according to the intersection over union (IoU), and the first five instance objects with the highest confidence are selected as instance objects for feature extraction.

[0097] In this embodiment, the DeepLabv3 model is used to extract features from each frame and the corresponding instance-level objects to obtain features with spatial information and high-level semantic information. At the same time, low-level semantic information is extracted for subsequent feature fusion. The DeepLabV3 model is a deep learning model for image semantic segmentation. The model captures multi-scale contextual information and improves the accuracy of feature extraction by adopting atrous convolution and ASPP modules (Atrous Spatial Pyramid Pooling).

[0098] The process of extracting the features of each frame of video and its corresponding instance-level object satisfies:

[0099] [x i ;z i,j ]=ε DeepLabv3 ([I i ;I i,j ])

[0100] [x l ]=ε DeepLabv3 ([Ii ])

[0101] where j = 1, 2, ···, m represents the index of the instance-level object extracted from the i-th frame, ε DeepLabv3 represents the encoder of the pre-trained DeepLabv3 model, x i represents the global instance-level feature, z i,j represents the local instance-level feature, x l Represents the low-level features within the frame, I i Represents the i-th frame in the video.

[0102] In this embodiment, the CAT operation is used to perform feature concatenation, and then the CONV convolution operation is performed on the concatenated features to further fuse the concatenated features. The concatenated spatiotemporal features combine the advantages of temporal features and spatial features.

[0103] The calculation expression of spatiotemporal characteristics is:

[0104]

[0105] In the formula, ρ i represents spatiotemporal features, CAT represents feature concatenation operations, and CONV represents convolution operations. Represents spatial features, Represents time characteristics.

[0106] In this embodiment, the global instance-level features and spatiotemporal features are input into a convolutional neural network based on a gated recurrent unit for updating, enhancing the fusion of spatiotemporal contextual information and retaining rich spatial details in relational propagation. Convolutional neural network (CNN) and gated recurrent unit (GRU) are two deep learning models. Convolutional neural network CNN is a neural network model for processing image data. It extracts features in images through convolution operations and pooling operations, and performs classification or regression prediction through a multi-layer neural network. GRU is a gated recurrent neural network that can effectively process sequence data and has a strong memory capacity.

[0107] The calculation process of high-level spatiotemporal features is

[0108]

[0109] In the formula, represents the high-level spatiotemporal features after k updates, and ConvGRU represents the convolutional neural network based on the gated recurrent unit. h0 = x is first input into the convolutional neural network based on the gated recurrent unit. i .

[0110] Finally, in order to maintain the simplicity and scalability of the present invention, an encoder is used to output the final saliency mask prediction result in combination with a convolutional layer and a prediction head of a Sigmoid activation function. The decoder of the output mask consists of a convolutional layer CONV, an upsampling layer Upsample and a Sigmoid activation function σ. At the same time, in order to generate a more accurate saliency mask, the present invention extracts low-level features within the frame and fuses them with high-level spatiotemporal features, thereby restoring more details of the saliency mask prediction map.

[0111] The process of fusing low-level features within a frame with high-level spatiotemporal features to generate saliency mask prediction results satisfies:

[0112]

[0113] In the formula, represents the decoder, y i Indicates the prediction result.

[0114] Example 2

[0115] This embodiment specifically describes the steps of obtaining the time features and the space features in Embodiment 1.

[0116] In this embodiment, the steps of randomly sampling a matching frame of any frame of video from the same video frame sequence, extracting the matching frame global instance-level features, integrating the global instance-level features into the matching frame global instance-level features, and obtaining the temporal features between video frames include:

[0117] First, the global instance-level features of the current frame are linearly projected and transformed, and then matrix multiplied with the global instance-level features of the matching frame to obtain the importance score describing the input features. This process satisfies:

[0118]

[0119] In the formula, Linear represents linear projection transformation, Represents x i The transpose of represents the global instance-level features of the matching frame, represents the matrix multiplication operation, and α i Represents the importance score between the input features;

[0120] Next, the importance of the global instance-level features is filtered and optimized, and the significant part of the information is retained to obtain the attention-optimized features. This process satisfies:

[0121] β i =x i ·softmax(α i )

[0122] In the formula, softmax represents the activation function, β i Features after attention optimization. The softmax activation function normalizes the output value to the range of 0 to 1. The softmax activation function assigns high probability values ​​to the parts with high importance scores between features, and finally multiplies them with the global instance-level features, which will highlight the features with higher correlation and retain the significant part of the information.

[0123] Finally, the attention-optimized features are used to extract temporal features through convolution operations. This process satisfies:

[0124]

[0125] In the formula, σ represents the Sigmoid activation function, Represents time characteristics.

[0126] In this embodiment, randomly sampling the matching frames of any frame of video can break the limitation of strong temporal dependency between two adjacent frames to enhance the diversity of training sample pairs. Due to the large training capacity and high computational complexity, only one matching frame is selected for inter-frame temporal feature encoding, and the DeepLabv3 model is used to extract the global instance-level features of the matching frame.

[0127] In this embodiment, an attention mechanism is adopted, and the current frame is used as Value (V) and Key (K), while the matching frame is used as Query (Q). This mechanism can capture the temporal correlation between frames and explicitly model the interaction relationship between frames, thereby improving the detection ability of salient objects in dynamic scenes. Query (Q) is a query vector, which measures the correlation with information at all other positions. Key (K) is a key vector, which represents the identification feature of each position. Value (V) is a value vector, which represents the information to be weighted and summed according to the degree of match between Query and Key. The current frame is used as Value (V) and Key (K): it provides the information Value that needs to be weighted, and the index (Key) used to calculate the attention weight. This design enables the model to pay attention to the information of the current frame itself and its association with other frames at the same time. The matching frame is used as Query (Q), which represents the query vector and searches for information related to the current frame.

[0128] In this embodiment, the steps of integrating local instance-level features into global instance-level features using a dense spatial attention mechanism to obtain spatial features include:

[0129] First, the attention weight score is calculated by linear projection transformation, which satisfies:

[0130]

[0131] In the formula, αi←j Represents the attention weight score, and Linear represents the linear projection transformation;

[0132] Next, the local instance-level features are weighted to highlight the significant features and obtain the attention-optimized features. This process satisfies:

[0133] β i←j =z i,j ·softmax(α i←j )

[0134] In the formula, β i←j represents the weighted features, and softmax represents the activation function, which normalizes the output value to the range of 0 to 1. The softmax activation function assigns a high probability value to the part with a high attention weight score, and finally multiplies it with the local instance-level features, which will highlight the features with higher correlation and retain the significant part of the information.

[0135] Then, local features are extracted through nonlinear transformation and convolution operations. This process satisfies:

[0136] z i←j =σ(CONV(β i←j ))*β i←j

[0137] In the formula, z i←j Indicates that the j-th instance feature is propagated to the local feature of the current i-th frame;

[0138] Finally, multiple local features are concatenated and convolved to obtain spatial features. This process satisfies:

[0139]

[0140] In the formula, represents spatial features, and CAT represents feature concatenation operations.

[0141] Example 3

[0142] like Figure 3 As shown in the structural diagram, in this embodiment, a video salient object detection system based on spatiotemporal context scene relationship propagation is also provided, and the system includes:

[0143] The instance-level object acquisition module is used to perform scene analysis on each frame of video in the video frame sequence to obtain the instance-level object corresponding to each frame of video;

[0144] A feature extraction module is used to extract features of each frame of video and its corresponding instance-level object, wherein the features include: global instance-level features, local instance-level features, and intra-frame low-level features;

[0145] The temporal feature extraction module is used to randomly sample the matching frame of any frame of video from the same video frame sequence, extract the global instance-level features of the matching frame, integrate the global instance-level features into the global instance-level features of the matching frame, and obtain the temporal features between video frames;

[0146] A spatial feature extraction module is used to integrate local instance-level features into global instance-level features using a dense spatial attention mechanism to obtain spatial features;

[0147] A feature splicing module, used for splicing the time feature and the space feature to obtain a spatiotemporal feature;

[0148] A high-level spatiotemporal feature generation module is used to input the spatiotemporal features and global instance-level features into a convolutional neural network based on a gated recurrent unit for updating to obtain high-level spatiotemporal features;

[0149] The detection result generation module is used to fuse the low-level features and high-level spatiotemporal features within the frame and decode them to generate the mask detection results of the video salient objects.

[0150] The embodiments are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A video salient object detection method based on spatiotemporal contextual scene relationship propagation, characterized in that: The following steps are involved: S1: Perform scene analysis on each frame in the video frame sequence to obtain the instance-level object corresponding to each frame; S2: extracting features of each frame of video and its corresponding instance-level object, wherein the features include: global instance-level features, local instance-level features, and intra-frame low-level features; S3: Randomly sample the matching frame of any frame of the video from the same video frame sequence, extract the global instance-level features of the matching frame, integrate the global instance-level features into the global instance-level features of the matching frame, and obtain the temporal features between video frames; S4: Use dense spatial attention mechanism to integrate local instance-level features into global instance-level features to obtain spatial features; S5: performing feature splicing on the time feature described in step S3 and the space feature described in step S4 to obtain a spatiotemporal feature; S6: Input the spatiotemporal features and global instance-level features into the convolutional neural network based on the gated recurrent unit for updating to obtain high-level spatiotemporal features; S7: After feature fusion, low-level features and high-level spatiotemporal features within the frame are decoded to generate video salient object mask detection results.

2. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 1, characterized in that: The process of obtaining the instance-level object corresponding to each frame of video satisfies: The instance-level scene parsing module is used to perform scene parsing on each frame of the video, identify and segment the instance-level objects corresponding to each frame of the video, and generate a mask corresponding to each instance-level object. The expression is: [S i,1 ,S i,2 ,···,S i,m ]=P SAMSeg (I i ) Where P SAMSeg represents the instance-level scene parsing module, S i,m represents the mask of the mth instance-level object in the i-th frame, I i Represents the i-th frame in the video.

3. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 2 is characterized in that: The step S1 further comprises: Generate a corresponding mask for each instance-level object, through the original frame I i Perform mask operation, extract the instance area, retain the mask area, and fill the non-mask area with 0. The expression is: [I i,1 ,I i,2 ,···,I i,m ]=I i ·[S i,1 ,S i,2 ,···,S i,m ] Among them, I i,m Represents the mth instance-level object in the i-th frame.

4. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 1, characterized in that: The process of extracting the features of each frame of video and its corresponding instance-level object satisfies: [x i ;z i,j ]=ε DeepLabv3 ([I i ;I i,j ]), [x] l ]=e DeepLabv3 ([I i ]). where j = 1, 2, ···, m represents the index of the instance-level object extracted from the i-th frame, ε DeepLabv3 represents the encoder of the pre-trained DeepLabv3 model, x i represents the global instance-level feature, z i,j represents the local instance-level feature, x l Represents the low-level features within the frame, I i Represents the i-th frame in the video.

5. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 4, characterized in that: The step S3 comprises: First, the global instance-level features of the current frame are linearly projected and transformed, and then matrix multiplied with the global instance-level features of the matching frame to obtain the importance score describing the input features. This process satisfies: In the formula, Linear represents linear projection transformation, Represents x i The transpose of represents the global instance-level features of the matching frame, represents the matrix multiplication operation, and α i Represents the importance score between the input features; Next, the importance of the global instance-level features is filtered and optimized, and the significant part of the information is retained to obtain the attention-optimized features. This process satisfies: b i =x i ·softmax(a i ) In the formula, softmax represents the activation function, β i Features after attention optimization; Finally, the attention-optimized features are used to extract temporal features through convolution operations. This process satisfies: In the formula, σ represents the Sigmoid activation function, Represents time characteristics.

6. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 4, characterized in that: The step S4 comprises: First, the attention weight score is calculated by linear projection transformation, which satisfies: In the formula, α i←j Represents the attention weight score, and Linear represents the linear projection transformation; Next, the local instance-level features are weighted to highlight the significant features and obtain the attention-optimized features. This process satisfies: β i←j =z i,j ·softmax(α i←j ) In the formula, β i←j represents the weighted features; Then, local features are extracted through nonlinear transformation and convolution operations. This process satisfies: z i←j =σ(CONV(β i←j ))*b i←j In the formula, z i←j Indicates that the j-th instance feature is propagated to the local feature of the current i-th frame; Finally, multiple local features are concatenated and convolved to obtain spatial features. This process satisfies: In the formula, represents spatial features, and CAT represents feature concatenation operations.

7. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 1, characterized in that: The calculation expression of the spatiotemporal feature is: In the formula, ρ i represents spatiotemporal features, CAT represents feature concatenation operations, and CONV represents convolution operations. Represents spatial features, Represents time characteristics.

8. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 7, characterized in that: The calculation process of the advanced spatiotemporal features is: In the formula, represents the high-level spatiotemporal features after k updates, and ConvGRU represents a convolutional neural network based on a gated recurrent unit.

9. The video salient object detection method based on spatiotemporal contextual scene relationship propagation according to claim 8, characterized in that: The process of fusing low-level features and high-level spatiotemporal features within a frame to generate a saliency mask prediction result satisfies: In the formula, represents the decoder, y i Indicates the prediction result.

10. A video salient object detection system based on spatiotemporal contextual scene relationship propagation, characterized in that: include: The instance-level object acquisition module is used to perform scene analysis on each frame of video in the video frame sequence to obtain the instance-level object corresponding to each frame of video; A feature extraction module is used to extract features of each frame of video and its corresponding instance-level object, wherein the features include: global instance-level features, local instance-level features, and intra-frame low-level features; The temporal feature extraction module is used to randomly sample the matching frame of any frame of video from the same video frame sequence, extract the global instance-level features of the matching frame, integrate the global instance-level features into the global instance-level features of the matching frame, and obtain the temporal features between video frames; A spatial feature extraction module is used to integrate local instance-level features into global instance-level features using a dense spatial attention mechanism to obtain spatial features; A feature splicing module, used for splicing the time feature and the space feature to obtain a spatiotemporal feature; The high-level spatiotemporal feature generation module is used to input the spatiotemporal features and global instance-level features into the convolutional neural network based on the gated recurrent unit for updating to obtain high-level spatiotemporal features; The detection result generation module is used to fuse the low-level features and high-level spatiotemporal features within the frame and decode them to generate the mask detection results of the video salient objects.

Images