Video saliency target detection method based on image annotation

Abstract

The invention provides a video saliency target detection method based on image annotation. The method comprises the following steps: S1, establishing a training set and a test set; s2, constructing a progressive saliency target detection network; s3, training the progressive saliency target detection network by using the training set, and identifying video frames in the test set by using the coarse positioning network to generate a first space-time positioning tag; s4, acquiring adjacent frames of the video frame, and identifying the adjacent frames based on an adjacent frame difference method to generate a second space-time positioning tag; step S5, inputting the first space-time positioning tag and the second space-time positioning tag into a double-flow positioning network for positioning the video salient object; and S6, the attention sampler and the fine segmentation network are used to identify and segment the images corresponding to the video frames in sequence to obtain corresponding salient object positioning maps. The method has the beneficial effects that the marking cost of the video saliency target detection data set can be reduced, and adaptation to rich scenes is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of video salient object detection, and in particular to a video salient object detection method based on image annotation. Background Art

[0002] Video salient object detection is an important task for intelligent video content understanding. It has a wide range of applications in real-world visual tasks, such as video object segmentation, video classification, video summarization, video surveillance, video compression, and autonomous driving. Therefore, video salient object detection has always been one of the research hotspots in the field of computer vision. Unlike isolated image salient object detection, video is more challenging to capture in terms of contextual associations and rich inter-frame dynamic information. Existing methods mainly use recursive networks or introduce optical flow information to supplement dynamic information. Although video salient object detection has made great progress, limited training samples and scene diversity restrict its further development and application. The main reason is the high cost of video pixel-level annotation.

[0003] Although unsupervised methods can completely eliminate the cost of annotation using hand-crafted low-level features, the performance of existing methods is poor and only works well in a few considered scenarios. Some studies have proposed a semi-supervised saliency detection method using sparsely labeled frames, which greatly reduces the cost of manual annotation, but still requires 20% pixel-level annotation. In addition, some methods design a weakly supervised video salient object detection model by annotating the foreground and background with graffiti to reduce the burden of pixel-level annotation. Although graffiti annotation can effectively reduce time and financial costs compared to pixel-level annotation, it still requires expensive and time-consuming annotation. The main reason is not only the large scale of video datasets, but also the reliance on high-cost gaze annotations, which are achieved by tracking the eye trajectories of many people. Although these deep learning-based methods can reduce the cost of video annotation, they are difficult to handle large-scale datasets containing richer scenes in the future. Summary of the invention

[0004] The technical problem to be solved by the present invention is to reduce the annotation cost of video salient object detection datasets and achieve adaptation to rich scenes. In order to overcome the defects of the above-mentioned prior art (or related technology), the present invention provides a video salient object detection method based on image annotation.

[0005] The present invention provides a video salient object detection method based on image annotation, comprising the following steps: Step S1, obtaining a pixel-level annotated image salient object dataset and an unannotated video salient object detection dataset, and preprocessing the image salient object dataset and the video salient object detection dataset and randomly dividing them into a training set and a test set; Step S2, constructing a progressive salient object detection network, the progressive salient object detection network comprising a coarse positioning network, an attention sampler and a fine segmentation network connected in sequence, the coarse positioning network performs position positioning of salient objects in the input image to obtain a main attention map, the attention sampler performs detail magnification of salient objects in the main attention map to obtain an attention magnification map, and the fine segmentation network performs boundary refinement and image segmentation of salient objects in the attention magnification map to obtain a salient object positioning map; Step S3, according to step S2, training the progressive salient object detection network by providing the input image through the training set, and then using the trained coarse positioning network to identify the salient area of ​​the salient object in the image corresponding to the video frame in the test set to generate a first spatiotemporal positioning label; Step S4, obtaining adjacent frames of the video frame, and locating a salient area having the same salient object as the video frame in the image corresponding to the adjacent frame based on an adjacent frame difference method to generate a second spatiotemporal positioning label; Step S5, inputting the first spatiotemporal positioning tag and the second spatiotemporal positioning tag into a pre-trained dual-stream positioning network to locate salient objects in the video; Step S6, using the trained attention sampler and the fine segmentation network to sequentially identify and segment the images corresponding to the video frames after the salient objects in the video are located to obtain the corresponding salient object location map.

[0006] Compared with the prior art, the video salient object detection method based on image annotation in this application has the following advantages: The present application proposes a progressive salient object detection network, which adopts a strategy of first positioning and then segmenting. The network consists of a coarse positioning network, an attention sampler and a fine segmentation network. By decoupling the image salient object data set annotated at the pixel level, the knowledge learned from the image can be efficiently transferred to the video, and the difference in positioning and the similarity in segmentation between image salient object detection and video salient object detection are utilized without expensive video annotation to reduce the annotation cost; and a novel method for generating spatiotemporal positioning labels is proposed, including generating high-salient frames and tracking salient objects in adjacent frames, thereby indirectly transferring the rich scene knowledge learned from the image to locate the salient objects in the video, and achieving adaptation to rich scenes; and the dual-stream positioning network can bridge static and dynamic information to predict the salient object positioning map in the video. The network can adapt to the data structure of sparse spatiotemporal positioning labels and add important motion information through optical flow images, so that the salient object positioning map obtained by segmentation is more accurate.

[0007] In a possible implementation, in step S1, the process of preprocessing the image salient object dataset and the video salient object detection dataset is: The pixel values ​​of all pixels in the R channel of the images in the image salient target dataset and the video salient target detection dataset are normalized to a mean of 0.485 and a variance of 0.229, the pixel values ​​of all pixels in the G channel are normalized to a mean of 0.456 and a variance of 0.224, and the pixel values ​​of all pixels in the B channel are normalized to a mean of 0.406 and a variance of 0.225, and then the images are data enhanced by horizontal flipping or random rotation.

[0008] In a possible implementation, the coarse positioning network constructed in step S2 uses a kernel size of The binary dilation operation enlarges the input image to obtain the location distribution of salient objects and preserve the background information around the salient objects, and then applies Gaussian blur using the same kernel size as the binary dilation operation to generate the subject attention map.

[0009] In one possible implementation, the attention sampler constructed in step S2 is The function decomposes the subject attention map into two dimensions: and , where w represents the width of the subject attention map, and h represents the height of the subject attention map. Represents the pixel value, and then the sampling function is used to sample the salient object to obtain the attention magnification map: ,in, represents the attention magnification map, express The inverse function of .

[0010] In a possible implementation, the fine segmentation network constructed in step S2 performs scale homogenization and resolution improvement processing on the attention magnification map to refine the boundaries of the salient objects, and then places the high-resolution salient objects in the central area of ​​the image and compresses the background information, and obtains the salient object positioning map containing the salient objects through segmentation processing.

[0011] In a possible implementation, in step S3, the generation of the first spatiotemporal positioning label includes a static saliency prediction branch, a dynamic saliency prediction branch and a high saliency frame discriminator. For the static saliency prediction branch, the coarse positioning network is used to identify the corresponding image of the video frame to obtain a static salient area of ​​the salient object. For the dynamic saliency prediction branch, the RAFT model is used to estimate the optical flow of the video frame and render it to obtain an optical flow image. The coarse positioning network is used to identify the optical flow image to obtain a dynamic salient area of ​​the salient object. Subsequently, the high saliency frame discriminator is used to determine the similarity between the static salient area and the dynamic salient area to generate a high saliency frame and the corresponding video salient area as the first spatiotemporal positioning label.

[0012] In a possible implementation, in step S3, in the process of using the trained coarse positioning network to identify the static salient area of ​​the image corresponding to the video frame and the dynamic salient area of ​​the optical flow image, when the static salient area and the dynamic salient area contain the same object in the same frame, the object is regarded as a salient object.

[0013] In a possible implementation, in step S4, a plurality of adjacent frames of the high-saliency frame are obtained based on the high-saliency frame, a spatiotemporal positioning map of each of the adjacent frames is calculated using an adjacent frame difference method, optical flow information between each of the adjacent frames is obtained using a RAFT optical flow algorithm to calculate the displacement of the salient object, and then based on the spatiotemporal positioning map and the displacement, a remapping process is performed using the OpenCV software library to obtain the salient object positioning map of each of the adjacent frames as the second spatiotemporal positioning label. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the steps of the present invention; Figure 2 Schematic diagram of the framework of the progressive salient object detection network of the present invention; Figure 3 A schematic diagram of a framework for generating a spatiotemporal positioning tag of the present invention; Figure 4A schematic diagram of a framework for expanding spatiotemporal positioning tags based on the adjacent frame tracking method of the present invention; Figure 5 It is a schematic diagram of the framework of the dual-stream positioning network of the present invention; Figure 6 The figure is a schematic diagram of the comparison results between the method of the present invention and the existing method on an open data set. DETAILED DESCRIPTION

[0015] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0016] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] See also Figure 1 The present application embodiment discloses a video salient object detection method based on image annotation, comprising: Step S1: Obtain an image salient object dataset and an unlabeled video salient object detection dataset, and preprocess the image; Step S2: construct a progressive salient object detection network based on knowledge review network (PAKRN) and train it using the preprocessed image input; Step S3: using the coarse localization network trained in step S2 to identify static salient regions in the video frame and dynamic salient regions in its optical flow image, when the static salient regions and the dynamic salient regions contain the same object in the same frame, these objects are regarded as video salient objects; by analyzing the similarity between the static and dynamic salient regions, high-salient frames are identified and their video salient regions are located, thereby generating spatiotemporal localization labels; Step S4: According to the continuity of salient objects in adjacent frames, the salient regions containing the same salient objects can be located by using the adjacent frame tracking method, thereby generating richer spatiotemporal localization labels; Step S5: input the spatiotemporal positioning labels generated in steps S3 and S4 into the dual-stream positioning network to train the positioning of salient objects in the video; Step S6: After locating the salient objects in step S5, the video frames are processed using the attention sampler, and the trained fine segmentation network is used to directly perform segmentation, and the segmentation results are mapped back to the original format.

[0018] Continue to see Figure 1, the step S1 specifically comprises: normalizing the image salient target dataset and the video salient target detection dataset of the input image, normalizing the pixel values ​​of all pixels in the R channel of the image to a mean of 0.485 and a variance of 0.229, normalizing the pixel values ​​of all pixels in the G channel to a mean of 0.456 and a variance of 0.224, and normalizing the pixel values ​​of all pixels in the B channel to a mean of 0.406 and a variance of 0.225, and then performing data enhancement on the image using horizontal flipping and random rotation with a probability of 0.5.

[0019] See also Figure 2 , the knowledge review-based progressive salient object detection network includes a coarse localization network (CLM), an attention sampler, and a fine segmentation network (FSM), as follows: (1) Coarse positioning network The goal of the coarse localization network CLM is to find the precise location of the salient object, laying a solid foundation for subsequent accurate segmentation. Its training label is the subject attention map, which mainly focuses on the location information to guide the network. In order to fully include the salient object and smooth the edge, the kernel size is The binary dilation operation enlarges the label. In addition, to help the model learn the object location distribution and preserve the surrounding background information, Gaussian blur is applied. The kernel size is 8, and the main attention map is generated using the same kernel size as the binary dilation operation. The main idea is to sample the pixels of the original image according to the attention value of the main attention map; (2) Attention Sampler After obtaining the subject attention maps, they are used to improve the resolution of the areas related to the salient objects in the image, thereby amplifying the details of the salient objects. The salient objects are enlarged to be closer to the image size, which will narrow the scale gap between the salient objects in the dataset. The sampler is based on the original image. And the subject attention map As input, generate an image that retains salient objects. First, pass The function decomposes the subject attention map into two dimensions: , , where w and h are the original images The width and height of yes The pixel value is then calculated to achieve sampling by calculating the inverse function of the distribution function. The sampling function is defined as follows: ,in yes is the inverse function of , specifically, areas with higher attention values ​​are sampled more densely; (3) The fine segmentation network is similar to other salient object detection (SOD) networks in terms of tasks and functions. FSM also needs to complete the tasks of salient object localization and segmentation. The biggest difference is that the input images of FSM have been preprocessed. The salient objects in these images have more uniform scales and higher resolutions, which can effectively help refine the boundaries of salient objects. In addition, in the processed images, high-resolution salient objects are located in the central area, while the background is compressed, which greatly reduces the difficulty of locating salient objects. In the last step, the output will be restored to the saliency map corresponding to the original image.

[0020] Continue to see Figure 2 The coarse positioning network and fine segmentation network structure are based on the knowledge review network (KRN). KRN is based on the U-shaped feature pyramid network (FPNs) and uses the pre-trained ResNet-50 as the backbone network. It is a bottom-up and top-down encoder-decoder that can fully combine multi-scale features to obtain rich semantic information. Among them, the knowledge review (KR) module is introduced to review the unlearned and diluted information by recombining the finest feature map with the features of each layer. After fusion, there are five groups of feature maps, named and , arranged from low to high resolution, different groups of feature maps retain different degrees of details and semantic information, but their sizes and numbers vary; first, by Convolution will and The channel is compressed to The same, then, through the upsampling operation to adjust these feature maps to The same size, then, by adding pixel by pixel Convolution combines them with To supplement the diluted and undiscovered important information, intermediate supervision is added to guide all feature maps to retain only useful information related to salient objects. Next, these four sets of fused feature maps are concatenated and The final saliency map is integrated through convolution. Convolution and upsampling operations generate, and the re-fusion of features in the KR module can make up for the missing or unfused useful information, but obtaining useful information as efficiently as possible during the feature integration process can further improve the usability of features. To this end, a simple SA module is added to the feature integration process to improve the efficiency of feature fusion; similar to FPNs, through multiple downsampling, average pooling and Convolution obtains multi-scale feature maps, and then simply merges all feature maps and performs a Convolution filtering, by combining multi-scale features, can extract more comprehensive information from different scale spaces to avoid missing important information; in addition, the SA module can further enhance the receptive field of the entire network. It should be noted that KRN is used in both CLM and FSM, but there are differences between them. Like the general salient object detection (SOD) network, FSM needs to accurately distinguish salient objects at the pixel level. Therefore, intermediate edge supervision is added to guide the features provided in the encoding process to have clear boundaries.

[0021] Continue to see Figure 2 , the loss function of body-attention supervision The calculation formula is as follows: , is a variant of the normalized glance path saliency, It is a variant of the Pearson correlation coefficient. The predicted subject attention map is recorded as P, and the annotated subject attention map is recorded as Q. Then, pixels with a value of 255 are extracted from Q to construct a new gaze true map F. This area represents the high probability area of ​​​​the salient target. NSS is used to measure the average standardized value of P at the gaze prediction point in the gaze true map F, emphasizing the importance of these gaze points for saliency detection. The NSS loss function is recorded as In the task of locating salient objects, we need to focus on the high-value pixels in F. The modified The loss function is: ,in, Indicates pixels, is the number of pixels with a value of 255 in F, function and Represent the mean and standard deviation of the image respectively. The Pearson correlation coefficient CC is often used to measure the linear correlation between two variables: is the covariance of P and Q, the modified Denoted as: , KLD is used to evaluate the similarity between two distributions. A lower KLD value indicates a higher similarity between P and S, which reflects a better performance of the model. The calculation process is: ,in, is a regularization term, and the loss function of the coarse positioning network is as follows: ,in, It is The loss function of the subject's attention supervision in the middle, and Set to 2 and 1; the loss function of saliency supervision is as follows: ,in and Binary cross entropy (BCE) loss and intersection over union (IoU) loss, respectively, and saliency supervision after sampling The loss function is the same, except that the corresponding truth map is sampled. is the loss function of edge supervision, constructed using binary cross entropy loss. The loss function of FSM is as follows: ,in, For the The significance loss after sampling in the middle, For the The loss of marginal supervision after sampling in the middle, , and are set to 2, 1, and 1 respectively; in the joint training of the coarse positioning network and the fine segmentation module, the total loss L is as follows: .

[0022] See also Figure 3 The generation of spatiotemporal localization labels includes a static saliency prediction branch, a dynamic saliency prediction branch, and a high saliency frame discriminator. For the static saliency prediction branch, the CLM trained on the image dataset is used to find static salient areas. In the dynamic saliency prediction branch, RAFT is used to estimate the optical flow of the video frame and render it. When the motion of the object is different from the background, its area in the rendered optical flow image has obvious color and clear boundaries; therefore, the dynamic salient area can be easily detected using CLM. After obtaining the predicted volume attention position map of the static frame and the optical flow image, the high saliency frame discriminator is used to determine whether their salient areas are similar. The present invention designs strict discrimination conditions to ensure the high quality of the generated high saliency frames. The intersection over union (IoU) is used to measure the volume attention generated by the static frame and the optical flow image. similarity of the graph; when the IoU value of a frame is higher than the threshold of 0.7, the frame is considered to be a high-saliency frame. In order to avoid some situations where the optical flow quality is poor, five optical flow images are generated at different frame intervals to obtain five dynamic saliency maps. Only one of them needs to meet the above conditions. If the quality of all optical flow images is poor, the frame will not be selected as a high-saliency frame. Therefore, the occurrence of optical flow model failure has little effect on the process. For these high-saliency frames, the static image salient areas predicted by CLM are used as spatiotemporal positioning labels; since the dynamic salient area may not contain the complete object due to the local motion of the salient object, the static salient area is obtained through the trained CLM and can contain the complete object with a higher probability. Finally, these frames and their corresponding salient object positioning maps are used as part of the training data for the subsequent dual-stream positioning network learning.

[0023] See also Figure 4 , given a high saliency frame And its corresponding spatiotemporal location diagram , using the adjacent frame tracking method to calculate the adjacent frame The space-time location map ,in express The first six frames and the last six frames of the image are obtained based on the RAFT optical flow algorithm. With adjacent frames The optical flow information between them is used to obtain the displacement of the salient object. According to the spatiotemporal localization map and optical flow image of the high-saliency frame, the salient object localization map of the adjacent frame is calculated by the remapping function of OpenCV. However, this may cause the remapped salient object localization map to not contain the complete object, and such spatiotemporal localization labels lack the incorrect shape of the salient object, which may reduce the performance of subsequent training. To avoid the above problems, a wider area containing the remapped salient object is cropped, and its salient object localization map is re-calculated using CLM. Specifically, the remapped salient object localization map is first dilated and eroded, and then the bounding box of the connection domain is found, and the size of the bounding box is doubled to contain the complete object. According to the coordinates of these bounding boxes, local images containing salient objects are generated by cropping adjacent frames. Finally, CLM is used to predict the localization maps of these local images and map them back to the original size. In addition, it may also happen that the adjacent frames can be high-saliency frames or different spatiotemporal localization maps are generated by different high-saliency frames. For this case, the spatiotemporal localization map is selected according to the proximity principle.

[0024] See also Figure 5The dual-stream localization network is mainly composed of a static feature extraction branch, a dynamic feature extraction branch and a feature fusion module. The static feature extraction branch and the dynamic feature extraction branch are based on the CLM pre-trained on the image salient target detection dataset as the backbone network, which can provide multi-scale features with rich semantic information. The input of the static feature extraction branch is an RGB image, while the input of the dynamic feature extraction branch is five optical flow images. To obtain these optical flow images, five images from the first five frames and the last five frames are randomly selected to calculate the optical flow image with the current frame. If the optical flow branch only inputs one optical flow frame, the network result is easily affected by the quality of the optical flow image. However, the five optical flow images contain more or less Different motion information can provide richer information. In addition, random selection is similar to data enhancement, which can improve the generalization ability of the model. The feature fusion module not only needs to combine static features and dynamic features to obtain more comprehensive semantic information, but also needs to restore the spatial size of the feature map and predict the high-resolution saliency map by fusing high-level and low-level features. The basic structure of this module is the same as the decoder of FPN. It integrates features from top to bottom layer by layer to obtain comprehensive information. The main difference is that the side aggregation (SA) module is introduced after the fusion of each layer of features. In the feature fusion module, the output feature maps of the static feature extraction branch and the dynamic feature extraction branch are used as inputs, respectively denoted as and ; Since the quality of each input optical flow image in the dynamic feature extraction branch is different, a weight is added to each feature map; Specifically, the feature map of the final output layer is obtained by The convolution is compressed into one dimension, and the weight of a certain optical flow image is obtained through the average pooling operation. These weights are then multiplied with the feature map of the corresponding optical flow image. Finally, the feature maps of the five optical flow images are added as the input of the feature fusion module. In addition, intermediate supervision is added after each fusion to guide the network to retain only useful information related to salient objects. Feature integration includes four steps. In the first fusion, , , processed by SA module and processed by SA module The output is summed at the pixel level Convolutional combination, in the subsequent series of fusion, the corresponding static features, the corresponding motion features and the features processed by the SA module from the previous layer are summed up at the pixel level Convolutional fusion, moreover, adds intermediate supervision after each fusion to guide the network to retain useful information only related to salient objects.

[0025] Continue to see Figure 5, This embodiment uses a combined loss function in the loss function of the subject attention graph supervision, which is composed of the linear correlation coefficient (CC), Kullback-Leibler divergence (KLD) and normalized scan path saliency (NSS), denoted as: , the loss function of the total two-stream positioning network is: ,in Indicates The loss of the intermediate attention supervision.

[0026] Continue to see Figure 1 ,In order to further verify the feasibility and effectiveness of the method of the present invention, the method of the present invention was experimented; In the experiment, we directly selected public video salient object detection datasets for training and testing, including ViSal, VOS, FBMS, DAVIS, and DAVSOD. We used three indicators to evaluate and compare different video salient object detection methods, including MAE, max F-measure, and S-measure. The comparison results are shown in the figure below. Figure 6 As shown, the following Tables 1 and 2 give quantitative comparison tables of the evaluation indicators of the method of the present invention and common image salient object detection methods and video salient object detection methods on ViSal, VOS, FBMS, DAVIS, and DAVSOD datasets: Table 1 Quantitative comparison of the evaluation indicators of the proposed method and common image / video salient object detection methods on ViSal, VOS and DAVSOD datasets ; Table 2 Quantitative comparison of the evaluation indicators of the proposed method and common image / video salient object detection methods on the FBMS and DAVIS datasets ; It can be seen from Table 1 and Table 2 above that the method of the present invention has better evaluation indicators.

[0027] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0028] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video salient object detection method based on image annotation, characterized in that: The following steps are involved: Step S1, obtaining a pixel-level annotated image salient object dataset and an unannotated video salient object detection dataset, and preprocessing the image salient object dataset and the video salient object detection dataset and randomly dividing them into a training set and a test set; Step S2, constructing a progressive salient object detection network, the progressive salient object detection network comprising a coarse positioning network, an attention sampler and a fine segmentation network connected in sequence, the coarse positioning network performs position positioning of salient objects in the input image to obtain a main attention map, the attention sampler performs detail magnification of salient objects in the main attention map to obtain an attention magnification map, and the fine segmentation network performs boundary refinement and image segmentation of salient objects in the attention magnification map to obtain a salient object positioning map; Step S3, according to step S2, training the progressive salient object detection network by providing the input image through the training set, and then using the trained coarse positioning network to identify the salient area of ​​the salient object in the image corresponding to the video frame in the test set to generate a first spatiotemporal positioning label; Step S4, obtaining adjacent frames of the video frame, and locating a salient area having the same salient object as the video frame in the image corresponding to the adjacent frame based on an adjacent frame difference method to generate a second spatiotemporal positioning label; Step S5, inputting the first spatiotemporal positioning tag and the second spatiotemporal positioning tag into a pre-trained dual-stream positioning network to locate salient objects in the video; Step S6, using the trained attention sampler and the fine segmentation network to sequentially identify and segment the images corresponding to the video frames after the salient objects in the video are located to obtain the corresponding salient object location map.

2. The video salient object detection method based on image annotation according to claim 1 is characterized in that: In step S1, the process of preprocessing the image salient object dataset and the video salient object detection dataset is as follows: The pixel values ​​of all pixels in the R channel of the images in the image salient target dataset and the video salient target detection dataset are normalized to a mean of 0.485 and a variance of 0.229, the pixel values ​​of all pixels in the G channel are normalized to a mean of 0.456 and a variance of 0.224, and the pixel values ​​of all pixels in the B channel are normalized to a mean of 0.406 and a variance of 0.225, and then the images are data enhanced by horizontal flipping or random rotation.

3. The video salient object detection method based on image annotation according to claim 1 is characterized in that: The coarse positioning network constructed in step S2 uses a kernel size of The binary dilation operation enlarges the input image to obtain the location distribution of salient objects and preserve the background information around the salient objects, and then applies Gaussian blur using the same kernel size as the binary dilation operation to generate the subject attention map.

4. The video salient object detection method based on image annotation according to claim 1 is characterized in that: The attention sampler constructed in step S2 is The function decomposes the subject attention map into two dimensions: and , where w represents the width of the subject attention map, and h represents the height of the subject attention map. Represents the pixel value, and then the sampling function is used to sample the salient object to obtain the attention magnification map: ,in, represents the attention magnification map, express The inverse function of .

5. The video salient object detection method based on image annotation according to claim 1, characterized in that: The fine segmentation network constructed in step S2 performs scale homogenization and resolution improvement processing on the attention magnification map to refine the boundaries of the salient objects, then places the high-resolution salient objects in the central area of ​​the image and compresses the background information, and obtains the salient object positioning map containing the salient objects through segmentation processing.

6. The video salient object detection method based on image annotation according to claim 1, characterized in that: In step S3, the generation of the first spatiotemporal positioning label includes a static saliency prediction branch, a dynamic saliency prediction branch and a high saliency frame discriminator. For the static saliency prediction branch, the coarse positioning network is used to identify the corresponding image of the video frame to obtain a static salient area of ​​the salient object. For the dynamic saliency prediction branch, the RAFT model is used to estimate the optical flow of the video frame and render it to obtain an optical flow image. The coarse positioning network is used to identify the optical flow image to obtain a dynamic salient area of ​​the salient object. Subsequently, the high saliency frame discriminator is used to determine the similarity between the static salient area and the dynamic salient area to generate a high saliency frame and the corresponding video salient area as the first spatiotemporal positioning label.

7. The method for detecting salient objects in a video based on image annotation according to claim 6, characterized in that: In the step S3, in the process of using the trained coarse positioning network to identify the static salient area of ​​the image corresponding to the video frame and the dynamic salient area of ​​the optical flow image, when the static salient area and the dynamic salient area contain the same object in the same frame, the object is regarded as a salient object.

8. The method for detecting salient objects in a video based on image annotation according to claim 6, characterized in that: In the step S4, a plurality of adjacent frames of the high-saliency frame are obtained based on the high-saliency frame, a spatiotemporal positioning map of each of the adjacent frames is calculated using an adjacent frame difference method, optical flow information between each of the adjacent frames is obtained using a RAFT optical flow algorithm to calculate the displacement of the salient object, and then based on the spatiotemporal positioning map and the displacement, a remapping process is performed using the OpenCV software library to obtain the salient object positioning map of each of the adjacent frames as the second spatiotemporal positioning label.

Images