A video object segmentation method and system based on autonomous frame selection and edge refinement

Abstract

A video object segmentation method based on autonomous frame selection and edge refinement, comprising the following steps: first, the user selects an object to be segmented and simply marks the boundary of the object. Then, a general video object segmentation algorithm is used to segment the target object, and the segmentation quality of each frame is estimated. Then, the autonomous frame selection module inputs the segmentation quality to recommend the most valuable frame to the user, and then inputs the frame into the edge refinement module for optimization. Finally, the video object segmentation algorithm, the autonomous frame selection module, the edge refinement module and the user form a cycle, and the predicted mask is iteratively optimized. After the iteration is completed, the final target segmentation result is obtained. Finally, each segmentation result is input into a visual analysis system to visualize the optimization process. The application also includes a video object segmentation system based on autonomous frame selection and edge refinement. The application achieves a good balance between segmentation accuracy and time consumption.

Description

Technical Field

[0001] This invention relates to a video target segmentation method and system based on autonomous frame selection and edge refinement. Background Technology

[0002] With the rapid development of internet information technology, people's demand for image information is increasing, and video has become an important carrier of information dissemination, leading to a proliferation of various video applications. Consequently, the demand for video processing is also growing, and how to extract key information from videos has become a focus of attention for many scholars. In early video processing, there was an over-reliance on manual viewing and retrieval of key information. However, with the continuous increase in video data and the growing amount of information contained within videos, this manual retrieval method is no longer practical, and utilizing the powerful computing capabilities of computers has become an important approach.

[0003] Video object segmentation (VOS) aims to separate one or more objects of interest from the background in a given video. Unlike image segmentation, video object segmentation utilizes temporal information between frames to continuously segment the pixels of the object of interest, combining pixel tracking and object segmentation.

[0004] Video object segmentation can be mainly divided into two types: unsupervised and semi-supervised. Unsupervised video object segmentation, lacking human interaction, often relies solely on the algorithm's understanding of the video to automatically segment objects, which often falls short of expectations. Semi-supervised video object segmentation, on the other hand, requires manual annotation of the first frame. Currently, semi-supervised video object segmentation still faces many challenges, two of which are particularly critical. First, the interactive selection of keyframes is extremely time-consuming, and collecting the dataset requires significant time investment. To reduce the cost of manual annotation, this invention designs an interactive video object segmentation method and system, aiming to achieve satisfactory segmentation results with minimal human supervision. Existing methods typically select the worst-performing frame to correct the segmentation result. However, the frame with the worst quality evaluation metric is not necessarily the most helpful for improving the overall video segmentation quality. Second, the quality of existing instance segmentation masks remains unsatisfactory. Existing methods propose position consistency and target consistency, where target consistency refers to the regularity of target movement between frames. If the positions of some entities in a frame are unreasonable, the probability that the entity is a target object decreases. Target consistency implies that object processing requires an object-level constraint. However, existing research results still show coarse object boundaries and lack alignment. In conclusion, research on video object segmentation techniques based on autonomous frame selection and edge refinement has significant research value and economic benefits. Summary of the Invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a video target segmentation method and system based on autonomous frame selection and edge refinement.

[0006] This invention, through an intelligent keyframe recommendation module and a keyframe edge optimization module, can quickly and accurately segment selected targets in a video. The main focus of this invention is to reduce manual intervention while maintaining flexibility in keyframe selection. During the refinement of keyframe targets, the emphasis is on refining the target edges. Initially, the user selects a target object of greatest interest and simply marks its boundaries. Then, the general VOS algorithm is used to segment the target object, and the segmentation quality of each frame is estimated. The segmentation quality is then used as input to recommend the most valuable frame to the user, and this frame undergoes edge optimization. The VOS algorithm, agent, and user then form a loop to iteratively optimize the predicted mask. Finally, the results of each iteration are input into the visual analysis system.

[0007] The video target segmentation method based on autonomous frame selection and edge refinement of the present invention includes the following steps:

[0008] (1) Obtain the video dataset and extract visual and semantic features for each frame in the video dataset;

[0009] (2) Autonomous Recommendation of Keyframes; This invention compares the performance of the recommendation strategy and the random selection strategy by obtaining the mean and variance of the segmentation results, and designs a reward function to enable the autonomous frame selection module to learn how to recommend keyframes. The analysis process is as follows:

[0010] (2-1) Label keyframes and provide initial segmentation results; The user provides a video to be segmented. The user first selects a target to be segmented and selects a frame containing the target for labeling. Next, a general segmentation algorithm is used to segment the video target provided by the user and obtain the segmentation results.

[0011] (2-2) Learning Recommended Frames; This invention designs a network based on bidirectional long short-term memory for autonomous learning. The input of this network is the state, and the output is a series of actions. The agent's role is to recommend keyframe indices. This invention uses the quality of the segmentation result as the score for each frame. Furthermore, this invention also considers the number of times each frame has been recommended historically. Therefore, state C... t The statistical indicator is defined by formula (1):

[0012] C t =CONCAT(f m h m (1)

[0013] f acts as a proxy for the mask, h represents historical recommendations, and the index m is the maximum number of iterations. CONCAT represents the join operation.

[0014] (2-3) Setting the reward function; To achieve the optimal result, a target reward is designed based on the performance obtained from the time sequence of the target action to be segmented, until the defined number of iterations is reached. In interactive VOS, it is difficult to calculate the value of each frame for the overall video sequence, so this invention designs a target reward based on the final performance f achieved by the action sequence, until the maximum number of iterations t is reached. The strategy learned by this invention has a higher average performance than the randomly selected strategy. In order to obtain the performance p of the randomly selected strategy, n experiments are first conducted on each training video sequence using the randomly selected strategy. By comparing the performance between the recommended strategy and the randomly selected strategy, a corresponding reward function can be designed. Therefore, the following reward function formula (2) is proposed:

[0015]

[0016] in The reward metric after the maximum number of iterations is represented, where P represents performance, u represents the expected average, and v represents the variance.

[0017] (2-4) Optimize the reward function; when the performance P is greater than the average performance u of the random selection strategy, the reward in formula (2) is positive; otherwise, the reward will be negative. In practice based on real-world scenarios, it was found that simply making the performance better than the average performance of the random selection strategy is not enough. This invention only sets the reward to positive when the performance P is greater than u+v, and the final reward function is formula (3):

[0018]

[0019] (2-5) Segmentation Quality Assessment; To assess segmentation quality, this invention uses a quality assessment module to evaluate the segmentation quality of each frame. First, the bounding box of the segmented target is calculated based on the segmentation probability map. Then, the bounding box is enlarged by a factor of x. To ignore background regions, this invention crops the probability map corresponding to the RGB image based on the enlarged bounding box. Then, the cropped RGB image and the probability map are concatenated as input to the segmentation quality assessment module to obtain the segmentation quality estimate for each object of interest.

[0020] (3) Optimize keyframe edges; After recommending keyframes through the above steps, edge optimization is performed on the mask of these keyframes. Two key issues lead to low-quality boundary segmentation: firstly, the low spatial resolution of the output; and secondly, the pixels around the object boundary only occupy a small portion of the entire image, making classification difficult. This invention improves these two problems by refining the network and extracting edge image patches. The analysis process is as follows:

[0021] (3-1) Extracting edge image patches: Given an instance mask of a keyframe selected by an autonomous frame selection module, the edges of this mask are then refined. This invention uses a sliding window method to extract a series of image patches at the edges, densely allocating a set of square bounding boxes at the mask boundaries. The central regions of these bounding boxes should cover the boundary pixels. Because the obtained boxes still contain overlap and redundancy, they are filtered. Next, the extracted edge image patches and mask patches are reshaped to the same size and fed together into the refinement network.

[0022] (3-2) Refinement Network: This invention extracts and refines a series of small boundary image patches along the predicted target mask boundary. Refinement is accomplished through a higher-resolution boundary image patch refinement network. The refinement network performs semantic segmentation separately for each extracted boundary image patch. The refinement network can fuse features at various levels, improving high-resolution performance. By increasing the input size, the boundary image patches can achieve higher resolution than previous methods. The edge refinement module of this invention first extracts a series of small image patches along the predicted target boundary. After fusion with the mask, the boundary image patches are fed into the refinement network, which performs semantic segmentation to refine the coarse boundaries. Subsequently, the optimized mask is reassembled into a higher-quality target mask.

[0023] (3-3) Reconstructing the target mask: After refining each image patch, they need to be reconstructed onto the target mask. Adjacent image patches may overlap; the average value is taken as the final result to determine whether a pixel belongs to the target to be segmented or the background.

[0024] (4) Iteratively optimize the predicted mask and display the results of each iteration in the visualization analysis system; customize the number of iterations, and each iteration will perform an autonomous frame selection, and then put the keyframes recommended by the system into the edge refinement module. After completing one iteration, input the feature values ​​of each iteration, the score of each frame, and the average score of each iteration into the visualization analysis system, and clearly display the results of each iteration through heatmaps, bar charts, line area charts, and the frame-by-frame video display module.

[0025] Preferably, n in step (2-3) is set to between 10 and 20.

[0026] Preferably, in step (2-5), x is set to between 2 and 5.

[0027] Preferably, the Yes-Net algorithm is used for filtering in step (3-1).

[0028] Preferably, the refinement network in step (3-2) adopts a multi-HRNet network.

[0029] The present invention also includes a video target segmentation system based on autonomous frame selection and edge refinement, comprising:

[0030] The video dataset acquisition module is used to acquire video datasets and extract visual and semantic features from each frame in the video dataset.

[0031] A keyframe autonomous recommendation module is used to autonomously recommend keyframes. The performance of the recommendation strategy and the random selection strategy is compared by obtaining the mean and variance of the segmentation results. A reward function is designed to allow the autonomous frame selection module to learn how to recommend keyframes.

[0032] The keyframe optimization module is used to optimize the edges of keyframes. After keyframe recommendation is completed through the above steps, edge optimization is performed on the mask of this keyframe. Two key issues lead to low-quality boundary segmentation: firstly, the low spatial resolution of the output; and secondly, the pixels around the object boundary only occupy a small portion of the entire image, making classification difficult. These two issues are improved by refining the network and extracting edge image patches.

[0033] The mask iteration optimization module is used to iteratively optimize the predicted mask and display the results of each iteration in the visualization analysis system. The number of iterations can be customized. Each iteration will perform an autonomous frame selection and then put the keyframes recommended by the system into the edge refinement module. After completing an iteration, the feature values ​​of each iteration, the score of each frame, and the average score of each iteration are input into the visualization analysis system. The results of each iteration are clearly displayed through heatmaps, bar charts, line area charts, and the frame-by-frame video display module.

[0034] The advantages of this invention are: This invention optimizes the video target segmentation algorithm through an autonomous frame selection module and an edge refinement module. The autonomous frame selection module provides the user with the most valuable keyframe for annotation, achieving optimal overall segmentation results. The edge refinement module optimizes the mask quality of the keyframe, disseminating keyframe information throughout the entire video sequence. The video target segmentation method proposed in this invention, combining these two modules, exhibits excellent segmentation accuracy. Furthermore, this invention integrates video target segmentation with visual analysis, enabling users to gain a deeper understanding of the segmentation process and achieving a good balance between segmentation accuracy and time consumption. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the overall architecture of the method of the present invention.

[0036] Figure 2 The diagram illustrates the visual analysis system of the present invention, wherein module 1 is a feature-reflected area map, module 2 is a heatmap of frames in the track selection, module 3 is a bar chart, and module 4 is a track of the video sequence.

[0037] Figure 3 This is a structural diagram of the system of the present invention. Detailed Implementation Plan

[0038] This invention relates to a video target segmentation method and system based on autonomous frame selection and edge refinement. A specific implementation scheme is used as an example to segment a moving pet. The invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0039] Example 1

[0040] The following is in conjunction with the appendix Figures 1-2 A video target segmentation method based on autonomous frame selection and edge refinement includes the following steps:

[0041] (1) Obtain a pet movement video dataset. The pet dataset is 5 hours and 16 minutes long and 21.3G in size. In this implementation, HOG feature extraction is used and combined with SVM classifier to extract visual and semantic features for each frame in the video dataset.

[0042] (2) Autonomous Keyframe Recommendation; This invention compares the performance of the recommendation strategy and the random selection strategy by obtaining the mean and variance of the segmentation results, and designs a reward function to enable the autonomous frame selection module to learn how to recommend keyframes. The flowchart of this module is as follows. Figure 1 As shown in the autonomous frame selection module, the analysis process is as follows:

[0043] (2-1) Label keyframes and provide initial segmentation results; The user provides a video to be segmented. The user first selects a target to be segmented and selects a frame containing the target for labeling. Next, a general segmentation algorithm is used to segment the video target provided by the user and obtain the segmentation results.

[0044] (2-2) Learning Recommended Frames; This invention designs a network based on bidirectional long short-term memory for autonomous learning. The input of this network is the state, and the output is a series of actions. The agent's role is to recommend keyframe indices. This invention uses the quality of the segmentation result as the score for each frame. Furthermore, this invention also considers the number of times each frame has been recommended historically. Therefore, state C... t The statistical indicator is defined by formula (1):

[0045] C t =CONCAT(f m ,h m (1)

[0046] f acts as a proxy for the mask, h represents historical recommendations, and the index m is the maximum number of iterations. CONCAT represents the join operation.

[0047] (2-3) Setting the reward function; To achieve the optimal result, a target reward is designed based on the performance obtained from the time sequence of the target action to be segmented, until the defined number of iterations is reached. In interactive VOS, it is difficult to calculate the value of each frame for the overall video sequence, so this invention designs a target reward based on the final performance f achieved by the action sequence, until the maximum number of iterations t is reached. The strategy learned by this invention has higher performance than the average performance of the randomly selected strategy. In order to obtain the performance p of the randomly selected strategy, 10 experiments are first conducted on each training video sequence using the randomly selected strategy. By comparing the performance between the recommended strategy and the randomly selected strategy, a corresponding reward function can be designed. Therefore, the following reward function formula (2) is proposed:

[0048]

[0049] in The reward metric after the maximum number of iterations is represented, where P represents performance, u represents the expected average, and v represents the variance.

[0050] (2-4) Optimize the reward function; when the performance P is greater than the average performance u of the random selection strategy, the reward in formula (2) is positive; otherwise, the reward will be negative. In practice based on pet movement scenarios, it was found that simply making the performance better than the average performance of the random selection strategy is not enough. This invention only sets the reward to positive when the performance P is greater than u+v, and the final reward function is formula (3):

[0051]

[0052] (2-5) Segmentation Quality Assessment; To assess segmentation quality, this invention uses a quality assessment module to evaluate the segmentation quality of each frame. First, the bounding box of the segmented target is calculated based on the segmentation probability map. Then, the bounding box is enlarged by a factor of 3. To ignore the background region, this invention crops the probability map corresponding to the RGB image based on the enlarged bounding box. Then, the cropped RGB image and the probability map are concatenated as input to the segmentation quality assessment module to obtain the segmentation quality estimate of the target pet object.

[0053] (3) Optimize keyframe edges; After recommending keyframes through the above steps, edge optimization is performed on the mask of these keyframes. Two key issues lead to low-quality boundary segmentation: firstly, the low spatial resolution of the output; and secondly, the pixels around the object boundary only occupy a small portion of the entire image, making classification difficult. This invention improves these two problems by refining the network and extracting edge image patches. The analysis process is as follows:

[0054] (3-1) Extracting Edge Image Patches: Given an instance mask of a keyframe selected by an autonomous frame selection module, the edges of this mask are then refined. This invention uses a sliding window method to extract a series of image patches at the edges, densely allocating a set of square bounding boxes at the mask boundaries. The central regions of these bounding boxes should cover the boundary pixels. Because the obtained boxes still contain overlaps and redundancies, this invention uses the Yes-Net algorithm for filtering. Next, the extracted edge image patches and mask patches are reshaped to the same size and fed together into the refinement network.

[0055] (3-2) Refinement Network: This invention extracts and refines a series of small boundary image patches along the predicted target mask boundary. Refinement is accomplished through a higher-resolution boundary image patch refinement network. This network performs semantic segmentation individually for each extracted boundary image patch. It has 6 input channels and 4 output channels. This invention employs a multi-HRNet network, which can fuse features at various levels, improving high-resolution performance. By increasing the input size, the boundary image patches achieve a higher resolution than previous methods. The edge refinement module of this invention first extracts a series of small image patches along the predicted target boundary. After fusion with the mask, the boundary image patches are fed into the refinement network, which performs semantic segmentation to refine the coarse boundaries. Subsequently, the optimized mask is reassembled into a higher-quality target mask.

[0056] (3-3) Reconstruct the target mask; After refining each image patch, they need to be reconstructed onto the target mask. Adjacent image patches may overlap. The average value is taken as the final result, and a threshold between 0.63 and 0.72 is used to determine whether a pixel belongs to a pet instance or the background.

[0057] (4) Iteratively optimize the predicted mask and display the results of each iteration in the visual analysis system; the method allows for a custom number of iterations, with each iteration performing an autonomous frame selection, and then placing the keyframes recommended by the system into the edge refinement module. The iterative process is as follows: Figure 1 As shown in the diagram. After each iteration, the segmentation results of the user-labeled target can be obtained from the pet's movement video data. Then, the feature values ​​of each iteration, the score of each frame, and the average score of each iteration are input into the visual analytics system, as shown in the diagram. Figure 2 As shown. Figure 2 Module 1 is a feature polygonal area plot, which is used to display the time distribution of the input features in each iteration. Figure 2 Module 2 is a heatmap, which represents the target heatmap of the selected frame in Module 4. The higher the saturation, the greater the probability that the pixel is the segmentation target. Figure 2Module 3 is a bar chart where the horizontal axis represents the sequence number of each iteration, and the vertical axis represents the average score of each iteration. Figure 2 Module 4 is the frame-by-frame video display module, where you can click on the horizontal axis to see the score of that specific frame.

[0058] This invention discloses a video target segmentation method based on autonomous frame selection and edge refinement, comprising the following steps: First, the user selects an object to be segmented and simply marks its boundaries. Then, a general video target segmentation algorithm is used to segment the target object, and the segmentation quality of each frame is estimated. Next, an autonomous frame selection module takes the segmentation quality as input and recommends the most valuable frame to the user, then puts that frame into an edge refinement module for optimization. Finally, the video target segmentation algorithm, the autonomous frame selection module, the edge refinement module, and the user form a loop to iteratively optimize the predicted mask, obtaining the final target segmentation result after iteration. Finally, each segmentation result is input into a visual analysis system to visualize the optimization process.

[0059] Example 2

[0060] See attached document Figures 1-2 The present invention also includes a video target segmentation system based on autonomous frame selection and edge refinement, comprising:

[0061] The video dataset acquisition module is used to acquire video datasets and extract visual and semantic features from each frame in the video dataset.

[0062] A keyframe autonomous recommendation module is used to autonomously recommend keyframes. The performance of the recommendation strategy and the random selection strategy is compared by obtaining the mean and variance of the segmentation results. A reward function is designed to allow the autonomous frame selection module to learn how to recommend keyframes.

[0063] The keyframe optimization module is used to optimize the edges of keyframes. After keyframe recommendation is completed through the above steps, edge optimization is performed on the mask of this keyframe. Two key issues lead to low-quality boundary segmentation: firstly, the low spatial resolution of the output; and secondly, the pixels around the object boundary only occupy a small portion of the entire image, making classification difficult. These two issues are improved by refining the network and extracting edge image patches.

[0064] The mask iteration optimization module is used to iteratively optimize the predicted mask and display the results of each iteration in the visualization analysis system. The number of iterations can be customized. Each iteration will perform an autonomous frame selection and then put the keyframes recommended by the system into the edge refinement module. After completing an iteration, the feature values ​​of each iteration, the score of each frame, and the average score of each iteration are input into the visualization analysis system. The results of each iteration are clearly displayed through heatmaps, bar charts, line area charts, and the frame-by-frame video display module.

[0065] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A video object segmentation method based on autonomous key frame selection and edge refinement, comprising the following steps: (1) Obtain a video dataset, and extract visual features and semantic features for each frame in the video dataset; (2) Autonomously recommend key frames; compare the performance of the recommendation strategy and the random selection strategy by obtaining the mean and variance of the segmentation results, and design a reward function to let the autonomous key frame selection module learn how to recommend key frames, the analysis process is as follows: (2-1) Label the key frame and give the initial segmentation result; the user provides a video to be segmented, the user first selects a target to be segmented, and selects a frame image containing the target for labeling; next, a general segmentation algorithm is used to perform video object segmentation on the video provided by the user and obtain the segmentation result; (2-2) Learn to recommend frames; A bidirectional long short-term memory network is designed for self-learning, the input of which is state and the output is continuous action, and the agent is to recommend key frame index; the quality of segmentation result is selected as the score of each frame; in addition, the number of historical recommendations of each frame is also considered; therefore, the state The statistical index is defined as formula (1): (1) as a proxy for a mask, denotes a history recommendation, where the subscript is the maximum number of iterations; denotes a concatenation operation; (2-3) Set up a reward function; to achieve the optimal result, a target reward is designed according to the performance of the time sequence of the target action to be segmented until a defined number of iterations is reached; in interactive VOS, it is difficult to calculate the value of each frame for the whole video sequence, so the final performance reached according to the action sequence A target reward is designed until the maximum number of iterations is reached ; The learned strategy has higher average performance than random selection; To get the performance of the random selection strategy , first, use the random selection strategy for each training video sequence times experiment; by comparing the performance between the recommendation strategy and the random selection strategy, the corresponding reward function can be designed, so the following reward function formula (2) is proposed: (2) wherein represents a reward metric after a maximum number of iterations, represents a performance, represents an expected average value, represents a variance; (2-4) Optimize the reward function; when performance Greater than random selection strategy when average performance When the reward in formula (2) is positive, the reward will be negative; otherwise, the reward will be negative. In practice based on real-world scenarios, it has been found that simply outperforming the random selection strategy on average is insufficient; performance alone is not enough. Greater than + When the reward is set to positive, the final reward function is formula (3): (3) (2-5) Segmentation quality evaluation; in order to evaluate the segmentation quality, a quality evaluation module is used to evaluate the segmentation quality of each frame; first, the bounding box of the segmentation target is calculated according to the segmentation probability map; then, the bounding box is enlarged to times; in order to ignore the background area, the probability map corresponding to the RGB image is cropped according to the enlarged bounding box; then, the cropped RGB image and the probability map are concatenated as the input of the segmentation quality evaluation module, and the segmentation quality estimation of each object of interest is obtained; (3) Optimize the edge of the key frame; after the recommendation of the key frame is completed through the above steps, the mask of the key frame is optimized; there are two key problems that lead to low-quality boundary segmentation, first, the low spatial resolution of the output, and second, the pixels around the object boundary only account for a small part of the entire image, which is difficult to classify; the two problems are improved by the refinement network and the extraction of edge image blocks, the analysis process is as follows: (3-1) Extract edge image blocks; given an instance mask of a key frame selected by the autonomous key frame selection module, then refine the edge of the mask; use a sliding window method to extract a series of image blocks at the edge, a set of square bounding boxes are densely distributed at the mask boundary, the center area of these bounding boxes should cover the boundary pixels; because the obtained boxes still contain overlaps and redundancies, filtering is performed; next, the extracted edge image blocks and the mask blocks are all resized to the same size and put into the refinement network together; (3-2) Refinement network; a series of small boundary image blocks are extracted and refined along the predicted target mask boundary; refinement is done through a higher resolution boundary image block refinement network; the refinement network performs semantic segmentation on each extracted boundary image block; the refinement network fuses features of various levels to improve high-resolution performance; by increasing the input size, the boundary image block can get higher resolution than the previous method; the edge refinement module first extracts a series of small image blocks along the predicted target boundary; after fusion with the mask, the boundary image blocks are put into the refinement network, which performs semantic segmentation to refine the rough boundary; then, the optimized mask is reassembled into a higher quality target mask; (3-3) Reassemble the target mask; after refining each image block, they need to be reassembled onto the target mask; adjacent image blocks may overlap, and their average value is taken as the final result to determine whether a pixel belongs to the target to be segmented or the background. (4) iteratively optimize the predicted mask, and show the results of each iteration in the visualization analysis system; customize the number of iterations, and perform autonomous frame selection once per iteration, then put the keyframes recommended by the system into the edge refinement module; after completing an iteration, input the feature value of each iteration, the score of each frame, and the average score of each iteration into the visual analysis system, and clearly display the results of each iteration through the heat map, column chart, line area chart, and frame-by-frame video display module.

2. The method of claim 1, wherein: In step (2-3) is set to between 10 and 20.

3. The method of claim 1, wherein: In step (2-5) is set to between 2 and 5.

4. The method of claim 1, wherein: The Yes-Net algorithm is used for filtering in step (3-1).

5. The method of claim 1, wherein: The refinement network in step (3-2) adopts a multi-HRNet network.

6. A system for video object segmentation based on autonomous key frame selection and edge refinement, for implementing the method of any one of claims 1-5, characterized in that: It includes: A video dataset acquisition module is used to acquire a video dataset and extract visual features and semantic features for each frame in the video dataset. An autonomous keyframe recommendation module is used to autonomously recommend keyframes; the performance of the recommendation strategy and the random selection strategy is compared by obtaining the mean and variance of the segmentation result, and a reward function is designed to enable the autonomous frame selection module to learn how to recommend keyframes. A keyframe optimization module is used to optimize the edges of keyframes; the mask of the keyframe is optimized; there are two key problems that lead to low-quality boundary segmentation, first, the low spatial resolution of the output, and second, the pixels around the object boundary only account for a small part of the entire image, which is difficult to classify; the two problems are improved through the refinement network and the extraction of edge image blocks. A mask iteration optimization module is used to iteratively optimize the predicted mask and display the results of each iteration in the visualization analysis system; the number of iterations is customized, autonomous frame selection is performed once per iteration, and then the keyframes recommended by the system are put into the edge refinement module; after completing an iteration, the feature value of each iteration, the score of each frame, and the average score of each iteration are input into the visual analysis system, and the results of each iteration are clearly displayed through the heat map, column chart, line area chart, and frame-by-frame video display module.

Images