Video object segmentation method, device, equipment and program product

Abstract

The application discloses a video object segmentation method, device, equipment and program product, and relates to the technical field of computer vision. The method comprises the following steps: adopting an interactive understanding network, identifying a target object based on an interactive frame image, a reference segmentation mask and a user-labeled picture to obtain an initial segmentation mask; extracting an initial boundary mask block along a predicted object boundary of the target object in the initial segmentation mask, and extracting an initial boundary image block corresponding to the initial boundary mask block in the interactive frame image; adopting a boundary optimization network, determining an interactive frame boundary mask block according to the initial boundary mask block and the initial boundary image block; adjusting the corresponding initial boundary mask block in the initial segmentation mask according to the interactive frame boundary mask block to obtain an interactive frame segmentation mask; and adopting a time sequence propagation network, determining a current frame segmentation mask according to a current frame image, a historical frame image and a historical frame segmentation mask. The technical scheme of the embodiment of the application can improve the segmentation quality of the video object.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video object segmentation method, apparatus, device, and program product. Background Technology

[0002] Video object segmentation is an important research area that has received widespread attention in the field of computer vision in recent years. Its research goal is to segment the target object of interest from video frames, and it has been widely applied in the fields of autonomous driving and video editing.

[0003] Common video object segmentation methods can be categorized into three types: unsupervised video object segmentation, semi-supervised video object segmentation, and interactive video object segmentation. Semi-supervised video object segmentation requires the segmentation of the real objects in the first frame of the video; unsupervised video object segmentation does not require information about the objects to be segmented; interactive video object segmentation only requires the user to add simple annotations to the target object in a single frame of the video (e.g., drawing simple lines, clicking, or using a bounding box), and then the algorithm obtains the segmentation result of the target object in the entire video. Interactive video object segmentation can be viewed as a combination of two tasks: interactive understanding and time propagation. The task of interactive understanding is to generate accurate annotations using limited user interaction information, while the task of time propagation is to generate annotations for the entire video using the annotation information. Among the three types of video object segmentation methods mentioned above, interactive video object segmentation is superior. However, interactive video object segmentation methods struggle to guarantee the segmentation quality of the target object. Summary of the Invention

[0004] This invention provides a video object segmentation method, apparatus, device, and program product, which improves the boundary clarity of video object segmentation, thereby improving the segmentation quality of video objects.

[0005] According to one aspect of the present invention, a video object segmentation method is provided, the method comprising:

[0006] The interaction frame image, reference segmentation mask, and user-annotated image of the interaction frame image are acquired. An interaction understanding network is used to identify the target object based on the interaction frame image, reference segmentation mask, and user-annotated image to obtain an initial segmentation mask.

[0007] Along the predicted object boundary of the target object in the initial segmentation mask, an initial boundary mask block is extracted, and in the interactive frame image, the initial boundary image block corresponding to the initial boundary mask block is extracted;

[0008] A boundary optimization network is used to optimize the initial boundary mask block based on the initial boundary mask block and the initial boundary image block to obtain the interactive frame boundary mask block;

[0009] Based on the interactive frame boundary mask block, the corresponding initial boundary mask block in the initial segmentation mask is adjusted to obtain the interactive frame segmentation mask;

[0010] Obtain the current frame image, and determine the historical frame image and historical frame segmentation mask based on the current frame image, the interactive frame image, and the interactive frame segmentation mask;

[0011] A time-series propagation network is used to determine the current frame segmentation mask based on the current frame image, the historical frame images, and the historical frame segmentation mask.

[0012] According to another aspect of the present invention, a video object segmentation apparatus is provided, the apparatus comprising:

[0013] The interaction frame image, reference segmentation mask, and user-annotated image of the interaction frame image are acquired. An interaction understanding network is used to identify the target object based on the interaction frame image, reference segmentation mask, and user-annotated image to obtain an initial segmentation mask.

[0014] Along the predicted object boundary of the target object in the initial segmentation mask, an initial boundary mask block is extracted, and in the interactive frame image, the initial boundary image block corresponding to the initial boundary mask block is extracted;

[0015] A boundary optimization network is used to optimize the initial boundary mask block based on the initial boundary mask block and the initial boundary image block to obtain the interactive frame boundary mask block;

[0016] Based on the interactive frame boundary mask block, the corresponding initial boundary mask block in the initial segmentation mask is adjusted to obtain the interactive frame segmentation mask;

[0017] Obtain the current frame image, and determine the historical frame image and historical frame segmentation mask based on the current frame image, the interactive frame image, and the interactive frame segmentation mask;

[0018] A time-series propagation network is used to determine the current frame segmentation mask based on the current frame image, the historical frame images, and the historical frame segmentation mask.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] At least one processor; and

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video object segmentation method according to any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the video object segmentation method according to any embodiment of the present invention.

[0024] The technical solution of this invention involves acquiring an interactive frame image, a reference segmentation mask, and user-annotated images of the interactive frame image. An interactive understanding network is used to identify target objects based on these elements, obtaining an initial segmentation mask. An initial boundary mask block is extracted along the predicted object boundary of the target object in the initial segmentation mask. Additionally, an initial boundary image block corresponding to the initial boundary mask block is extracted from the interactive frame image. A boundary optimization network is used to optimize the initial boundary mask block based on the initial boundary mask block and the initial boundary image block, resulting in an interactive frame boundary mask block. Finally, based on the interactive frame boundary mask block, the initial segmentation mask is optimized to identify the target object in the interactive frame image. The initial boundary mask block is adjusted to obtain the interactive frame segmentation mask, which realizes the boundary adjustment of the initial segmentation mask corresponding to the interactive frame image and improves the boundary clarity of the interactive frame segmentation mask. By acquiring the current frame image and determining the historical frame image and historical frame segmentation mask based on the current frame image, interactive frame image, and interactive frame segmentation mask, a time-series propagation network is used to determine the current frame segmentation mask based on the current frame image, historical frame image, and historical frame segmentation mask. Through time-series propagation of the interactive frame segmentation mask, the current frame segmentation mask of the current frame image is obtained, which improves the boundary clarity of the current frame segmentation mask and thus improves the segmentation quality of video objects.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a video object segmentation method provided in Embodiment 1 of the present invention;

[0028] Figure 2 This is an architecture diagram of an interactive understanding network provided according to Embodiment 1 of the present invention;

[0029] Figure 3 This is a flowchart of a video object segmentation method provided in Embodiment 2 of the present invention;

[0030] Figure 4 This is an overall architecture diagram of a video object segmentation method provided in Embodiment 3 of the present invention;

[0031] Figure 5 This is an architecture diagram of a boundary optimization network provided according to Embodiment 3 of the present invention;

[0032] Figure 6 This is an architecture diagram of a time-series propagation network provided according to Embodiment 3 of the present invention;

[0033] Figure 7 This is a schematic diagram of a fusion spatial prior module provided according to Embodiment 3 of the present invention;

[0034] Figure 8 This is a segmentation convergence graph of an interactive understanding network provided in Embodiment 3 of the present invention;

[0035] Figure 9 This is a schematic diagram of the structure of a video object segmentation device according to Embodiment 4 of the present invention;

[0036] Figure 10 A schematic diagram of the structure of an electronic device that implements the video object segmentation method of this invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] Example 1

[0040] Figure 1 This is a flowchart illustrating a video object segmentation method according to Embodiment 1 of the present invention. This embodiment of the invention is applicable to situations where an interactive video object segmentation method is used to segment video objects. This method can be executed by a video object segmentation device, which can be implemented in hardware and / or software, and can be configured in an electronic device that carries video object segmentation functionality.

[0041] See Figure 1 The video object segmentation method shown includes:

[0042] S101. Obtain the interactive frame image, the reference segmentation mask, and the user-annotated image of the interactive frame image. Then, use the interactive understanding network to identify the target object based on the interactive frame image, the reference segmentation mask, and the user-annotated image to obtain the initial segmentation mask.

[0043] The interactive frame image can be a frame selected and labeled by the user in the video. Essentially, the interactive frame image can be an image used for segmentation mask optimization during the interactive understanding process. For example, the interactive frame image can be an RGB (Red-Green-Blue, the three primary colors) image. The reference segmentation mask can be a segmentation mask referenced by the interactive understanding network. Optionally, the reference segmentation mask can include a mask obtained after the previous round of interactive understanding or a high-quality mask of a frame selected by the user. For example, the reference segmentation mask can be a probability map between 0 and 1. The user-annotated image can be an image of the target object labeled by the user based on the interactive frame image. The user-annotated image is provided by the user. For example, the user-annotated image can include positive doodles and negative doodles. Positive doodles can represent positive labels on the target object, i.e., labels on the target object that the user wants to segment. Negative doodles can represent negative labels on the target object, i.e., labels on parts that the user does not want to segment.

[0044] An interactive understanding network can be used to identify target objects based on an interactive frame image, a reference segmentation mask, and a user-annotated image, thereby obtaining an initial segmentation mask corresponding to the interactive frame image. Optionally, the structure of the interactive understanding network may include an input layer, an encoder, and a decoder. For example, Figure 2 A network framework diagram for interactive understanding of the network. For example... Figure 2 As shown, the input to the Interactive Understanding Network includes the interactive frame image, a reference segmentation mask, a positive doodle image, and a negative doodle image. The input layer consists of a first connection layer and two convolutions. The reshaping module connects the feature maps from all six channels together, i.e., the interactive frame image, the reference segmentation mask, the positive doodle image, and the negative doodle image. The two convolutions are used to adjust the feature maps from all six channels. The encoder can be a pre-trained ResNet (Residual Network) backbone network. The decoder can include ASPP, skip connection layers, a second connection layer, 1×1 convolutions, and upsampling layers. After decoding by the decoder of the Interactive Understanding Network, an initial segmentation mask is obtained. The initial segmentation mask represents the initial segmentation result of the target object generated by the Interactive Understanding Network. Based on the initial segmentation mask, it can be determined which part of the mask's boundary needs to be optimized. A boundary optimization model (i.e., a boundary optimization network) can be connected to the output of the Interactive Understanding Network to optimize the initial boundary mask block, resulting in the interactive frame boundary mask block.

[0045] Specifically, the process involves acquiring the user-selected interaction frame image, the reference segmentation mask needed for the interaction understanding process, and the user-annotated image of the interaction frame image. The interaction frame image, reference segmentation mask, and user-annotated image are then input into the interaction understanding network, which identifies the target object to obtain the initial segmentation mask.

[0046] S102. Extract the initial boundary mask block along the predicted object boundary of the target object in the initial segmentation mask, and extract the initial boundary image block corresponding to the initial boundary mask block in the interactive frame image.

[0047] The target object can be the object the user wants to segment. For example, the target object could be a vehicle, pedestrian, or obstacle. The predicted object boundary can be the boundary of the target object in the initial segmentation mask. The initial boundary mask block can be a boundary mask block in the initial segmentation mask that contains the predicted object boundary of the target object. The initial boundary image block can be the boundary image block corresponding to the initial boundary mask block in the interactive frame image.

[0048] Specifically, using a preset sliding window, initial boundary mask blocks are extracted along the predicted object boundary of the target object in the initial segmentation mask. The central region of each initial boundary mask block should cover the pixels of the predicted object boundary. Alternatively, a preset sliding window can be used to extract the initial boundary image blocks corresponding to each initial boundary mask block from the interactive frame image.

[0049] Optionally, the initial boundary mask block and the initial boundary image block can have different sizes. In this case, a uniform scaling operation can be performed on the initial boundary mask block and the initial boundary image block to make them the same size.

[0050] In an optional embodiment of the present invention, after extracting the initial boundary mask blocks along the predicted object boundary of the target object in the initial segmentation mask, the method includes: filtering each initial boundary mask block using the non-maximum suppression (NMS) method and updating each initial boundary mask block.

[0051] Because the extracted initial boundary mask blocks still contain a large amount of overlap and redundancy along the predicted object boundary in the initial segmentation mask, non-maximum suppression (NMS) can be used to filter the initial boundary mask blocks, thus eliminating redundant initial boundary mask blocks. Correspondingly, by using NMS to filter each initial boundary mask block and updating them, the corresponding initial boundary image blocks can be extracted from the interactive frame image, which can also avoid redundancy in the initial boundary image blocks.

[0052] This approach can reduce the amount of data that the boundary optimization network needs to process, thereby improving the optimization efficiency of the initial boundary mask block.

[0053] S103. Using a boundary optimization network, the initial boundary mask block is optimized based on the initial boundary mask block and the initial boundary image block to obtain the interactive frame boundary mask block.

[0054] Boundary optimization networks can be used to optimize the boundaries of an initial boundary mask patch. Essentially, the boundary optimization network performs binary segmentation on the initial boundary mask patch alone. The initial boundary image patch can provide positional guidance for the target object to be segmented, thereby accelerating the training and convergence of the boundary optimization network. The interactive frame boundary mask patch can be the result of the boundary optimization network optimizing the initial boundary mask patch.

[0055] Specifically, the initial boundary mask block and the initial boundary image block are input into the boundary optimization network to optimize the initial boundary mask block and obtain the interactive frame boundary mask block.

[0056] In an optional embodiment of the present invention, adjusting the corresponding initial boundary mask block in the initial segmentation mask according to the interaction frame boundary mask block to obtain the interaction frame segmentation mask includes: for a single initial boundary mask block, comparing the interaction frame pixel value of the interaction frame boundary mask block with the initial pixel value of the corresponding initial boundary mask block; for a single initial boundary mask block, if the interaction frame pixel value is consistent with the corresponding initial pixel value, not adjusting the initial boundary mask block; for a single initial boundary mask block, if the interaction frame pixel value is inconsistent with the corresponding initial pixel value, adjusting the initial boundary mask block according to the interaction frame boundary mask block; and when all initial boundary masks in the initial segmentation mask have been adjusted, generating the interaction frame segmentation mask block according to the adjustment results of each initial boundary mask block.

[0057] Interactive frame pixel values ​​can be used to characterize the overall pixels of the interactive frame boundary mask block. For example, interactive frame pixel values ​​may include the average pixel value or the sum of the pixel values ​​of the interactive frame boundary mask block. Initial pixel values ​​can be used to characterize the overall pixels of the initial boundary mask block. For example, initial pixel values ​​may include the average pixel value or the sum of the pixel values ​​of the initial boundary mask block. The interactive frame pixel values ​​and initial pixel values ​​are calculated in the same way. For example, the interactive frame pixel value is the average pixel value of the interactive frame boundary mask block; correspondingly, the initial pixel value is the average pixel value of the initial boundary mask block. The consistency between the interactive frame pixel value and the corresponding initial pixel value can be understood as the partial predicted object boundary of the target object predicted by the initial boundary mask block being consistent with the partial predicted object boundary of the target object contained in the interactive frame boundary mask block; that is, the partial predicted object boundary of the target object predicted by the initial boundary mask block has sufficient boundary sharpness and boundary accuracy. The pixel values ​​of the interactive frame are inconsistent with the corresponding initial pixel values. The partial predicted object boundary of the target object predicted by the initial boundary mask block is inconsistent with the partial predicted object boundary of the target object contained in the interactive frame boundary mask block. In other words, the boundary optimization network optimizes the partial predicted object boundary of the target object predicted by the initial boundary mask block.

[0058] Specifically, for a single initial boundary mask block, the interactive frame pixel values ​​of the interactive frame boundary mask block are compared with the initial pixel values ​​of the corresponding initial boundary mask block. If the interactive frame pixel values ​​match the corresponding initial pixel values, the initial boundary mask block is not adjusted. If the interactive frame pixel values ​​do not match the corresponding initial pixel values, the initial boundary mask block is replaced by the interactive frame boundary mask block. Once all initial boundary masks in the initial segmentation mask have been adjusted, the interactive frame segmentation mask block is generated based on the adjustment results of each initial boundary mask block.

[0059] This scheme compares the interactive frame pixel values ​​of the interactive frame boundary mask block with the initial pixel values ​​of the corresponding initial boundary mask block. When the interactive frame pixel values ​​match the corresponding initial pixel values, the initial boundary mask block is not adjusted. When the interactive frame pixel values ​​do not match the corresponding initial pixel values, the initial boundary mask block is adjusted based on the interactive frame boundary mask block. This avoids the indiscriminate adjustment of the initial boundary mask block using the interactive frame boundary mask block, and introduces a comparison process between the interactive frame boundary mask block and the corresponding initial boundary mask block. When the interactive frame pixel values ​​do not match the corresponding initial pixel values, the initial boundary mask block is adjusted specifically using the interactive frame boundary mask block, improving the effectiveness of the initial boundary mask block adjustment. At the same time, by comparing the interactive frame pixel values ​​with the corresponding initial pixel values, the comparison efficiency between the interactive frame boundary mask block and the initial boundary mask block is improved, thereby increasing the generation efficiency of the interactive frame segmentation mask.

[0060] In an optional embodiment of the present invention, after generating an interactive frame segmentation mask block based on the adjustment results of each initial boundary mask block for a single initial boundary mask block, the method further includes: performing overlap detection on the pixels of each interactive frame boundary mask block; when overlapping pixels exist in different interactive frame boundary mask blocks, calculating the average value of overlapping pixels in each interactive frame boundary overlapping mask block; comparing the average value of overlapping pixels in each interactive frame boundary overlapping mask block with a preset pixel value threshold; when the average value of overlapping pixels in each interactive frame overlap block is greater than the preset pixel value threshold, determining the overlapping pixels as the foreground of the target object; and when the average value of overlapping pixels in each interactive frame overlap block is less than or equal to the preset pixel value threshold, determining the overlapping pixels as the background of the target object.

[0061] Overlapping pixels exist between boundary masks of different interaction frames. This means that for the same overlapping pixel, the prediction results for the target object may differ between overlapping boundary masks of different interaction frames. If the initial boundary mask is directly replaced by an interaction frame boundary mask, at least two prediction results may appear at the overlapping pixels, affecting the accuracy of the predicted object boundary at the overlapping pixels. Overlapping pixels can be pixels that overlap between different interaction frame boundary masks. Overlapping pixels can belong to at least two interaction frame boundary masks. Interaction frame boundary overlapping masks can be interaction frame boundary masks with overlapping pixels. The average value of overlapping pixels in interaction frames can be used to characterize the average pixel value of each overlapping boundary mask in interaction frames at a single overlapping pixel. The preset pixel value threshold can be a pre-defined lower limit of the average value of overlapping pixels in interaction frames when the pixel is the foreground of the target object. The preset comprehensive pixel value threshold can be set and adjusted by technicians based on experience. For example, the preset pixel value threshold can be 0.5.

[0062] This scheme introduces overlap detection for each interactive frame boundary mask block in the interactive frame segmentation mask, enabling further adjustment of the boundary overlap region of the interactive frame segmentation mask and improving the accuracy of the boundary overlap region in the interactive frame segmentation mask. Simultaneously, when interactive frame boundary mask blocks overlap, a comprehensive pixel value for the interactive frame overlap region is calculated based on the interactive frame overlap pixel values ​​of the overlapping mask blocks in the boundary overlap region. This comprehensive pixel value is compared with a preset comprehensive pixel value threshold. If the comprehensive pixel value is greater than or equal to the preset comprehensive pixel value threshold, the boundary overlap region is determined to be the foreground region of the target object; if it is less than the preset comprehensive pixel value threshold, the boundary overlap region is determined to be the background region of the target object. This improves the adjustment efficiency of the boundary overlap region in the interactive frame segmentation mask.

[0063] S104. Based on the interactive frame boundary mask block, adjust the corresponding initial boundary mask block in the initial segmentation mask to obtain the interactive frame segmentation mask.

[0064] Specifically, interactive frame boundary mask blocks can be used to replace the corresponding initial boundary mask blocks in the initial segmentation mask, update the initial segmentation mask, and obtain the interactive frame segmentation mask.

[0065] S105. Obtain the current frame image, and determine the historical frame image and historical frame segmentation mask based on the current frame image, the interactive frame image, and the interactive frame segmentation mask.

[0066] The current frame image can be a frame image of the segmentation mask predicted by the temporal propagation network. Optionally, the interactive frame image can be the previous frame image or several frames above the current frame image in the video. That is, the interactive frame image can be a historical frame image of the current frame image. The historical frame image can be an image of a frame preceding the current frame image in the video. The historical frame segmentation mask can be the segmentation mask corresponding to the historical frame image.

[0067] Optionally, during the initial time-series propagation of the network, the next frame of the interactive frame image can be acquired to obtain the current frame image. The interactive frame image can be used as a historical frame image, and the interactive frame segmentation mask can be used as a historical frame segmentation mask.

[0068] Optionally, after the temporal propagation network has performed multiple rounds of temporal propagation—that is, after the corresponding current frame segmentation mask has been predicted according to the temporal order of the frames in the video—the frame image corresponding to the earliest moment before temporal propagation is determined as the current frame image. Interactive frame images and each current frame image after temporal propagation can be determined as historical frame images. The interactive frame segmentation mask is determined as the historical frame segmentation mask corresponding to the interactive frame image; the current frame segmentation mask predicted through temporal propagation is determined as the historical frame segmentation mask corresponding to the historical frame image.

[0069] S106. Using a time-series propagation network, determine the segmentation mask for the current frame based on the current frame image, historical frame images, and historical frame segmentation masks.

[0070] The temporal propagation network can identify the current frame segmentation mask based on the current frame image, historical frame images, and historical frame segmentation masks. The number of historical frame images can be at least one. The historical frame segmentation mask can be used to provide boundary optimization for the temporal propagation of the current frame image, thereby improving the boundary sharpness of the current frame segmentation mask. The current frame segmentation mask can be the segmentation mask identification result of the temporal propagation network for the current frame image.

[0071] Specifically, the current frame image, historical frame images, and historical frame segmentation masks are input into the temporal propagation network to predict the segmentation mask corresponding to the current frame image, thus obtaining the current frame segmentation mask.

[0072] The technical solution of this invention involves acquiring an interactive frame image, a reference segmentation mask, and user-annotated images of the interactive frame image. An interactive understanding network is used to identify target objects based on these elements, obtaining an initial segmentation mask. An initial boundary mask block is extracted along the predicted object boundary of the target object in the initial segmentation mask. Additionally, an initial boundary image block corresponding to the initial boundary mask block is extracted from the interactive frame image. A boundary optimization network is used to optimize the initial boundary mask block based on the initial boundary mask block and the initial boundary image block, resulting in an interactive frame boundary mask block. Finally, based on the interactive frame boundary mask block, the initial segmentation mask is optimized to identify the target object in the interactive frame image. The initial boundary mask block is adjusted to obtain the interactive frame segmentation mask, which realizes the boundary adjustment of the initial segmentation mask corresponding to the interactive frame image and improves the boundary clarity of the interactive frame segmentation mask. By acquiring the current frame image and determining the historical frame image and historical frame segmentation mask based on the current frame image, interactive frame image, and interactive frame segmentation mask, a time-series propagation network is used to determine the current frame segmentation mask based on the current frame image, historical frame image, and historical frame segmentation mask. Through time-series propagation of the interactive frame segmentation mask, the current frame segmentation mask of the current frame image is obtained, which improves the boundary clarity of the current frame segmentation mask and thus improves the segmentation quality of video objects.

[0073] Example 2

[0074] Figure 3 This is a flowchart of a video object segmentation method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment of the present invention specifies "using a time-series propagation network to determine the current frame segmentation mask based on the current frame image, historical frame images, and historical frame segmentation masks" as follows: "Using a memory encoder of the time-series propagation network to encode the historical frame images and corresponding historical frame segmentation masks to obtain historical frame key-value pairs; using a query encoder of the time-series propagation network to encode the current frame image to obtain current frame key-value pairs; using a fusion spatial prior module of the time-series propagation network to generate a comprehensive feature map of the current frame based on the historical frame key-value pairs and the current frame key-value pairs; using a fusion spatial prior module of the time-series propagation network to perform convolution and nonlinear activation operations on the comprehensive feature map of the current frame to obtain a spatial prior probability map; using a fusion spatial prior module of the time-series propagation network to perform convolution and nonlinear activation operations on the comprehensive feature map of the current frame to obtain a spatial prior probability map; using a fusion spatial prior module of the time-series propagation network to perform convolution and nonlinear activation operations on the comprehensive feature map of the current frame to obtain a spatial prior probability map." The feature map and the spatial prior probability map are multiplied to obtain the spatially weighted feature map of the current frame. A decoder using a temporal propagation network decodes the spatially weighted feature map of the current frame to obtain the segmentation mask of the current frame. This mask integrates information from the current frame image, historical frame images, and historical frame segmentation masks, taking into account both the appearance details of the current frame image and the recognition results of target objects in historical frame images. This improves the accuracy of target object recognition. Compared to using a fixed Gaussian function to generate heatmaps, the introduction of a spatial prior probability map can more accurately identify the location of target objects, avoiding the identification of pixels in the current frame image that are similar to but unrelated to the target object. This reduces the false positive rate of the current frame segmentation mask recognition, strengthens the spatial information of the target object, and further improves the accuracy of video object recognition. It should be noted that parts not described in detail in this embodiment can be found in the descriptions of other embodiments.

[0075] See Figure 3 The video object segmentation method shown includes:

[0076] S301. Acquire the interactive frame image, the reference segmentation mask, and the user-annotated image of the interactive frame image. Then, use the interactive understanding network to identify the target object based on the interactive frame image, the reference segmentation mask, and the user-annotated image to obtain the initial segmentation mask.

[0077] S302. Extract the initial boundary mask block along the predicted object boundary of the target object in the initial segmentation mask, and extract the initial boundary image block corresponding to the initial boundary mask block in the interactive frame image.

[0078] S303. Using a boundary optimization network, the initial boundary mask block is optimized based on the initial boundary mask block and the initial boundary image block to obtain the interactive frame boundary mask block.

[0079] S304. Based on the interactive frame boundary mask block, adjust the corresponding initial boundary mask block in the initial segmentation mask to obtain the interactive frame segmentation mask.

[0080] S305. Obtain the current frame image, and determine the historical frame image and historical frame segmentation mask based on the current frame image, the interactive frame image, and the interactive frame segmentation mask.

[0081] S306. A memory encoder using a time-propagation network is used to encode the historical frame image and the corresponding historical frame segmentation mask to obtain historical frame key-value pairs.

[0082] A memory encoder can be used to encode a single historical frame image and its corresponding historical frame segmentation mask to obtain historical frame key-value pairs. These pairs can be the encoded output of the historical frame image and its corresponding segmentation mask. The historical frame key-value pairs store not only detailed information about the historical frame image but also semantic information used to distinguish target objects (i.e., which regions are foreground and which are background regions of the target object). A historical frame key-value pair can include a historical frame key and a historical frame value.

[0083] Specifically, historical frame images and corresponding historical frame segmentation masks can be input into the memory encoder of the temporal propagation network to obtain historical frame key-value pairs.

[0084] Optionally, the historical frame image can consist of multiple frames. A memory encoder using a time-propagation network can be used to encode each individual historical frame image and its corresponding historical frame segmentation mask, resulting in a single set of historical frame key-value pairs.

[0085] S307. A query encoder using a time-series propagation network is used to encode the current frame image to obtain the current frame key-value pairs.

[0086] The query encoder can be used to encode the current frame image, obtaining current frame key-value pairs. The current frame key-value pairs can be the encoded output form of the current frame image. They store detailed information about the current frame image and are the basis for generating the current frame segmentation mask. Each current frame key-value pair can include a current frame key and a current frame value.

[0087] Specifically, the current frame image can be input into the query encoder of the time-series propagation network to obtain the current frame key-value pairs.

[0088] S308. A fusion spatial prior module using a time-series propagation network is used to generate a comprehensive feature map of the current frame based on historical frame key-value pairs and current frame key-value pairs.

[0089] The fusion spatial prior module can be used to fuse key-value pairs from historical frames and key-value pairs from the current frame to obtain a comprehensive feature map of the current frame. This comprehensive feature map can be a feature map that incorporates information from both the current and historical frames.

[0090] Specifically, the fusion spatial prior module of the temporal propagation network is used to directly sum the key-value pairs of the historical frames and the key-value pairs of the current frame according to the pixel position to obtain the comprehensive feature map of the current frame.

[0091] S309. Using a temporal propagation network, a fusion spatial prior module is used to perform convolution and nonlinear activation operations on the current frame's comprehensive feature map to obtain a spatial prior probability map.

[0092] Spatial prior probability maps can be used to initially identify the location of target objects in the current frame image. They provide a coarse spatial prior for the current frame segmentation mask. Based on the spatial prior probability map, the location of target objects can be identified more accurately, avoiding the identification of pixels in the current frame image that are similar to but unrelated to the target object, thus reducing the false positive rate of the current frame segmentation mask.

[0093] Specifically, the spatial prior fusion module of the temporal propagation network is used to perform convolution and nonlinear activation operations on the comprehensive feature map of the current frame to obtain the spatial prior probability map.

[0094] S310. Using the fusion spatial prior module of the temporal propagation network, the current frame comprehensive feature map and the spatial prior probability map are multiplied to obtain the current frame spatial weighted feature map.

[0095] The spatially weighted feature map of the current frame is the result of multiplying the comprehensive feature map of the current frame with the spatial prior probability map. The spatially weighted feature map of the current frame can be used to more prominently represent the position of the target object in the current frame image.

[0096] Specifically, by using the fusion spatial prior module of the temporal propagation network, the comprehensive feature map of the current frame and the spatial prior probability map can be multiplied to obtain the spatial weighted feature map of the current frame.

[0097] S311. A decoder using a time-propagation network decodes the spatially weighted feature map of the current frame to obtain the segmentation mask of the current frame.

[0098] The decoder can be used to encode the spatially weighted feature map of the current frame to obtain the segmentation mask of the current frame.

[0099] Specifically, a decoder using a temporal propagation network can be used to decode the spatially weighted feature map of the current frame to obtain the segmentation mask for the current frame.

[0100] In an optional embodiment of the present invention, a fusion spatial prior module of a time-series propagation network is used to generate a comprehensive feature map of the current frame based on historical frame key-value pairs and current frame key-value pairs. This includes: using the fusion spatial prior module of a time-series propagation network to determine the historical frame weights of historical frame values ​​based on historical frame keys and current frame keys; using the fusion spatial prior module of a time-series propagation network to synthesize each historical frame value based on the historical frame weights to obtain a comprehensive feature map of historical frames; and using the fusion spatial prior module of a time-series propagation network to connect the comprehensive feature map of historical frames and the current frame value to obtain a comprehensive feature map of the current frame.

[0101] The history frame key can be used to characterize high-level visual features and semantic information at a specific location in a history frame. The current frame key can be used to characterize high-level visual features and semantic information at a specific location in the current frame. Both the history frame key and the current frame key can be used for addressing. Both contain rich semantic information. They can be used to evaluate the similarity between the history frame and the current frame. This allows us to determine which information in the current frame should be sampled to ensure robustness to changes in the target object in the current frame. Comparatively, the history frame value and the current frame value store more refined information than the history frame key and the current frame key. They can be used to generate a more refined segmentation mask for the current frame. The history frame key and the history frame value are the encoding results of the memory encoder on the history frame image and the corresponding history frame segmentation mask. The current frame key and the current frame value are the encoding results of the query encoder on the current frame image. The history frame weight can be used to characterize the similarity between the history frame key and the current frame key. Historical frame weights can be used to characterize the correlation between a pixel at a certain position in a historical frame and a pixel at a certain position in the current frame. Historical frame weights can be used to perform a weighted summation of historical frame values. The comprehensive historical frame feature map is a feature map obtained by weighting and summing the corresponding historical frame values ​​using the historical frame weights.

[0102] Specifically, the fusion spatial prior module of the temporal propagation network is used to calculate the similarity between historical frame keys and current frame keys, obtaining the historical frame weights of historical frame values. The fusion spatial prior module of the temporal propagation network then uses these historical frame weights as the weights of the corresponding historical frame values, performing a weighted summation of all historical frame values ​​to obtain the comprehensive feature map of historical frames. Finally, the fusion spatial prior module of the temporal propagation network is used to concatenate the comprehensive feature map of historical frames with the current frame value to obtain the comprehensive feature map of the current frame.

[0103] This scheme employs a fusion spatial prior module of a temporal propagation network to determine the historical frame weights of historical frame values ​​based on historical frame keys and the current frame key. Then, using the same fusion spatial prior module, it synthesizes each historical frame value based on these weights to obtain a comprehensive historical frame feature map. Finally, the fusion spatial prior module connects the comprehensive historical frame feature map with the current frame value to obtain the comprehensive feature map of the current frame. By introducing historical frame weights and synthesizing each historical frame value, this scheme highlights historical frame values ​​that are relevant to the current frame, improving the correlation between the comprehensive historical frame feature map and the current frame. This, in turn, enhances the targeting and accuracy of target object recognition in the current frame segmentation mask.

[0104] In an optional embodiment of the present invention, before the fusion spatial prior module using a time-propagation network performs convolution and nonlinear activation operations on the current frame comprehensive feature map to obtain a spatial prior probability map, the method further includes: selecting the previous frame segmentation mask corresponding to the previous frame image of the current frame image from each historical frame segmentation mask; concatenating the current frame comprehensive feature map and the previous frame segmentation mask to update the current frame comprehensive feature map.

[0105] The previous frame segmentation mask can be the historical frame segmentation mask corresponding to the previous frame image of the current frame image. Compared with other historical frame segmentation masks, the previous frame segmentation mask has a stronger correlation with the current frame image.

[0106] Specifically, before performing convolution and nonlinear activation operations on the current frame's comprehensive feature map using the temporal propagation network's fusion spatial prior module to obtain the spatial prior probability map, the previous frame's segmentation mask corresponding to the current frame image can be selected from the segmentation masks of each historical frame. The current frame's comprehensive feature map and the previous frame's segmentation mask can then be concatenated to update the current frame's comprehensive feature map.

[0107] This scheme connects the segmentation mask of the previous frame with the comprehensive feature map of the current frame, updates the comprehensive feature map of the current frame, and thus updates the spatial prior probability map. Based on the combination of the current frame and historical frames, it adds a strong spatial prior from the previous frame, making full use of the consistency between frames. It integrates the strong spatial prior information of the segmentation mask of the previous frame (i.e., the updated spatial prior probability map) and appearance matching information (i.e., the comprehensive feature map of the current frame), further effectively overcoming the false positive prediction problem caused by similar objects and improving the accuracy of video object segmentation.

[0108] The technical solution of this invention employs a memory encoder in a time-propagation network to encode historical frame images and corresponding historical frame segmentation masks, obtaining historical frame key-value pairs. A query encoder in the time-propagation network encodes the current frame image, obtaining current frame key-value pairs. A fusion spatial prior module in the time-propagation network generates a comprehensive feature map of the current frame based on the historical and current frame key-value pairs. This fusion integrates information from the current frame image, historical frame images, and historical frame segmentation masks, taking into account both the appearance details of the current frame image and the recognition results of target objects in historical frame images, thus improving the accuracy of target object recognition. Furthermore, the fusion spatial prior module in the time-propagation network performs convolution operations and nonlinear activation on the comprehensive feature map of the current frame. The process involves obtaining a spatial prior probability map. Compared to generating heatmaps using a fixed Gaussian function, this method introduces a spatial prior probability map, which can more accurately identify the location of target objects. It avoids identifying pixels in the current frame that are similar to the target object but unrelated, reducing the false positive rate of the current frame segmentation mask recognition and further improving the accuracy of target object recognition. By employing a spatial prior fusion module in a temporal propagation network, the current frame's comprehensive feature map and spatial prior probability map are multiplied to obtain a spatially weighted feature map, which strengthens the spatial information of the target object. Finally, by using a decoder in a temporal propagation network, the current frame's spatially weighted feature map is decoded to obtain the current frame segmentation mask, thereby further improving the accuracy of video object recognition.

[0109] Example 3

[0110] Figure 4 This is an overall architecture diagram of a video object segmentation method provided in Embodiment 3 of the present invention. Based on the above embodiments, the present invention provides a preferred embodiment. It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments.

[0111] See Figure 4 The video object segmentation method shown includes:

[0112] After acquiring user interaction images, the interaction understanding network generates an interaction frame segmentation mask for the target object corresponding to the interaction frame image. The temporal propagation network, after acquiring the fine-grained mask of the target object (i.e., the interaction frame segmentation mask), propagates it throughout the entire video, generating the current frame segmentation mask for the target object in all frames. This scheme employs a decoupled modular framework, making the training of the interaction understanding network and the temporal propagation network convenient and saving redundant time from calling the temporal propagation network for each interaction. Figure 2This is the architecture diagram of the Interactive Understanding Network. It addresses the issues of blurred interactive backgrounds and boundaries. The input to the Interactive Understanding Network includes interactive frame images (i.e., RGB images), a reference segmentation mask, and user interaction images (i.e., positive and negative doodles). The doodles, provided by the user, represent positive and negative annotations for the target object, respectively. These inputs are concatenated along the channel dimension to form the input tensor. The reference segmentation mask is represented as a probability map between 0 and 1. First, the first connection layer concatenates the feature maps from the six channels mentioned above. Then, the feature maps are adjusted through two 1×1 convolutions. Following this are the encoder and decoder parts. The boundary optimization model (i.e., the boundary optimization network) is located after the decoder, post-processing the initial segmentation mask output by the entire interactive understanding network, optimizing the boundaries of the initial segmentation mask to output the interactive frame segmentation mask.

[0113] in, Figure 5 An architecture diagram for optimizing the network at the boundary. Specifically:

[0114] First, obtain an initial segmentation mask generated by the interactive understanding network.

[0115] Second, a sliding window style algorithm is used to densely allocate a set of square bounding boxes along the predicted object boundary of the target object in the initial segmentation mask. A series of initial boundary mask blocks are extracted, with the constraint that the central region of each square bounding box should cover the boundary pixels. However, since the obtained initial mask boundary blocks still contain a large amount of overlap and redundancy, a non-maximum suppression (NMS) algorithm can be used to filter a subset of the initial mask boundary blocks and update the initial boundary mask blocks. Then, in the interactive frame image, square bounding boxes of the same size are used to extract the initial boundary image blocks corresponding to the initial boundary mask blocks.

[0116] Third, the initial boundary mask block is concatenated with its corresponding initial boundary image block and input into the boundary optimization network, resulting in a clearer boundary mask block for the interactive frame. The advantage of inputting the initial boundary mask block is that it provides positional guidance for the target object to be segmented and accelerates the training and convergence of the boundary optimization network. The boundary optimization network performs binary segmentation on each extracted initial boundary mask block individually, thereby optimizing the initial boundary mask blocks at the predicted object boundaries. The boundary optimization network can solve the problem of coarse initial segmentation mask boundaries predicted by traditional interactive understanding networks.

[0117] Fourth, the initial boundary mask blocks in the initial segmentation mask predicted by the interactive understanding network are replaced by interactive frame boundary mask blocks, and the optimized initial boundary mask blocks are reassembled to generate the interactive frame segmentation mask. The unoptimized initial boundary mask blocks can remain unchanged. Specifically, pixel detection can be performed on the interactive frame pixel values ​​of the interactive frame boundary mask blocks output by the boundary optimization network and the initial pixel values ​​of the corresponding initial boundary mask blocks. For interactive frame boundary mask blocks with unchanged pixels, no optimization is required; for interactive frame boundary mask blocks with changing pixels, the interactive frame boundary mask blocks are used to replace the initial boundary mask blocks.

[0118] Optionally, for overlapping regions of adjacent interactive frame boundary mask blocks, a pixel may be covered by multiple optimized interactive frame boundary mask blocks. The average interactive frame pixel value of that pixel can be obtained by averaging the individual pixel values ​​of these interactive frame boundary mask blocks. A threshold of 0.5 can be used: if the average interactive frame pixel value of a pixel is greater than 0.5, then that pixel is considered the foreground of the target object; otherwise, it is considered the background of the target object.

[0119] Figure 6 This is a diagram of the architecture of a time-series propagation network. The entire process of the time-series propagation network can be divided into a memory part and a query part. The memory part encodes historical frame images and their corresponding historical frame segmentation masks; the query part encodes the current frame image in the video. Both the memory and query parts output their encodings in key-value pairs; that is, the memory part outputs historical frame key-value pairs, and the query part outputs the current frame key-value pairs. Specifically:

[0120] First, a memory encoder is used to encode the historical frame images and their corresponding historical frame masks to obtain historical frame key-value pairs. Specifically, the memory encoder processes multiple historical frame images and their corresponding historical frame segmentation masks stored in memory, and generates a set of corresponding "historical frame key-value pairs" for each historical frame image.

[0121] Second, a query encoder is used to encode the current frame image to obtain the current frame key-value pairs.

[0122] Third, through the memory retrieval module that integrates spatial prior information (i.e., the spatial prior fusion module), the historical frame weights between the current frame key encoded by the query encoder and each historical frame key encoded by the memory encoder are calculated. The historical frame weights are then used to address the historical frame values ​​of the memory encoder. The historical frame values ​​are combined using the historical frame weights to output the current frame comprehensive feature map. The spatial prior fusion module connects the current frame comprehensive feature map with the previous frame segmentation mask to update the current frame comprehensive feature map. The spatial prior fusion module performs convolution and nonlinear activation operations on the current frame comprehensive feature map to obtain a spatial prior probability map. Finally, the spatial prior fusion module multiplies the current frame comprehensive feature map and the spatial prior probability map to obtain the current frame spatial weighted feature map.

[0123] Finally, the spatial weighted feature map of the current frame is obtained through the decoder and the segmentation mask of the current frame is reconstructed.

[0124] Figure 7 This is a schematic diagram for integrating the spatial prior module. Specifically:

[0125] First, by calculating the current frame key and history frame key The historical frame weight is calculated by comparing the similarity between all pixels of the historical frame key and the current frame key. For example, the historical frame weight can be obtained by comparing the spatial similarity of the pixels of each historical frame key and the current frame key.

[0126] ;

[0127] In the formula, For historical frame weights, where, Indicates a connection; It represents the pixel at the i-th spatial position in the current frame key; Let j be the pixel at the j-th spatial position in the historical frame key; This indicates dot product.

[0128] Then, by weighting the historical frame values ​​according to their weights, a weighted sum is calculated to retrieve the historical frame values ​​from the memory portion. Generate a comprehensive feature map of historical frames. Then, compare the comprehensive feature map of historical frames with the current frame value. Connect them to generate Shape of the current frame comprehensive feature map .

[0129] For example, the comprehensive feature map of the current frame can be calculated using the following formula:

[0130] ;

[0131] In the formula, This represents the pixel value at the i-th spatial location in the current integrated feature map; Let be the pixel at the i-th spatial position in the current frame; Z represents the normalization factor. For historical frame weights, Indicates a connection; This refers to the i-th spatial position pixel in the current frame key; Let j be the pixel at the j-th spatial position in the historical frame key; Let be the pixel at the j-th spatial location in the historical frame values.

[0132] Third, synthesize the feature map of the current frame. Connect it with the previous frame mask (i.e., the previous frame segmentation mask) to generate Based on shape features, update the comprehensive feature map of the current frame. .

[0133] Fourth, synthesize the updated feature map of the current frame. A convolution operation with a kernel size of 3 and a non-linear activation operation are used to generate a shape of... The single-channel spatial probability map is used as the spatial prior probability map. The spatial prior probability map provides a rough prior for predicting the segmentation mask of the current frame. The weights of the spatial prior probability map enable the temporal propagation network to more accurately and completely identify the location of the target object, avoiding the identification of pixels in the current frame image that are similar to the target object but are not related to it, thus reducing the false positive rate of target object identification.

[0134] Finally, the spatial prior probability map is combined with the updated current frame feature map. Multiply them to obtain the spatially weighted feature map of the current frame.

[0135] The testing experiments of this invention were primarily validated on the DAVIS (Davis Benchmark Dataset, densely annotated video segmentation) benchmark dataset. The DAVIS dataset is a high-quality, high-resolution densely annotated video segmentation dataset available in 480p and 1080p resolutions. For interactive video object segmentation, performance is generally measured using two metrics: Area Under the Curve (AUC) and accuracy at 60 seconds (J&F@60s). Both metrics are calculated on a curve that shows the average of region similarity (J) and contour accuracy (F) as a function of time. AUC measures the overall accuracy over multiple rounds of segmentation. J&F@60s measures accuracy with a 60-second time budget. Specifically, this metric is calculated by interpolating the curve at 60 seconds.

[0136] Table 1 presents the experimental results of our proposed video object segmentation method on the DAVIS validation set. Since interactive tasks are relatively new, the models listed are all from the last three years. Our proposed video object segmentation method performs well on the most important J&F@60s metric, outperforming most methods from recent years, demonstrating a good balance between accuracy and speed.

[0137] Table 1. Experimental results of the video object segmentation method on the DAVIS validation set.

[0138]

[0139] An ideal interactive segmentation method should deliver predictable performance for objects of unknown categories or unusual user needs. Therefore, the ideal characteristic of an interactive segmentation method is convergence, which is also our evaluation metric for interactive segmentation methods. That is, as the number of clicks increases, the accuracy of segmentation can be improved, eventually reaching a certain level of accuracy. Figure 8 The graph shows the segmentation convergence of the interactive understanding network, with the horizontal axis representing the number of interactions per frame. It can be seen that, compared to the high-performance DOS, the interactive understanding network optimized with a boundary optimization network proposed in this paper achieves significantly higher segmentation accuracy within three clicks than either DOS or the interactive understanding network without a boundary optimization network, demonstrating a significant advantage in convergence speed. Specifically, the interactive understanding network without a boundary optimization network (gray curve) eliminates the boundary optimization network compared to the interactive understanding network with a boundary optimization network (yellow curve), while keeping other settings unchanged. The interactive understanding network with the boundary optimization network (gray curve) shows a 1 percentage point improvement in mIoU after saturation, confirming the effectiveness of the boundary optimization network in improving the segmentation accuracy of edge information.

[0140] Table 2 presents a quantitative evaluation of the effectiveness of the modules in the proposed video object segmentation device, based on ablation experiments conducted on the DAVIS validation set. The addition of a boundary optimization network, a fusion spatial prior module, and a feature fusion strategy combining historical frame key-value pairs and current frame key-value pairs to the Baseline (a method combining the baseline model in interaction understanding and the baseline model in temporal understanding) further improves performance. The last row replaces the interaction understanding network with a ground truth mask, showcasing the ideal experimental results from perfect interaction, demonstrating that there is still room for optimization in the interaction understanding network.

[0141] Table 2 Ablation experimental results on the DAVIS validation set

[0142]

[0143] Among them, IUTPv1 refers to the device with the addition of a boundary optimization network; IUTPv2 refers to the device with the addition of a boundary optimization network and a fusion spatial prior module; IUTPv3 refers to the device with the addition of a boundary optimization network, a fusion spatial prior module, and a feature fusion strategy; and IUTPv4 refers to the device that uses real labeled masks directly for interactive segmentation with the addition of a fusion spatial prior module.

[0144] The technical solution of this invention addresses the problems of blurred boundaries and interference from similar backgrounds during time propagation in interactive understanding. Specifically, user interaction image generation involves a process of downscaling and upscaling. When the low-resolution mask is restored to its original size, the boundary information is very coarse, resulting in an imperfect initial segmentation mask. Furthermore, the number of pixels at the boundary positions is relatively small compared to the entire image, and they are inherently difficult to classify. Therefore, a boundary optimization network is proposed. This network extracts the blurred information at the mask boundary positions using bounding boxes and filters it using non-maximum suppression. The boundary optimization network then optimizes the bounding boxes at the edge positions, improving the accuracy of mask boundary prediction. In real-world scenarios, videos often focus on many similar objects, such as cars on a road or pedestrians on a sidewalk. When the feature differences between the target and similar background objects are small, the background can easily be misclassified as part of the segmentation result, leading to appearance confusion and false positive predictions. Therefore, a spatial prior constraint fusion module is proposed. Based on the key-value combination of the current frame and historical frames, a strong spatial prior from the previous frame is added. By making full use of the consistency between frames, spatial prior information and appearance matching information are fused together, effectively overcoming the false positive prediction problem caused by similar objects.

[0145] Example 4

[0146] Figure 9 This is a schematic diagram of a video object segmentation device provided in Embodiment 4 of the present invention. This embodiment of the present invention is applicable to situations where video objects are segmented using an interactive video object segmentation method. The device can execute the video object segmentation method and can be implemented in hardware and / or software. The device can be configured in an electronic device that carries video object segmentation functionality.

[0147] See Figure 9The video object segmentation device shown includes: an initial segmentation mask generation module 901, an initial boundary mask block extraction module 902, a boundary mask block optimization module 903, an interactive frame segmentation mask generation module 904, a historical frame segmentation mask acquisition module 905, and a current frame segmentation mask generation module 906. The initial segmentation mask generation module 901 acquires an interactive frame image, a reference segmentation mask, and a user-annotated image of the interactive frame image, and uses an interactive understanding network to identify target objects based on the interactive frame image, the reference segmentation mask, and the user-annotated image to obtain an initial segmentation mask. The initial boundary mask block extraction module 902 extracts initial boundary mask blocks along the predicted object boundaries of the target object in the initial segmentation mask, and extracts initial boundary image blocks corresponding to the initial boundary mask blocks in the interactive frame image. The boundary mask block optimization module 903 uses a boundary optimization network to optimize the segmentation mask based on the initial boundary mask blocks and the initial boundary image blocks. The initial boundary mask block is optimized to obtain an interactive frame boundary mask block; the interactive frame segmentation mask generation module 904 is used to adjust the corresponding initial boundary mask block in the initial segmentation mask according to the interactive frame boundary mask block to obtain an interactive frame segmentation mask; the historical frame segmentation mask acquisition module 905 is used to acquire the current frame image and determine the historical frame image and historical frame segmentation mask according to the current frame image, the interactive frame image and the interactive frame segmentation mask; the current frame segmentation mask generation module 906 is used to determine the current frame segmentation mask using a time-series propagation network according to the current frame image, the historical frame image and the historical frame segmentation mask.

[0148] The technical solution of this invention involves acquiring an interactive frame image, a reference segmentation mask, and user-annotated images of the interactive frame image. An interactive understanding network is used to identify target objects based on these elements, obtaining an initial segmentation mask. An initial boundary mask block is extracted along the predicted object boundary of the target object in the initial segmentation mask. Additionally, an initial boundary image block corresponding to the initial boundary mask block is extracted from the interactive frame image. A boundary optimization network is used to optimize the initial boundary mask block based on the initial boundary mask block and the initial boundary image block, resulting in an interactive frame boundary mask block. Finally, based on the interactive frame boundary mask block, the initial segmentation mask is optimized to identify the target object in the interactive frame image. The initial boundary mask block is adjusted to obtain the interactive frame segmentation mask, which realizes the boundary adjustment of the initial segmentation mask corresponding to the interactive frame image and improves the boundary clarity of the interactive frame segmentation mask. By acquiring the current frame image and determining the historical frame image and historical frame segmentation mask based on the current frame image, interactive frame image, and interactive frame segmentation mask, a time-series propagation network is used to determine the current frame segmentation mask based on the current frame image, historical frame image, and historical frame segmentation mask. Through time-series propagation of the interactive frame segmentation mask, the current frame segmentation mask of the current frame image is obtained, which improves the boundary clarity of the current frame segmentation mask and thus improves the segmentation quality of video objects.

[0149] In an optional embodiment of the present invention, the apparatus includes: an initial boundary mask block filtering module, configured to, after extracting initial boundary mask blocks along the predicted object boundary of the target object in the initial segmentation mask, filter each of the initial boundary mask blocks using a non-maximum suppression method and update each of the initial boundary mask blocks.

[0150] In an optional embodiment of the present invention, the interactive frame segmentation mask generation module 904 includes: an interactive frame pixel value comparison unit, configured to compare the interactive frame pixel value of the interactive frame boundary mask block with the initial pixel value of the corresponding initial boundary mask block for a single initial boundary mask block; a first initial boundary mask block adjustment unit, configured to not adjust the initial boundary mask block for a single initial boundary mask block when the interactive frame pixel value is consistent with the corresponding initial pixel value; a second initial boundary mask block adjustment unit, configured to adjust the initial boundary mask block for a single initial boundary mask block according to the interactive frame boundary mask block when the interactive frame pixel value is inconsistent with the corresponding initial pixel value; and an interactive frame segmentation mask generation unit, configured to generate an interactive frame segmentation mask block according to the adjustment result of each initial boundary mask block when all initial boundary masks in the initial segmentation mask have been adjusted.

[0151] In an optional embodiment of the present invention, the interactive frame segmentation mask generation module 904 further includes: an interactive frame boundary mask block overlap detection unit, configured to perform overlap detection on the pixels of each interactive frame boundary mask block after generating the interactive frame segmentation mask block according to the adjustment results of each initial boundary mask block; an interactive frame overlap pixel mean calculation unit, configured to calculate the interactive frame overlap pixel mean of the overlapping pixels in each interactive frame boundary overlap mask block when overlapping pixels exist in the interactive frame boundary mask block; an interactive frame overlap pixel mean detection unit, configured to compare the interactive frame overlap pixel mean of the overlapping pixels with a preset pixel value threshold; a foreground determination unit, configured to determine the overlapping pixels as the foreground of the target object when the interactive frame overlap pixel mean of the overlapping pixels is greater than the preset pixel value threshold; and a background determination unit, configured to determine the overlapping pixels as the background of the target object when the interactive frame overlap pixel mean of the overlapping pixels is less than or equal to the preset pixel value threshold.

[0152] In an optional embodiment of the present invention, the current frame segmentation mask generation module 906 includes: a historical frame key-value pair extraction unit, configured to encode the historical frame image and the corresponding historical frame segmentation mask using a memory encoder of a time-propagation network to obtain historical frame key-value pairs; a current frame key-value pair extraction unit, configured to encode the current frame image using a query encoder of the time-propagation network to obtain current frame key-value pairs; and a current frame comprehensive feature map generation unit, configured to generate a current frame based on the historical frame key-value pairs and the current frame key-value pairs using a fusion spatial prior module of the time-propagation network. The system comprises: a comprehensive feature map generation unit; a spatial prior probability map generation unit, used to perform convolution and nonlinear activation operations on the comprehensive feature map of the current frame using the fusion spatial prior module of the temporal propagation network to obtain a spatial prior probability map; a current frame spatial weighted feature map generation unit, used to multiply the comprehensive feature map of the current frame and the spatial prior probability map using the fusion spatial prior module of the temporal propagation network to obtain a current frame spatial weighted feature map; and a current frame segmentation mask generation unit, used to decode the current frame spatial weighted feature map using the decoder of the temporal propagation network to obtain a current frame segmentation mask.

[0153] In an optional embodiment of the present invention, the current frame comprehensive feature map generation unit includes: a historical frame weight calculation subunit, used to determine the historical frame weight of the historical frame value based on the historical frame key and the current frame key using the fusion spatial prior module of the time-series propagation network; a historical frame comprehensive feature map generation subunit, used to synthesize each of the historical frame values ​​based on the historical frame weight using the fusion spatial prior module of the time-series propagation network to obtain a historical frame comprehensive feature map; and a current frame comprehensive feature map generation subunit, used to connect the historical frame comprehensive feature map and the current frame value using the fusion spatial prior module of the time-series propagation network to obtain a current frame comprehensive feature map.

[0154] In an optional embodiment of the present invention, the current frame segmentation mask generation module 906 further includes: a previous frame segmentation mask filtering unit, used to filter the previous frame segmentation mask corresponding to the previous frame image of the current frame image from among the historical frame segmentation masks before the fusion spatial prior module using the temporal propagation network performs convolution operation and nonlinear activation operation on the current frame comprehensive feature map to obtain a spatial prior probability map; and a current frame comprehensive feature map updating unit, used to connect the current frame comprehensive feature map and the previous frame segmentation mask to update the current frame comprehensive feature map.

[0155] The video object segmentation apparatus provided in the embodiments of the present invention can execute the video object segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0156] In the technical solutions of this invention, the acquisition, storage, and application of interactive frame images, reference segmentation masks, user-annotated images, and current frame images all comply with relevant laws and regulations and do not violate public order and good morals.

[0157] Example 5

[0158] Figure 10 A schematic diagram of an electronic device 1000 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0159] like Figure 10As shown, the electronic device 1000 includes at least one processor 1001 and a memory, such as a read-only memory (ROM) 1002 or a random access memory (RAM) 1003, communicatively connected to the at least one processor 1001. The memory stores computer programs executable by the at least one processor. The processor 1001 can perform various appropriate actions and processes based on the computer program stored in the ROM 1002 or loaded into the RAM 1003 from storage unit 1008. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The processor 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0160] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0161] Processor 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 1001 performs the various methods and processes described above, such as video object segmentation methods.

[0162] In some embodiments, the video object segmentation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by processor 1001, one or more steps of the video object segmentation method described above may be performed. Alternatively, in other embodiments, processor 1001 may be configured to perform the video object segmentation method by any other suitable means (e.g., by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0167] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0168] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0169] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0170] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video object segmentation method, characterized in that, The method includes: The interaction frame image, reference segmentation mask, and user-annotated image of the interaction frame image are acquired. An interaction understanding network is used to identify the target object based on the interaction frame image, reference segmentation mask, and user-annotated image to obtain an initial segmentation mask. Along the predicted object boundary of the target object in the initial segmentation mask, an initial boundary mask block is extracted, and in the interactive frame image, an initial boundary image block corresponding to the initial boundary mask block is extracted; A boundary optimization network is used to optimize the initial boundary mask block based on the initial boundary mask block and the initial boundary image block to obtain the interactive frame boundary mask block; Based on the interactive frame boundary mask block, the corresponding initial boundary mask block in the initial segmentation mask is adjusted to obtain the interactive frame segmentation mask; Obtain the current frame image, and determine the historical frame image and historical frame segmentation mask based on the current frame image, the interactive frame image, and the interactive frame segmentation mask; A time-series propagation network is used to determine the current frame segmentation mask based on the current frame image, the historical frame images, and the historical frame segmentation mask.

2. The method according to claim 1, characterized in that, After extracting the initial boundary mask block along the predicted object boundary of the target object in the initial segmentation mask, the process includes: The non-maximum suppression method is used to filter each of the initial boundary mask blocks and update each of the initial boundary mask blocks.

3. The method according to claim 1, characterized in that, The step of adjusting the corresponding initial boundary mask block in the initial segmentation mask based on the interaction frame boundary mask block to obtain the interaction frame segmentation mask includes: For a single initial boundary mask block, the interaction frame pixel value of the interaction frame boundary mask block is compared with the initial pixel value of the corresponding initial boundary mask block; For a single initial boundary mask block, if the pixel value of the interaction frame is consistent with the corresponding initial pixel value, the initial boundary mask block is not adjusted; For a single initial boundary mask block, when the pixel value of the interaction frame is inconsistent with the corresponding initial pixel value, the initial boundary mask block is adjusted according to the interaction frame boundary mask block; When all the initial boundary mask blocks in the initial segmentation mask have been adjusted, an interactive frame segmentation mask block is generated based on the adjustment results of each initial boundary mask block.

4. The method according to claim 3, characterized in that, After generating the interactive frame segmentation mask block based on the adjustment results of each of the initial boundary mask blocks, the method further includes: Overlap detection is performed on the pixels of each interactive frame boundary mask block; When there are overlapping pixels in the boundary mask block of the interaction frame, calculate the average value of the overlapping pixels in the interaction frame corresponding to each boundary overlapping mask block of the interaction frame. The average value of overlapping pixels in the interactive frames of the overlapping pixels is compared with a preset pixel value threshold. When the average value of overlapping pixels in the interactive frames of the overlapping pixels is greater than a preset pixel value threshold, the overlapping pixels are determined to be the foreground of the target object; When the average value of overlapping pixels in the interactive frames of the overlapping pixels is less than or equal to a preset pixel value threshold, the overlapping pixels are determined to be the background of the target object.

5. The method according to claim 1, characterized in that, The step of employing a time-series propagation network to determine the current frame segmentation mask based on the current frame image, the historical frame images, and the historical frame segmentation mask includes: A memory encoder using a time-propagation network is used to encode the historical frame image and the corresponding historical frame segmentation mask to obtain historical frame key-value pairs; The query encoder of the time-series propagation network is used to encode the current frame image to obtain the current frame key-value pairs; Using the fusion spatial prior module of the time-series propagation network, a comprehensive feature map of the current frame is generated based on the historical frame key-value pairs and the current frame key-value pairs. The spatial prior fusion module of the time-series propagation network is used to perform convolution and nonlinear activation operations on the comprehensive feature map of the current frame to obtain a spatial prior probability map. The fusion spatial prior module of the time-series propagation network is used to multiply the current frame comprehensive feature map and the spatial prior probability map to obtain the current frame spatial weighted feature map; The decoder of the time-series propagation network is used to decode the spatially weighted feature map of the current frame to obtain the segmentation mask of the current frame.

6. The method according to claim 5, characterized in that, The fusion spatial prior module employing the temporal propagation network generates a comprehensive feature map of the current frame based on the historical frame key-value pairs and the current frame key-value pairs, including: The fusion spatial prior module of the time-series propagation network is used to determine the historical frame weight of the historical frame value based on the historical frame key and the current frame key. Using the fusion spatial prior module of the time-series propagation network, the historical frame values ​​are integrated according to the historical frame weights to obtain a comprehensive feature map of historical frames; The fusion spatial prior module of the time-series propagation network is used to connect the historical frame comprehensive feature map and the current frame value to obtain the current frame comprehensive feature map.

7. The method according to claim 5, characterized in that, Before the fusion spatial prior module employing the temporal propagation network performs convolution and nonlinear activation operations on the current frame synthesized feature map to obtain the spatial prior probability map, the method further includes: Among the historical frame segmentation masks, the previous frame segmentation mask corresponding to the previous frame image of the current frame image is selected; The current frame integrated feature map and the previous frame segmentation mask are connected to update the current frame integrated feature map.

8. A video object segmentation device, characterized in that, The device includes: The initial segmentation mask generation module is used to acquire the interactive frame image, the reference segmentation mask, and the user-annotated image of the interactive frame image, and to use an interactive understanding network to identify the target object based on the interactive frame image, the reference segmentation mask, and the user-annotated image to obtain the initial segmentation mask; The initial boundary mask block extraction module is used to extract the initial boundary mask block along the predicted object boundary of the target object in the initial segmentation mask, and to extract the initial boundary image block corresponding to the initial boundary mask block in the interactive frame image; A boundary mask block optimization module is used to optimize the initial boundary mask block based on the initial boundary mask block and the initial boundary image block using a boundary optimization network to obtain an interactive frame boundary mask block. An interactive frame segmentation mask generation module is used to adjust the corresponding initial boundary mask block in the initial segmentation mask according to the interactive frame boundary mask block to obtain an interactive frame segmentation mask. The historical frame segmentation mask acquisition module is used to acquire the current frame image and determine the historical frame image and historical frame segmentation mask based on the current frame image, the interactive frame image and the interactive frame segmentation mask; The current frame segmentation mask generation module is used to determine the current frame segmentation mask using a time-series propagation network based on the current frame image, the historical frame image, and the historical frame segmentation mask.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video object segmentation method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the video object segmentation method according to any one of claims 1-7.

Images