Video target segmentation method, system, terminal and storage medium

Abstract

The present invention discloses a method, system, terminal, and storage medium for target video segmentation. The method includes: determining past frame features associated with a video frame to be segmented based on past segmentation results; determining spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and unsegmented video frames that are temporally subsequent to the video frame to be segmented; and generating a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features. The technical solution of this application improves video segmentation performance.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, system, terminal and storage medium for video target segmentation. Background Art

[0002] Video object segmentation, especially video portrait segmentation, is particularly widely used in the industry. Currently, the general algorithm process is to maintain a memory block to store the segmentation results of the previous frame. When segmenting the t-th frame of the video, the segmentation results of the t-th frame stored in the memory block are referenced to obtain a temporally stable segmentation result. However, if there are lens splicing or camera position changes in the t-th frame, the image content of the t-th frame will be suddenly different from the previous video sequence. The appearance and position of the target to be segmented in the image will also change dramatically. No information related to the target in the t-th frame will be found in the memory block, resulting in poor segmentation results. Summary of the Invention

[0003] The embodiments of the present application aim to solve the problem of poor video segmentation effect by providing a method, system, terminal and storage medium for target segmentation of a video.

[0004] An embodiment of the present application provides a method for segmenting an object in a video, the method comprising:

[0005] Determining past frame features associated with the video frame to be segmented based on past segmentation results;

[0006] Determining the spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and an unsegmented video frame that is subsequent in time sequence to the video frame to be segmented;

[0007] Based on the spatiotemporal global features and the past frame features, a segmentation result corresponding to the video frame to be segmented is generated.

[0008] In one embodiment, the step of determining the spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and the unsegmented video frames that are temporally subsequent to the video frame to be segmented comprises:

[0009] Inputting the past segmentation results and the unsegmented video frames into a spatiotemporal global convolutional network, and obtaining an output result of the spatiotemporal global convolutional network;

[0010] The output result is used as the input of the spatial domain convolutional network, and the spatiotemporal global features are obtained through the spatial domain convolutional network.

[0011] In one embodiment, the spatiotemporal global convolutional network includes a two-dimensional spatial convolution kernel and a one-dimensional temporal convolution kernel, and the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel are connected in series; the step of inputting the past segmentation result and the unsegmented video frame into the spatiotemporal global convolutional network and obtaining the output result of the spatiotemporal global convolutional network includes:

[0012] The past segmentation result and the unsegmented video frame are sequentially passed through the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel to obtain a video frame after spatiotemporal global convolution;

[0013] The past segmentation results, the unsegmented video frames, and the video frames after the spatiotemporal global convolution are added together to obtain an output result of the spatiotemporal global convolutional network.

[0014] In one embodiment, the step of inputting the past segmentation results and the unsegmented video frame into a spatiotemporal global convolutional network and obtaining an output result of the spatiotemporal global convolutional network includes:

[0015] Changing the pixel value of the background area in the previous segmentation result;

[0016] The past segmentation result whose pixel value has been changed and the unsegmented video frame are input into a spatiotemporal global convolutional network, and an output result of the spatiotemporal global convolutional network is obtained.

[0017] In one embodiment, the step of generating a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features includes:

[0018] Determining downsampling features of the video frame to be segmented;

[0019] Determining a splicing feature of the spatiotemporal global feature and the past frame feature;

[0020] Inputting the down-sampling features and the splicing features into a decoding network, and obtaining an output result of the decoding network;

[0021] The segmentation result corresponding to the video frame to be segmented is obtained according to the output result of the decoding network.

[0022] In one embodiment, the step of determining past frame features associated with the video frame to be segmented based on past segmentation results includes:

[0023] Determining index features of the video frame to be segmented;

[0024] Determining a target past segmentation result that matches the index feature;

[0025] The past frame features are determined based on the target past segmentation results.

[0026] In one embodiment, after the step of generating a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features, the method further includes:

[0027] The segmentation result corresponding to the video frame to be segmented is used to update the previous segmentation result.

[0028] In addition, to achieve the above-mentioned purpose, the present invention also provides a video object segmentation system, which includes:

[0029] A past frame feature determination module, configured to determine past frame features associated with the video frame to be segmented based on past segmentation results;

[0030] a spatiotemporal global feature determination module, configured to determine the spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and an unsegmented video frame that is subsequent in time sequence to the video frame to be segmented;

[0031] The segmentation result determination module is used to generate a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides an intelligent terminal, which includes: a memory, a processor, and a target segmentation program for a video stored in the memory and runnable on the processor. When the target segmentation program for the video is executed by the processor, the steps of the above-mentioned target segmentation method for the video are implemented.

[0033] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium on which a video target segmentation program is stored. When the video target segmentation program is executed by a processor, the steps of the above-mentioned video target segmentation method are implemented.

[0034] The technical solutions of a method, system, terminal and storage medium for target segmentation of a video provided in the embodiments of the present application adopt a method of determining past frame features associated with the video frame to be segmented based on past segmentation results, determining spatiotemporal global features associated with the video frame to be segmented based on past segmentation results and unsegmented video frames that are temporally subsequent to the video frame to be segmented, and generating a technical solution of segmentation results corresponding to the video frame to be segmented based on the spatiotemporal global features and past frame features. Since the spatiotemporal global features are added in the process of video target segmentation, the target is segmented using the global information of the video, making the segmentation results more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the structure of an intelligent terminal involved in an embodiment of the present invention;

[0036] Figure 2 1. A schematic flow chart of a first embodiment of a method for object segmentation of a video according to the present invention;

[0037] Figure 3 This is a functional module diagram of the target segmentation system of the video of the present invention;

[0038] Figure 4 Schematic diagram of the process of video target segmentation using spatiotemporal global features in the present invention;

[0039] Figure 5 Schematic diagram of the connection of the spatiotemporal global convolution module of the present invention;

[0040] Figure 6 Schematic diagram of the spatiotemporal global feature extraction network of the present invention;

[0041] Figure 7 Schematic diagram of the input video frame of the spatiotemporal global feature extraction network of the present invention.

[0042] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings are only an embodiment diagram, not the entire invention. DETAILED DESCRIPTION

[0043] In order to solve the problem of unstable video segmentation effect, the present application proposes a video target segmentation method, which adopts the method of determining past frame features associated with the video frame to be segmented based on past segmentation results, determining the spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and the unsegmented video frames that are temporally located after the video frame to be segmented, and generating a technical solution of segmentation results corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features. Since the spatiotemporal global features are added in the video target segmentation process, the target is segmented using the global information of the video, making the segmentation result more stable.

[0044] To better understand the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0045] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0046] It should be noted that Figure 1 This is a structural diagram of the hardware operating environment of the smart terminal.

[0047] like Figure 1 As shown, the intelligent terminal may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art will understand that Figure 1 The structure of the smart terminal shown in the figure does not constitute a limitation on the smart terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0049] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a video target segmentation program. The operating system is a program that manages and controls the hardware and software resources of the intelligent terminal, and the video target segmentation program and other software or programs are executed.

[0050] exist Figure 1 In the smart terminal shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used to communicate data with the background server; the processor 1001 can be used to call the target segmentation program of the video stored in the memory 1005.

[0051] In this embodiment, the intelligent terminal includes: a memory 1005, a processor 1001, and a target segmentation program for a video stored in the memory and executable on the processor, wherein:

[0052] When the processor 1001 calls the target segmentation program of the video stored in the memory 1005, it performs the following operations:

[0053] Determining past frame features associated with the video frame to be segmented based on past segmentation results;

[0054] Determining the spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and an unsegmented video frame that is subsequent in time sequence to the video frame to be segmented;

[0055] Based on the spatiotemporal global features and the past frame features, a segmentation result corresponding to the video frame to be segmented is generated.

[0056] When the processor 1001 calls the target segmentation program of the video stored in the memory 1005, it also performs the following operations:

[0057] Inputting the past segmentation results and the unsegmented video frames into a spatiotemporal global convolutional network, and obtaining an output result of the spatiotemporal global convolutional network;

[0058] The output result is used as the input of the spatial domain convolutional network, and the spatiotemporal global features are obtained through the spatial domain convolutional network.

[0059] When the processor 1001 calls the target segmentation program of the video stored in the memory 1005, it also performs the following operations:

[0060] The past segmentation result and the unsegmented video frame are sequentially passed through the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel to obtain a video frame after spatiotemporal global convolution;

[0061] The past segmentation results, the unsegmented video frames, and the video frames after the spatiotemporal global convolution are added together to obtain an output result of the spatiotemporal global convolutional network.

[0062] When the processor 1001 calls the target segmentation program of the video stored in the memory 1005, it also performs the following operations:

[0063] Changing the pixel value of the background area in the previous segmentation result;

[0064] The past segmentation result whose pixel value has been changed and the unsegmented video frame are input into a spatiotemporal global convolutional network, and an output result of the spatiotemporal global convolutional network is obtained.

[0065] When the processor 1001 calls the target segmentation program of the video stored in the memory 1005, it also performs the following operations:

[0066] Determining downsampling features of the video frame to be segmented;

[0067] Determining a splicing feature of the spatiotemporal global feature and the past frame feature;

[0068] Inputting the down-sampling features and the splicing features into a decoding network, and obtaining an output result of the decoding network;

[0069] The segmentation result corresponding to the video frame to be segmented is obtained according to the output result of the decoding network.

[0070] When the processor 1001 calls the target segmentation program of the video stored in the memory 1005, it also performs the following operations:

[0071] Determining index features of the video frame to be segmented;

[0072] Determining a target past segmentation result that matches the index feature;

[0073] The past frame features are determined based on the target past segmentation results.

[0074] When the processor 1001 calls the target segmentation program of the video stored in the memory 1005, it also performs the following operations:

[0075] The segmentation result corresponding to the video frame to be segmented is used to update the previous segmentation result.

[0076] The technical solutions of this application will be introduced below in the form of embodiments.

[0077] like Figure 2 As shown, in the first embodiment of the present application, the object segmentation method of the video of the present application includes the following steps:

[0078] Step S110 , determining past frame features associated with the video frame to be segmented based on past segmentation results.

[0079] In this embodiment, the past frame feature is a feature of a video frame that is located before the video frame to be segmented in time sequence, and the past segmentation result is a segmentation result of a video frame that is located before the video frame to be segmented in time sequence. In the process of obtaining the target through video segmentation, the video frames in the video frame sequence are extracted in sequence. The segmented video frames and the video frames that are located before the video frame to be segmented in time sequence are regarded as past frames, and the segmentation results of the past frames, i.e., the past segmentation results, are stored in the past frame segmentation result memory module. In the process of performing target segmentation on subsequent video frames, i.e., the video frames to be segmented, the past segmentation results stored in the memory module are referred to, and a temporally stable segmentation result is obtained. Therefore, in the process of extracting the video frame to be segmented for segmentation, the present application determines the past frame feature associated with the video frame to be segmented based on the past segmentation results in the past frame segmentation result memory module.

[0080] Step S120 , determining the spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and the unsegmented video frames that are temporally subsequent to the video frame to be segmented.

[0081] In this embodiment, the unsegmented video frame is a video frame that is temporally subsequent to the video frame to be segmented. A spatiotemporal global feature is obtained by inputting past segmentation results that are temporally preceding the video frame to be segmented and unsegmented video frames that are temporally subsequent to the video frame to be segmented into a spatiotemporal global feature extraction network. This spatiotemporal global feature is then associated with the video frame to be segmented. This spatiotemporal global feature is used to characterize the correlation between the past segmentation results and the unsegmented video frame.

[0082] Step S130 : generating a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features.

[0083] In this embodiment, after obtaining the spatiotemporal global features and the features of the past frames, these features are input into the decoding network to obtain the segmentation results corresponding to the video frame to be segmented. Optionally, in the process of segmenting the target in the video frame, methods such as threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories can also be used to segment the target in the video.

[0084] On the premise of utilizing the segmentation information of past frames, the spatiotemporal global features of the target to be segmented in the entire video are used to improve the global stability of the segmentation results and maintain good segmentation effects in cross-border situations.

[0085] According to the above technical solution, when segmenting the video frame to be segmented, this embodiment fully considers the video information of the past frames before the video frame to be segmented and the unsegmented video frames after the video frame to be segmented, so that a good segmentation effect is produced when a shot switch occurs in the video frame to be segmented.

[0086] In one embodiment, determining the spatiotemporal global features associated with the video frame to be segmented based on the past segmentation results and unsegmented video frames that are temporally subsequent to the video frame to be segmented specifically includes the following steps:

[0087] Step S121: input the past segmentation results and the unsegmented video frames into a spatiotemporal global convolutional network, and obtain an output result of the spatiotemporal global convolutional network.

[0088] Step S122: Using the output result as the input of a spatial convolutional network, and obtaining the spatiotemporal global features through the spatial convolutional network.

[0089] In this embodiment, referring to Figure 7 , extract the past segmentation results from the past frame segmentation result memory module (i.e. Figure 7 The segmented frames in the video are extracted, and the unsegmented video frames whose timing is after the video frames to be segmented are extracted from the video frame sequence (i.e. Figure 7The frames to be segmented in the video are input into the spatiotemporal global convolutional network. Figure 6 As shown in the figure, the spatiotemporal global feature extraction network consists of two parts: a spatiotemporal global convolutional network and a spatial domain convolutional network. The spatiotemporal global convolutional network is composed of several spatiotemporal global convolutional modules, and the spatial domain convolutional network is also composed of several ordinary convolutional layers. The video frame passes through several spatiotemporal global convolutional modules to obtain rich target information, and then passes through several ordinary convolutional layers to obtain spatiotemporal global features. Specifically, the past segmentation results and the unsegmented video frame sequence are input into the spatiotemporal global convolutional network, and the spatiotemporal convolution operations are performed by several spatiotemporal global convolutional modules in the spatiotemporal global convolutional network to obtain the output results of the spatiotemporal global convolutional network.

[0090] In this embodiment, after obtaining the output of the spatiotemporal global convolutional network, the output is further input into the spatial convolutional network. The spatial convolution operation is performed by the ordinary convolution layer in the spatial convolutional network to obtain the spatiotemporal global features.

[0091] For example, the previous segmentation results and the unsegmented video frames can be arranged in time sequence to obtain a video frame sequence. This video frame sequence can be represented as 1*H*W*3T, where T is the number of video frames, H is the height of the video frame, and W is the width of the video frame. The video frames in the video frame sequence are input into the spatiotemporal global convolutional network to obtain an output result of 1*H / r*W / r*3T. The output result is then input into the spatial domain convolutional network to obtain a spatiotemporal global feature of 1*H / r*W / r*C, where r is the scaling ratio between the original video frame and the spatiotemporal global feature, and r can be 4, 8, 16, etc. C represents the channel of the video frame.

[0092] According to the above technical solution, this embodiment inputs the past segmentation results and the unsegmented video frames into a spatiotemporal global convolutional network and a spatial domain convolutional network to extract spatiotemporal global features.

[0093] In one embodiment, inputting the past segmentation results and the unsegmented video frames into a spatiotemporal global convolutional network and obtaining an output result of the spatiotemporal global convolutional network specifically comprises the following steps:

[0094] Step S1211, the previous segmentation result and the unsegmented video frame are sequentially subjected to the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel to obtain a video frame after spatiotemporal global convolution;

[0095] Step S1212: Add the previous segmentation results, the unsegmented video frames, and the video frames after the spatiotemporal global convolution to obtain the output result of the spatiotemporal global convolutional network.

[0096] In this embodiment, referring to Figure 5 , Figure 5 It is the specific structure of the spatiotemporal global convolutional network, which includes a two-dimensional spatial convolution kernel and a one-dimensional temporal convolution kernel, and the two-dimensional spatial convolution kernel is connected in series with the one-dimensional temporal convolution kernel. Specifically, the spatiotemporal global convolutional network is composed of a two-dimensional spatial convolution kernel of size S*S*n and a one-dimensional temporal convolution kernel of size 1*1*t in series, and the input past segmentation results and unsegmented video frames are added to the output spatiotemporal global convolution video frames to achieve residual learning. This computing module can process images of different frames in layers, extract hierarchical features, and can realize the flow of target features in the time dimension through temporal convolution. Among them, S, n, and t are all adjustable parameters, for example, S can be adjusted to 3, 5, or 7; n and t are both equal to 3T.

[0097] According to the above technical solution, this embodiment uses a spatiotemporal global convolutional network to perform spatiotemporal convolution operations on past segmentation results and the unsegmented video, thereby achieving hierarchical processing of different video frames.

[0098] In one embodiment, inputting the past segmentation results and the unsegmented video frames into a spatiotemporal global convolutional network and obtaining an output result of the spatiotemporal global convolutional network specifically comprises the following steps:

[0099] Step S2211, changing the pixel value of the background area in the previous segmentation result;

[0100] Step S2212: Input the past segmentation result with the changed pixel value and the unsegmented video frame into a spatiotemporal global convolutional network, and obtain an output result of the spatiotemporal global convolutional network.

[0101] In this embodiment, referring to Figure 7 The input to the spatiotemporal global feature extraction network consists of two parts: j frames of past frames and k frames of unsegmented video frames. Based on the previous segmentation results, the background pixel values ​​of the past frames are set to 0 to provide strong information about the segmented object. Unsegmented video frames do not require special processing. The modified pixel values ​​of the past segmentation results and the unsegmented video frames are then fed into the spatiotemporal global convolutional network, resulting in the network's output.

[0102] According to the above technical solution, this embodiment adopts a technical means of changing the pixel values ​​of the background area in the past segmentation results and then performing spatiotemporal global convolution processing, so that the past frames form strong information of the target to be segmented.

[0103] In one embodiment, generating a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features specifically includes the following steps:

[0104] Step S131, determining the downsampling features of the video frame to be segmented;

[0105] Step S132, determining a splicing feature of the spatiotemporal global feature and the past frame feature;

[0106] Step S133: inputting the down-sampling features and the splicing features into a decoding network, and obtaining an output result of the decoding network;

[0107] Step S134: obtaining a segmentation result corresponding to the video frame to be segmented according to the output result of the decoding network.

[0108] In this embodiment, the shape dimension of the spatiotemporal global feature is 1*H / r*W / r*C, where the values ​​of H and W are equal to the features of the previous frame. During the network's inference process, the spatiotemporal global feature is spliced ​​together with the features of the previous frame to form a new feature data, namely the spliced ​​feature. The video frame to be segmented is input into the spatiotemporal global feature extraction network to generate downsampled features of the video frame to be segmented. The downsampled features include 16x downsampled features, 8x downsampled features, and 4x downsampled features. The spliced ​​features are input into the decoding network together with the above-mentioned 16x downsampled features, 8x downsampled features, and 4x downsampled features to obtain the output result of the decoding network. In the decoding network, the spliced ​​features are sequentially downsampled by 16x, 8x, and 4x to obtain the output result of the decoding network. After obtaining the output result of the decoding network, the size of the output result needs to be converted to the same size as the video frame to be segmented, thereby obtaining the segmentation result corresponding to the video frame to be segmented.

[0109] Optionally, 16x downsampling features, 8x downsampling features, 4x downsampling features, spatiotemporal global features and past frame features can be input into the decoding network together, and the spatiotemporal global features and past frame features are spliced ​​in the decoding network. The spliced ​​features are then sequentially downsampled by 16x, 8x and 4x to obtain the output result of the decoding network.

[0110] According to the above technical solution, this embodiment adopts the method of inputting the downsampling features, spatiotemporal global features and past frame features of the video frame to be segmented into the decoding network to obtain the segmentation results corresponding to the video frame to be segmented, and adding a spatiotemporal global feature extraction network. During inference, spatiotemporal global features containing richer target information can be obtained, making the segmentation results more stable.

[0111] In one embodiment, determining the past frame features associated with the video frame to be segmented based on the past segmentation results specifically includes the following steps:

[0112] Step S111, determining the index features of the video frame to be segmented;

[0113] Step S112, determining the target past segmentation result that matches the index feature;

[0114] Step S113: determining the past frame features based on the target past segmentation results.

[0115] In this embodiment, the past segmentation results are stored in the past frame segmentation result memory module, and the past segmentation results include past frame features. The video frame to be segmented is passed through the feature extraction network to obtain an index feature. Feature matching is performed between the index feature and the past segmentation results in the past frame segmentation result memory module, and then the past frame features are determined based on the matched target past segmentation results. Specifically, the feature matching process is essentially a matching in the form of key-value pairs. The index feature represents the key of the video frame to be segmented, and the value corresponding to the key is matched in the past frame segmentation result memory module, that is, the target past segmentation result. The video feature represented by the target past segmentation result is used as the past frame feature associated with the video frame to be segmented.

[0116] According to the above technical solution, this embodiment matches the past frame features associated with the video frame to be segmented from the past frame segmentation result memory module through index features, thereby obtaining the past frame features for subsequent spatiotemporal global feature analysis.

[0117] In one embodiment, the method for object segmentation of a video further comprises the following steps:

[0118] Step S111, determining the index features of the video frame to be segmented;

[0119] Step S112, determining the target past segmentation result that matches the index feature;

[0120] Step S113, determining the past frame features based on the target past segmentation results;

[0121] Step S120, determining a spatiotemporal global feature associated with the video frame to be segmented based on the past segmentation results and an unsegmented video frame that is temporally subsequent to the video frame to be segmented;

[0122] Step S130, generating a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features;

[0123] Step S310: update the previous segmentation result using the segmentation result corresponding to the video frame to be segmented.

[0124] In this embodiment, referring to Figure 4, the video frame to be segmented passes through the feature extraction network to generate 4 feature data: index feature, 16x downsampling feature, 8x downsampling feature, and 4x downsampling feature. The above index feature is used to extract the past frame features of the target of the video frame to be segmented from the past frame segmentation result memory module through the feature matching module. A certain number of past frames that have obtained segmentation results and a certain number of unsegmented video frames that have not yet been segmented in the video to be segmented are sent to the spatiotemporal global feature extraction network to obtain the spatiotemporal global features of the target to be segmented between the above past frames and the unsegmented video frames. The above-mentioned past frame features, spatiotemporal global features, 16x downsampling features, 8x downsampling features, and 4x downsampling features are sent to the decoding network to obtain the segmentation results of the video frame to be segmented. After obtaining the segmentation result corresponding to the video frame to be segmented, the segmentation result is used to update the past frame segmentation result memory module.

[0125] According to the above technical solution, this embodiment adopts a technical means of updating the memory module of the previous frame segmentation results after obtaining the segmentation results. Therefore, in the process of analyzing the subsequent video frames to be segmented, the corresponding past segmentation results can be obtained from the memory module for analysis, thereby improving the segmentation stability.

[0126] The embodiment of the present invention provides an embodiment of a method for object segmentation of a video. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0127] like Figure 3 As shown, the present application provides a video object segmentation system, which includes:

[0128] The past frame feature determination module 10 is configured to determine past frame features associated with the video frame to be segmented based on the past segmentation results. In one embodiment, the past frame feature determination module 10 is further configured to determine an index feature of the video frame to be segmented; determine a target past segmentation result that matches the index feature; and determine the past frame features based on the target past segmentation result.

[0129] The spatiotemporal global feature determination module 20 is configured to determine the spatiotemporal global features associated with the video frame to be segmented based on the previous segmentation results and the unsegmented video frames that are temporally subsequent to the video frame to be segmented. In one embodiment, the spatiotemporal global feature determination module 20 is further configured to input the previous segmentation results and the unsegmented video frames into a spatiotemporal global convolutional network, obtain the output of the spatiotemporal global convolutional network, and use the output as the input of a spatial convolutional network to obtain the spatiotemporal global features through the spatial convolutional network. In one embodiment, the spatiotemporal global convolutional network includes a two-dimensional spatial convolution kernel and a one-dimensional temporal convolution kernel, and the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel are connected in series; the spatiotemporal global feature determination module 20 is further configured to sequentially pass the past segmentation results and the unsegmented video frame through the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel to obtain a spatiotemporally globally convolved video frame; add the past segmentation results, the unsegmented video frame, and the spatiotemporally globally convolved video frame to obtain an output result of the spatiotemporal global convolutional network. In one embodiment, the spatiotemporal global feature determination module 20 is further configured to modify the pixel values ​​of the background area in the past segmentation results; input the past segmentation results with the modified pixel values ​​and the unsegmented video frame into the spatiotemporal global convolutional network, and obtain the output result of the spatiotemporal global convolutional network.

[0130] The segmentation result determination module 30 is configured to generate a segmentation result corresponding to the video frame to be segmented based on the spatiotemporal global features and the past frame features. In one embodiment, the segmentation result determination module 30 is further configured to determine a downsampling feature of the video frame to be segmented; determine a concatenation feature of the spatiotemporal global features and the past frame features; input the downsampling feature and the concatenation feature into a decoding network and obtain an output result of the decoding network; and obtain a segmentation result corresponding to the video frame to be segmented based on the output result of the decoding network.

[0131] In one embodiment, after the segmentation result determination module 30, an updating module is further included, and the updating module is used to update the previous segmentation result with the segmentation result corresponding to the video frame to be segmented.

[0132] The specific implementation of the video object segmentation system of the present invention is basically the same as the embodiments of the above-mentioned video object segmentation method, and will not be repeated here.

[0133] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a target segmentation program for a video. When the target segmentation program for a video is executed by a processor, it implements the various steps of the target segmentation method for the video as described above and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0134] Since the storage medium provided in the embodiments of this application is the storage medium used to implement the method of the embodiments of this application, those skilled in the art will be able to understand the specific structure and variations of the storage medium based on the method described in the embodiments of this application, and therefore will not be described in detail here. All storage media used in the method of the embodiments of this application fall within the scope of protection to be provided by this application.

[0135] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0139] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0140] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0141] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for object segmentation in a video, characterized in that: The target segmentation method of the video includes: Determining past frame features associated with the video frame to be segmented based on past segmentation results; Inputting the past segmentation results and the unsegmented video frames that are temporally subsequent to the video frames to be segmented into a spatiotemporal global convolutional network, and obtaining an output result of the spatiotemporal global convolutional network; The output result is used as the input of the spatial convolutional network to obtain the spatiotemporal global features through the spatial convolutional network; Determining downsampling features of the video frame to be segmented; Determining a splicing feature of the spatiotemporal global feature and the past frame feature; Inputting the down-sampling features and the splicing features into a decoding network, and obtaining an output result of the decoding network; The segmentation result corresponding to the video frame to be segmented is obtained according to the output result of the decoding network.

2. The method for video object segmentation according to claim 1, wherein: The spatiotemporal global convolutional network includes a two-dimensional spatial convolution kernel and a one-dimensional temporal convolution kernel, and the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel are connected in series; the step of inputting the past segmentation result and the unsegmented video frame whose time sequence is after the video frame to be segmented into the spatiotemporal global convolutional network and obtaining the output result of the spatiotemporal global convolutional network includes: The past segmentation result and the unsegmented video frame are sequentially passed through the two-dimensional spatial convolution kernel and the one-dimensional temporal convolution kernel to obtain a video frame after spatiotemporal global convolution; The past segmentation results, the unsegmented video frames, and the video frames after the spatiotemporal global convolution are added together to obtain an output result of the spatiotemporal global convolutional network.

3. The method for object segmentation of a video according to claim 1, wherein: Inputting the past segmentation results and the unsegmented video frames that are temporally subsequent to the video frames to be segmented into the spatiotemporal global convolutional network, and obtaining the output results of the spatiotemporal global convolutional network includes: Changing the pixel value of the background area in the previous segmentation result; The past segmentation result whose pixel value has been changed and the unsegmented video frame are input into a spatiotemporal global convolutional network, and an output result of the spatiotemporal global convolutional network is obtained.

4. The method for object segmentation of a video according to claim 1, wherein: The step of determining the past frame features associated with the video frame to be segmented based on the past segmentation results includes: Determining index features of the video frame to be segmented; Determining a target past segmentation result that matches the index feature; The past frame features are determined based on the target past segmentation results.

5. The method for video object segmentation according to claim 4, wherein: After the step of obtaining the segmentation result corresponding to the video frame to be segmented according to the output result of the decoding network, the method further includes: The segmentation result corresponding to the video frame to be segmented is used to update the previous segmentation result.

6. A video object segmentation system, characterized in that: The object segmentation system of the video includes: A past frame feature determination module, configured to determine past frame features associated with the video frame to be segmented based on past segmentation results; a spatiotemporal global feature determination module, configured to input the past segmentation results and the unsegmented video frames that are temporally subsequent to the video frame to be segmented into a spatiotemporal global convolutional network, and obtain an output result of the spatiotemporal global convolutional network; use the output result as an input of a spatial domain convolutional network, and obtain spatiotemporal global features through the spatial domain convolutional network; The segmentation result determination module is used to determine the downsampling features of the video frame to be segmented; determine the splicing features of the spatiotemporal global features and the features of the past frames; input the downsampling features and the splicing features into a decoding network and obtain the output results of the decoding network; and obtain the segmentation result corresponding to the video frame to be segmented based on the output results of the decoding network.

7. An intelligent terminal, characterized in that: The intelligent terminal includes: a memory, a processor, and a video target segmentation program stored in the memory and executable on the processor. When the video target segmentation program is executed by the processor, the steps of the video target segmentation method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a video target segmentation program, and when the video target segmentation program is executed by a processor, the steps of the video target segmentation method according to any one of claims 1 to 5 are implemented.

Images