A video segmentation method, apparatus, computer device, and storage medium.
By encoding video frames and segmentation prompts, and combining temporal and spatial feature extraction, a segmentation mask image is generated, which solves the problem of poor video segmentation results in existing technologies and achieves more efficient and accurate video frame segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DOUYIN VISION CO LTD
- Filing Date
- 2023-06-27
- Publication Date
- 2026-05-26
Smart Images
Figure CN116797975B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more specifically, to a video segmentation method, apparatus, computer device, and storage medium. Background Technology
[0002] With the development of neural networks, more and more networks are being used for image segmentation, and the accuracy of image segmentation is getting higher and higher. Although video is composed of multiple image frames, the segmentation requirements for video are significantly different from those for image segmentation. Image segmentation only needs to consider the relationships between objects within the image itself, while video segmentation also needs to consider the relationships between different image frames. If image segmentation networks are directly applied to video segmentation, the segmentation results are poor. Summary of the Invention
[0003] This disclosure provides at least one video segmentation method, apparatus, computer device, and storage medium.
[0004] In a first aspect, embodiments of this disclosure provide a video segmentation method, including:
[0005] Obtain the video to be segmented, and the segmentation prompt information corresponding to the video to be segmented; wherein, the segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented;
[0006] The video frames of the video to be segmented are encoded using an image encoder to obtain a first encoded feature; and the segmentation prompt information is encoded using a prompt information encoder to obtain a second encoded feature.
[0007] The first encoded feature and the second encoded feature are input into the decoder, and temporal feature extraction is performed to obtain the first feature, and spatial feature extraction is performed to obtain the second feature.
[0008] Based on the first feature and the second feature, a segmentation mask image corresponding to the video frame is determined, and the video frame is segmented based on the segmentation mask image.
[0009] In one possible implementation, the video frames of the video to be segmented are video frames obtained after sampling and frame extraction; the encoding of the video frames of the video to be segmented based on the image encoder to obtain the first encoding feature includes:
[0010] For any given video frame, divide the video frame into multiple image blocks;
[0011] Determine the embedding representation vector of each image patch, and based on the embedding representation vector of each image patch, determine the embedding representation vector of the video frame.
[0012] The first coding feature is determined based on the embedding representation vector of each video frame.
[0013] In one possible implementation, inputting the first encoded feature and the second encoded feature into the decoder includes:
[0014] The first coding feature and the second coding feature are concatenated to obtain the third coding feature;
[0015] The third encoded feature is input into the decoder.
[0016] In one possible implementation, the decoder is used to extract the spatial features by means of:
[0017] The third encoded feature is input into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature.
[0018] In one possible implementation, the step of inputting the third encoded feature into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature includes:
[0019] The third encoded feature is normalized based on the first normalization layer to obtain the first normalized feature;
[0020] The first normalized feature is input into the fine-tuning structure layer to obtain the fine-tuning feature; and the first normalized feature is input into the multi-head self-attention module for feature extraction to obtain the intermediate feature;
[0021] The first normalized feature, the intermediate feature, and the fine-tuned feature are fused to obtain the first fused feature;
[0022] The first fused feature is normalized based on the second normalization layer to obtain the second normalized feature;
[0023] The second normalized feature is input into the multilayer perceptron to obtain the second feature.
[0024] In one possible implementation, the decoder is used to extract the temporal features by:
[0025] The third encoded feature is adjusted using the first channel to obtain the adjusted feature;
[0026] The adjusted features are input into an attention mechanism model containing a fine-tuned structural layer to obtain the third feature;
[0027] The third feature is adjusted using a second channel to obtain the first feature.
[0028] In one possible implementation, determining the segmentation mask image corresponding to the video frame based on the first feature and the second feature includes:
[0029] The first feature and the second feature are fused together to obtain the second fused feature;
[0030] Based on the second fusion feature, the segmentation mask image corresponding to the video frame is determined.
[0031] Secondly, embodiments of this disclosure also provide a video segmentation apparatus, characterized in that it includes:
[0032] The acquisition module is used to acquire the video to be segmented and the segmentation prompt information corresponding to the video to be segmented; wherein, the segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented;
[0033] The encoding module is used to encode the video frames of the video to be segmented based on the image encoder to obtain a first encoding feature; and to encode the segmentation prompt information based on the prompt information encoder to obtain a second encoding feature;
[0034] The decoding module is used to input the first encoded feature and the second encoded feature into the decoder, and to extract the first feature by extracting the temporal feature and the second feature by extracting the spatial feature, respectively.
[0035] The determining module is configured to determine the segmentation mask image corresponding to the video frame based on the first feature and the second feature, and to perform segmentation processing on the video frame based on the segmentation mask image.
[0036] Thirdly, embodiments of this disclosure also provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.
[0037] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation thereof.
[0038] The video segmentation method, apparatus, computer device, and storage medium provided in this disclosure, when segmenting a video to be segmented, after encoding the video frames and segmentation prompt information, can extract features from both the spatial and temporal domains. Then, based on the first feature obtained from the temporal feature extraction and the second feature obtained from the spatial feature extraction, a segmentation mask image corresponding to the video frame is determined, and segmentation processing is performed based on the segmentation mask image. Thus, because the temporal correlation between video frames is considered during feature extraction, the segmentation mask image obtained through this method yields more accurate segmentation results when segmenting the video frames.
[0039] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0040] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.
[0041] Figure 1 A schematic diagram of the architecture of a SAM model provided in an embodiment of this disclosure is shown;
[0042] Figure 2 A flowchart of a video segmentation method provided by an embodiment of this disclosure is shown;
[0043] Figure 3 A schematic diagram of the internal structure of a LoRA provided in an embodiment of this disclosure is shown;
[0044] Figure 4a The internal structure of a spatial feature extraction module provided in an embodiment of this disclosure is shown.
[0045] Figure 4b A schematic diagram of the decoder internal structure of the SAM model provided in this embodiment of the present disclosure is shown;
[0046] Figure 5 This diagram illustrates the internal structure of a time feature extraction module provided in an embodiment of the present disclosure.
[0047] Figure 6 This diagram illustrates the overall architecture of the video segmentation model provided in this embodiment.
[0048] Figure 7 A schematic diagram of the architecture of a video segmentation apparatus provided in an embodiment of this disclosure is shown;
[0049] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.
[0051] The video segmentation described in this disclosure can be understood as dividing a video into frames. However, unlike image segmentation, video frame segmentation needs to consider the relationships between frames. For example, a dynamic object in a video frame may be occluded in some video frames. If the video frame is segmented in isolation without considering the relationships between frames, the segmentation results of the dynamic object may differ in different video frames.
[0052] For example, if the video includes a moving cat, the cat may be segmented as a separate entity in some video frames, while in other video frames the cat may be occluded by other objects such as a table, thus segmenting the cat and other objects into a single entity.
[0053] To avoid this situation, related technologies have proposed segmenting the data in the temporal and spatial domains (such as two-stream networks). However, in this technology, optical flow information between frames is usually calculated, which is quite complex. Therefore, this method has low computational speed and accuracy.
[0054] Based on this, the video segmentation method, apparatus, computer device, and storage medium provided in this disclosure, when segmenting a video to be segmented, after encoding the video frames and segmentation prompt information, can extract features from the spatial and temporal domains respectively. Then, based on the first feature obtained from the temporal feature extraction and the second feature obtained from the spatial feature extraction, a segmentation mask image corresponding to the video frame is determined, and segmentation processing is performed based on the segmentation mask image. Thus, because the temporal correlation between video frames is considered during feature extraction, the segmentation mask image obtained by this method yields more accurate segmentation results when segmenting video frames. Furthermore, in the above method, optical flow information is not extracted during spatial feature extraction; instead, feature extraction is performed through prompt learning, resulting in higher computational speed and accuracy.
[0055] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0056] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0057] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0058] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0059] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0060] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0061] To facilitate understanding of this embodiment, a video segmentation method disclosed in this disclosure will first be described in detail. The execution entity of the video segmentation method provided in this disclosure is generally a server.
[0062] The video segmentation model described in this disclosure can be an improvement on the Segment Anything Model (SAM). For example... Figure 1 The diagram shows the architecture of the SAM model. Its input includes the image to be segmented and segmentation extraction information. After being input into the SAM model, the image to be segmented can first be input into the image encoder, then the segmentation prompt information can be input into the prompt encoder, and then input into the decoder for decoding to obtain the segmentation mask image. The image to be segmented is then segmented based on the segmentation mask image.
[0063] As can be seen from the above architecture, the SAM model only considers the features of the spatial domain when decoding, and only performs decoding in the spatial domain. Therefore, if the SAM model is directly applied to video segmentation, the segmentation effect is poor.
[0064] The video segmentation model disclosed herein is an improvement on the decoding module of the SAM model. The structural diagram of the specific decoding module in this application will be described below.
[0065] See Figure 2 The diagram shows a flowchart of a video segmentation method provided in an embodiment of this disclosure. The method includes steps 201 to 204, wherein:
[0066] Step 201: Obtain the video to be segmented and the segmentation prompt information corresponding to the video to be segmented; wherein, the segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented.
[0067] Step 202: Encode the video frames of the video to be segmented based on the image encoder to obtain the first encoding feature; and encode the segmentation prompt information based on the prompt information encoder to obtain the second encoding feature.
[0068] Step 203: Input the first encoded feature and the second encoded feature into the decoder, and perform temporal feature extraction to obtain the first feature and spatial feature extraction to obtain the second feature.
[0069] Step 204: Based on the first feature and the second feature, determine the segmentation mask image corresponding to the video frame, and perform segmentation processing on the video frame based on the segmentation mask image.
[0070] The following is a detailed explanation of the steps described above.
[0071] Regarding step 201,
[0072] The process of obtaining the video to be segmented can involve obtaining a video uploaded by the user or a video selected by the user from their local storage. Optionally, the length of the video to be segmented can be fixed, for example, 10 seconds.
[0073] The segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented. The segmentation requirements may include, for example, segmenting a specific entity in the video to be segmented, segmenting an entity with certain features in the video to be segmented, or segmenting an entity of a certain category in the video to be segmented.
[0074] The segmentation prompt information can be information input by the user, such as text information input by the user, or the location information of entities selected by the user in the video to be segmented.
[0075] In another possible implementation, the segmentation prompt information can be extracted from other segmented videos. For example, a user can simultaneously input the video to be segmented and a reference video, which is a segmented video, and then determine the segmentation prompt information from the segmentation results of the reference video.
[0076] Specifically, this disclosure does not limit the methods for obtaining other segmentation prompt information.
[0077] Regarding step 202,
[0078] Since the differences between adjacent video frames in the video to be segmented are small, in order to improve the segmentation speed, the video to be segmented can be sampled and frames extracted, and then the encoding and subsequent decoding processes in step 202 can be performed on the sampled and extracted video frames.
[0079] It should be noted that the steps of encoding by the image encoder and the prompt information encoder, and decoding by the decoder in this disclosure can all be completed by the video segmentation model. Step 202 and the subsequent steps can describe the operations performed internally by the video segmentation model.
[0080] In one possible implementation, when encoding the video frames (which may refer to the video frames after sampling and frame extraction) of the video to be segmented based on the image encoder, the following steps can be taken:
[0081] Step a1: For any given video frame, divide the video frame into multiple image blocks.
[0082] Step a2: Determine the embedding representation vector of each image block, and based on the embedding representation vector of each image block, determine the embedding representation vector of the video frame.
[0083] Step a3: Determine the first coding feature based on the embedding representation vector of each video frame.
[0084] Specifically, for any given video frame, when dividing the video frame into multiple image patches, the division can be based on a preset image patch size. For example, the image patch can be a square of length P, and the size of the video frame can be, for example, H*W. Then, the number of image patches after division, N = H*W / P. 2 .
[0085] Optionally, in order to ensure that the number of image blocks after division is a positive integer, the size of the video frame can be limited, for example, it can be a preset size, or the video frame can be processed into a preset size, wherein the preset size is an integer multiple of the area of the image block.
[0086] Video frame x∈R H*W*C Where H*W represents the size of the video frame and C represents the number of channels. The number of channels in the video frame is 3. After the video frame is segmented, the size of each image block is P*P*C. When determining the embedding representation vector of each image block, each image block can be mapped into a D-dimensional embedding representation vector through the linear mapping layer of the image encoder.
[0087] After determining the embedding representation vector of each image patch, the embedding representation vectors of each image patch can be combined to obtain the embedding representation vector of the video frame, which is x. p ∈R N*D , where N represents the number of image patches.
[0088] When determining the first coding feature based on the embedding representation vectors of each video frame, the embedding representation vectors of each video frame can be concatenated according to the positional order of each video frame in the video to be segmented to obtain the first coding feature.
[0089] In addition, to distinguish the embedding representation vectors of each video frame, a learnable flag vector x can be concatenated after the embedding representation vector of each video frame. class That is, when determining the first coding feature, it is based on the combined vector x0 = [x] of the flag bit vector and the embedded representation vector. class ;x p ]∈R (N+1)*DIf the number of video frames is T, then the dimension of the first encoded feature vector is z0∈R. T*(N+1)*D .
[0090] The encoding process of the prompt information encoder for the segmented prompt information is similar to the encoding process in the SAM model, and will not be described in detail here.
[0091] The above-mentioned image encoder has the same encoding process and parameters as the image encoder in the SAM model, and the prompt information encoder has the same encoding process and parameters as the prompt information encoder in the SAM model. In this disclosure, the video segmentation model can be regarded as a fine-tuning model of the SAM model.
[0092] Regarding step 203,
[0093] The decoder can be a unit for performing temporal feature extraction and spatial feature extraction. In one possible implementation, the decoder may include a temporal feature extraction module and a spatial feature extraction module, which are used for performing temporal feature extraction and spatial feature extraction, respectively.
[0094] Optionally, after inputting the first encoded feature and the second encoded feature into the decoder, the first encoded feature and the second encoded feature can be concatenated (e.g., by using a connect operation) to obtain a third encoded feature. Then, the third encoded feature is input into the temporal feature extraction module and the spatial feature extraction module of the decoder for feature extraction.
[0095] After inputting the third encoded feature into the decoder, the main operations include the following two aspects:
[0096] 1. Spatial feature extraction.
[0097] When performing spatial feature extraction, it is necessary to consider the relationships between entities within a video frame. Therefore, a multi-head attention mechanism can be used to obtain more feature information for spatial feature extraction.
[0098] Optionally, the third encoded feature can be input into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature.
[0099] Specifically, after inputting the third encoded feature into an attention mechanism model containing a fine-tuning structure layer, the following operations can be performed to obtain the second feature, including:
[0100] Step b1: Normalize the third encoded feature based on the first normalization layer to obtain the first normalized feature;
[0101] Step b2: Input the first normalized feature into the fine-tuning structure layer to obtain the fine-tuning feature; and input the first normalized feature into the multi-head self-attention module for feature extraction to obtain the intermediate feature;
[0102] Step b3: Fuse the first normalized feature, the intermediate feature, and the fine-tuned feature to obtain the first fused feature;
[0103] Step b4: Normalize the first fused feature based on the second normalization layer to obtain the second normalized feature;
[0104] Step b5: Input the second normalized feature into the multilayer perceptron to obtain the second feature.
[0105] The first normalization layer and the second normalization layer can both be LayerNorm layers, and the parameter values of the first normalization layer and the second normalization layer can be different.
[0106] The fine-tuning structure layer can be, for example, a low-rank adaptive (LoRA) layer, the internal structure of which is as follows: Figure 3 As shown, it includes two fully connected layers and one non-linear activation layer (GELU). The first fully connected layer (FC Down) is a linear layer with a parameter dimension of M1*M2. Its purpose is to downsample the parameters and reduce the number of model channels. The second fully connected layer (FC Up) is also a linear layer with a parameter dimension of M2*M1. Its purpose is to upsample the parameters and restore the number of model channels.
[0107] In the above parameter dimensions, M1 >> M2. The purpose is that after downsampling, even if M2 is small, enough information can still be captured because M1 >> M2, making it more inclined to adapt to more weight matrices rather than adapting to a single type of weight with a larger rank.
[0108] The internal structure of the spatial feature extraction module is as follows: Figure 4a As shown, it includes a fine-tuning structure layer LoRA, which is located next to the multi-head self-attention (MSA) module and is fused with the output of the multi-head self-attention module.
[0109] The overall process is as follows: First, the third encoded feature is input into the first normalization layer (LayerNorm) for normalization processing to obtain the first normalized feature. Then, the first normalized feature is input into the LoRA layer and MSA. The outputs of LoRA and MSA are then fused with the third encoded feature. After fusion, the feature is input into the second normalization layer (LayerNorm). The output of the second normalization layer is then input into the multilayer perceptron (MLP). Finally, the output of the MLP is fused with the feature input into the second normalization layer to obtain the second feature.
[0110] Compared to the SAM model, refer to Figure 4b The diagram shows the internal structure of the decoder for the SAM model, and... Figure 4a As can be seen from the comparison, the spatial feature extraction module in this disclosure adds a LoRA layer to the decoder of the SAM model.
[0111] Here, the LoRA layer is placed directly next to the MSA layer without skip connections. The purpose of this is to make the final video segmentation model closer to the SAM model in order to maintain model performance.
[0112] 2. Temporal feature extraction.
[0113] When extracting temporal features, it is necessary to consider the relationships between frames. For example, temporal feature extraction can be performed through the following steps:
[0114] Step c1: Adjust the third encoding feature using the first channel to obtain the adjusted feature.
[0115] Step c2: Input the adjusted feature into the attention mechanism model containing the fine-tuning structure layer to obtain the third feature.
[0116] Step c3: Adjust the third feature using the second channel to obtain the first feature.
[0117] The channel adjustment can be implemented, for example, through a reshape operation. The third encoding feature is z0∈R. T*(N+1)*D After the first channel adjustment, the adjusted feature can be z'0∈R (N+1)*T*D .
[0118] Here, if the third encoded feature is directly input into MSA and LoRA (the same as the spatial feature extraction module mentioned above) without first channel adjustment, the subsequent MSA and other operations will be performed in the (N+1) dimension, that is, in the image patch dimension. However, after first channel adjustment, since the dimension of the adjusted feature has changed, the subsequent MSA and other operations will be performed in the T dimension, where T represents the number of video frames. Therefore, the relationship between T video frames can be learned through this method.
[0119] After inputting the adjusted feature into the attention mechanism model containing the fine-tuning structure layer to obtain the third feature, the dimension of the third feature is the same as the dimension of the adjusted feature. However, since the second feature and the first feature need to be fused, in order to maintain the consistency of the feature dimension, the third feature needs to be adjusted in the second channel. The feature dimension of the adjusted second feature is the same as the feature dimension of the first feature.
[0120] Exemplary reference Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a time feature extraction module provided in an embodiment of the present disclosure, including a reshape operation for adjusting the first channel and a reshape operation for adjusting the second channel, with the remaining attention mechanism model including a fine-tuning structure layer. Figure 4a and Figure 4b The model structure described herein is the same and will not be repeated here.
[0121] It should be noted that although the attention mechanism models containing fine-tuning structural layers in the temporal feature extraction module and the spatial feature extraction module have the same model structure, the specific model parameter values may be different.
[0122] Regarding step 204,
[0123] In one possible implementation, when determining the segmentation mask image corresponding to the video frame based on the first feature and the second feature, the first feature and the second feature can be fused first to obtain a second fused feature; then, the segmentation mask image corresponding to the video frame can be determined based on the second fused feature.
[0124] For example, when determining the segmentation mask image corresponding to the video frame based on the second fusion feature, the second fusion feature can be input into a multilayer perceptron to determine the segmentation mask image.
[0125] Here, if the video frame is a video frame after frame sampling, the number of segmentation mask images is the same as the number after the video frame.
[0126] If the video frame is a video frame after frame sampling, and the video segmentation task is to segment each video frame in the video to be segmented, then it is also necessary to determine the segmentation mask image of the remaining video frames that have not been sampled through the segmentation mask image.
[0127] The video frame that has been extracted is designated as the first video frame, and the video frame that has not been extracted is designated as the second video frame. The segmentation mask image corresponding to the first video frame is determined through steps 201 to 204 above. The first video frame and the second video frame are combined to form the video to be segmented. The segmentation mask image of the second video frame can be determined based on the segmentation mask image of the first video frame, and the second video frame is segmented based on the segmentation mask image of the second video frame.
[0128] The segmentation mask image can be a binary image. In the segmentation mask image, the pixel position with a value of 1 corresponds to the pixel point in the video frame that needs to be segmented from the video frame, and the pixel position with a value of 0 corresponds to the pixel point in the video frame that does not need to be segmented.
[0129] Since the differences between adjacent N video frames are small, the segmentation mask image of the second video frame can be determined by the difference information between the video frames.
[0130] For example, if the a-th video frame is the first video frame mentioned above, then when determining the segmentation mask image of the (a+k)-th video frame (which is the second video frame), the pixel displacement information of the entity to be segmented between the a-th and (a+k)-th video frames can be determined. Then, based on the pixel displacement information, the positions of pixels with a value of 1 in the a-th video frame are adjusted, and the adjusted segmentation mask image is the segmentation mask image of the (a+k)-th video frame.
[0131] For any second video frame, when determining the segmentation mask image corresponding to that video frame, we can first determine the first video frame that is closest to that video frame, and then determine the segmentation mask image corresponding to that video frame based on the determined segmentation mask image of the first video frame.
[0132] If there are two closest first video frames, the similarity between these two first video frames and the second video frame can be calculated respectively. Based on the segmentation mask image of the first video frame with higher similarity, the segmentation mask image of the second video frame can be determined.
[0133] Alternatively, for any second video frame, when determining the segmentation mask image corresponding to that video frame, the first video frame with the highest similarity to that video frame can be directly determined, and then the segmentation mask image corresponding to that video frame can be determined based on the segmentation mask image of the determined first video frame.
[0134] In another possible implementation, when determining the segmentation mask image of the second video frame, optical flow information between the second video frame and the first video frame can be calculated, and then the segmentation mask image of the first video frame can be adjusted based on the optical flow information to determine the segmentation mask image of the second video frame.
[0135] Here, the first video frame involved in calculating optical flow information can be the first video frame adjacent to the second video frame. Although optical flow information is calculated, the complexity of optical flow calculation is low because the number of video frames involved in calculating optical flow information is small.
[0136] The overall architecture of the video segmentation model described above will be explained below with reference to the accompanying diagrams. (Refer to...) Figure 6 The diagram shown is an overall architecture diagram of a video segmentation model provided in this embodiment of the present disclosure. It mainly includes an image encoder, a prompt information encoder, and a decoder. The image encoder encodes the image (i.e., video frame) and then inputs it into the decoder. The prompt information encoder encodes the segmentation prompt information and then inputs it into the decoder. The decoder mainly includes two parts: a spatial feature extraction module and a temporal feature extraction module. The spatial feature extraction module and the temporal feature extraction module can respectively extract spatial features and temporal features from the input features. After the inputs of the two branches are fused, they can be used to determine the MASK, i.e., the segmentation mask image.
[0137] The video segmentation model described above can be trained as a fine-tuning process, requiring relatively little data. The specific training process may include the following steps:
[0138] Step d 1: Obtain the sample video and segmentation hints, as well as the labeled mask image corresponding to the sample video.
[0139] The labeled mask image can be manually labeled or a mask image labeled in other ways to represent the segmentation result, and the segmentation result is consistent with the segmentation prompt information.
[0140] Step d2: Input the sample video and segmentation prompt information into the video segmentation model to be trained, and determine the segmentation mask image predicted by the video segmentation model.
[0141] Step d3: Based on the predicted segmentation mask image and the labeled mask image, determine the loss value for this training, and adjust the parameter values of the video segmentation model to be trained based on the loss value.
[0142] The video segmentation method provided in this disclosure, when segmenting a video to be segmented, encodes the video frames and segmentation prompts, and then extracts features from both the spatial and temporal domains. Based on the first feature obtained from the temporal feature extraction and the second feature obtained from the spatial feature extraction, a segmentation mask image corresponding to each video frame is determined, and segmentation processing is performed based on the segmentation mask image. Thus, because the temporal correlation between video frames is considered during feature extraction, the segmentation mask image obtained through this method yields more accurate segmentation results when segmenting video frames.
[0143] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0144] Based on the same inventive concept, this disclosure also provides a video segmentation device corresponding to the video segmentation method. Since the principle of the device in this disclosure for solving the problem is similar to that of the video segmentation method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0145] Reference Figure 7 The diagram shown is an architectural schematic of a video segmentation device provided in an embodiment of this disclosure. The device includes: an acquisition module 701, an encoding module 702, a decoding module 703, and a determination module 704; wherein,
[0146] The acquisition module 701 is used to acquire the video to be segmented and the segmentation prompt information corresponding to the video to be segmented; wherein, the segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented;
[0147] The encoding module 702 is used to encode the video frames of the video to be segmented based on the image encoder to obtain a first encoding feature; and to encode the segmentation prompt information based on the prompt information encoder to obtain a second encoding feature;
[0148] The decoding module 703 is used to input the first encoded feature and the second encoded feature into the decoder, and to extract the first feature by extracting the temporal feature and the second feature by extracting the spatial feature.
[0149] The determining module 704 is used to determine the segmentation mask image corresponding to the video frame based on the first feature and the second feature, and to perform segmentation processing on the video frame based on the segmentation mask image.
[0150] In one possible implementation, the video frames of the video to be segmented are video frames obtained after sampling and frame extraction; the encoding module 702, when encoding the video frames of the video to be segmented based on the image encoder to obtain the first encoding feature, is used to:
[0151] For any given video frame, divide the video frame into multiple image blocks;
[0152] Determine the embedding representation vector of each image patch, and based on the embedding representation vector of each image patch, determine the embedding representation vector of the video frame.
[0153] The first coding feature is determined based on the embedding representation vector of each video frame.
[0154] In one possible implementation, the decoding module 703, when inputting the first encoded feature and the second encoded feature into the decoder, is used to:
[0155] The first coding feature and the second coding feature are concatenated to obtain the third coding feature;
[0156] The third encoded feature is input into the decoder.
[0157] In one possible implementation, the decoding module 703 is used to extract the spatial features by means of the following method:
[0158] The third encoded feature is input into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature.
[0159] In one possible implementation, the decoding module 703, when inputting the third encoded feature into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature, is used to:
[0160] The third encoded feature is normalized based on the first normalization layer to obtain the first normalized feature;
[0161] The first normalized feature is input into the fine-tuning structure layer to obtain the fine-tuning feature; and the first normalized feature is input into the multi-head self-attention module for feature extraction to obtain the intermediate feature;
[0162] The first normalized feature, the intermediate feature, and the fine-tuned feature are fused to obtain the first fused feature;
[0163] The first fused feature is normalized based on the second normalization layer to obtain the second normalized feature;
[0164] The second normalized feature is input into the multilayer perceptron to obtain the second feature.
[0165] In one possible implementation, the decoding module 703 is used to perform the time feature extraction by the following method:
[0166] The third encoded feature is adjusted using the first channel to obtain the adjusted feature;
[0167] The adjusted features are input into an attention mechanism model containing a fine-tuned structural layer to obtain the third feature;
[0168] The third feature is adjusted using a second channel to obtain the first feature.
[0169] In one possible implementation, the determining module 704, when determining the segmentation mask image corresponding to the video frame based on the first feature and the second feature, is used to:
[0170] The first feature and the second feature are fused together to obtain the second fused feature;
[0171] Based on the second fusion feature, the segmentation mask image corresponding to the video frame is determined.
[0172] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0173] Based on the same technical concept, this disclosure also provides a computer device. (See also...) Figure 8 The diagram shows the structure of a computer device 800 provided in this embodiment, including a processor 801, a memory 802, and a bus 803. The memory 802 stores execution instructions and includes main memory 8021 and external memory 8022. The main memory 8021, also called internal memory, is used to temporarily store computational data in the processor 801 and data exchanged with external memory 8022 such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the main memory 8021. When the computer device 800 is running, the processor 801 and the memory 802 communicate through the bus 803, causing the processor 801 to execute the following instructions:
[0174] Obtain the video to be segmented, and the segmentation prompt information corresponding to the video to be segmented; wherein, the segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented;
[0175] The video frames of the video to be segmented are encoded using an image encoder to obtain a first encoded feature; and the segmentation prompt information is encoded using a prompt information encoder to obtain a second encoded feature.
[0176] The first encoded feature and the second encoded feature are input into the decoder, and temporal feature extraction is performed to obtain the first feature, and spatial feature extraction is performed to obtain the second feature.
[0177] Based on the first feature and the second feature, a segmentation mask image corresponding to the video frame is determined, and the video frame is segmented based on the segmentation mask image.
[0178] In one possible implementation, the instructions executed by processor 801 specify that the video frames of the video to be segmented are video frames obtained after sampling and frame extraction; the step of encoding the video frames of the video to be segmented based on an image encoder to obtain a first encoded feature includes:
[0179] For any given video frame, divide the video frame into multiple image blocks;
[0180] Determine the embedding representation vector of each image patch, and based on the embedding representation vector of each image patch, determine the embedding representation vector of the video frame.
[0181] The first coding feature is determined based on the embedding representation vector of each video frame.
[0182] In one possible implementation, the instruction executed by processor 801, which involves inputting the first encoded feature and the second encoded feature into the decoder, includes:
[0183] The first coding feature and the second coding feature are concatenated to obtain the third coding feature;
[0184] The third encoded feature is input into the decoder.
[0185] In one possible implementation, in the instructions executed by the processor 801, the decoder is configured to perform the spatial feature extraction by the following method:
[0186] The third encoded feature is input into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature.
[0187] In one possible implementation, the instruction executed by processor 801, which involves inputting the third encoded feature into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature, includes:
[0188] The third encoded feature is normalized based on the first normalization layer to obtain the first normalized feature;
[0189] The first normalized feature is input into the fine-tuning structure layer to obtain the fine-tuning feature; and the first normalized feature is input into the multi-head self-attention module for feature extraction to obtain the intermediate feature;
[0190] The first normalized feature, the intermediate feature, and the fine-tuned feature are fused to obtain the first fused feature;
[0191] The first fused feature is normalized based on the second normalization layer to obtain the second normalized feature;
[0192] The second normalized feature is input into the multilayer perceptron to obtain the second feature.
[0193] In one possible implementation, in the instructions executed by the processor 801, the decoder is configured to perform the temporal feature extraction by the following method:
[0194] The third encoded feature is adjusted using the first channel to obtain the adjusted feature;
[0195] The adjusted features are input into an attention mechanism model containing a fine-tuned structural layer to obtain the third feature;
[0196] The third feature is adjusted using a second channel to obtain the first feature.
[0197] In one possible implementation, the instructions executed by processor 801, wherein determining the segmentation mask image corresponding to the video frame based on the first feature and the second feature, includes:
[0198] The first feature and the second feature are fused together to obtain the second fused feature;
[0199] Based on the second fusion feature, the segmentation mask image corresponding to the video frame is determined.
[0200] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video segmentation method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0201] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the video segmentation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0202] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0203] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0205] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0206] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0207] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.
Claims
1. A video segmentation method, characterized in that, include: Obtain the video to be segmented, and the segmentation prompt information corresponding to the video to be segmented; wherein, the segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented; The video frames of the video to be segmented are encoded using an image encoder to obtain a first encoded feature; and the segmentation prompt information is encoded using a prompt information encoder to obtain a second encoded feature. The first encoded feature and the second encoded feature are input into the decoder, where temporal feature extraction is performed to obtain the first feature, and spatial feature extraction is performed to obtain the second feature; and Based on the first feature and the second feature, a segmentation mask image corresponding to the video frame is determined, and the video frame is segmented based on the segmentation mask image; The step of inputting the first encoded feature and the second encoded feature into the decoder includes: concatenating the first encoded feature and the second encoded feature to obtain a third encoded feature; and inputting the third encoded feature into the decoder; and The extraction of the time features includes: extracting the time features based on the third encoded features.
2. The method according to claim 1, characterized in that, The video frames of the video to be segmented are video frames obtained after sampling and frame extraction; the encoding of the video frames of the video to be segmented based on the image encoder to obtain the first encoding features includes: For any given video frame, divide the video frame into multiple image blocks; Determine the embedding representation vector of each image patch, and based on the embedding representation vector of each image patch, determine the embedding representation vector of the video frame. The first coding feature is determined based on the embedding representation vector of each video frame.
3. The method according to claim 1, characterized in that, The decoder is used to extract the spatial features using the following method: The third encoded feature is input into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature.
4. The method according to claim 3, characterized in that, The step of inputting the third encoded feature into an attention mechanism model containing a fine-tuning structure layer to obtain the second feature includes: The third encoded feature is normalized based on the first normalization layer to obtain the first normalized feature; The first normalized feature is input into the fine-tuning structure layer to obtain the fine-tuning feature; and the first normalized feature is input into the multi-head self-attention module for feature extraction to obtain the intermediate feature; The first normalized feature, the intermediate feature, and the fine-tuned feature are fused to obtain the first fused feature; The first fused feature is normalized based on the second normalization layer to obtain the second normalized feature; The second normalized feature is input into the multilayer perceptron to obtain the second feature.
5. The method according to claim 1, characterized in that, The decoder is used to extract the temporal features using the following method: The third encoded feature is adjusted using the first channel to obtain the adjusted feature; The adjusted features are input into an attention mechanism model containing a fine-tuned structure layer to obtain the third feature; The third feature is adjusted using a second channel to obtain the first feature.
6. The method according to claim 1, characterized in that, The step of determining the segmentation mask image corresponding to the video frame based on the first feature and the second feature includes: The first feature and the second feature are fused together to obtain the second fused feature; Based on the second fusion feature, the segmentation mask image corresponding to the video frame is determined.
7. A video segmentation device, characterized in that, include: The acquisition module is used to acquire the video to be segmented and the segmentation prompt information corresponding to the video to be segmented; wherein, the segmentation prompt information is used to characterize the segmentation requirements of the video to be segmented; The encoding module is used to encode the video frames of the video to be segmented based on the image encoder to obtain a first encoding feature; and to encode the segmentation prompt information based on the prompt information encoder to obtain a second encoding feature; The decoding module is used to input the first encoded feature and the second encoded feature into the decoder, and to perform temporal feature extraction to obtain the first feature, and spatial feature extraction to obtain the second feature; and The determining module is configured to determine the segmentation mask image corresponding to the video frame based on the first feature and the second feature, and to perform segmentation processing on the video frame based on the segmentation mask image; The step of inputting the first encoded feature and the second encoded feature into the decoder includes: concatenating the first encoded feature and the second encoded feature to obtain a third encoded feature; and inputting the third encoded feature into the decoder; and The extraction of the time features includes: extracting the time features based on the third encoded features.
8. A computer device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the video segmentation method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the video segmentation method as described in any one of claims 1 to 6.