Video segmentation methods and video editing methods

By acquiring the reference mask frame and the target frame, and combining the object's motion features to construct the current segmentation prompt information, the video segmentation process is dynamically adjusted. This solves the problem of insufficient accuracy of existing video segmentation models in complex scenes, and achieves higher segmentation accuracy and robustness.

CN123118657APending Publication Date: 2026-11-17ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510618773.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-13
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing video segmentation models suffer from insufficient accuracy when handling interactive video segmentation in complex scenes, resulting in poor accuracy of video segmentation results.

Method used

By acquiring the reference mask frame and target frame of the original video, a preliminary segmentation is performed using a video segmentation model. The current segmentation prompt information is constructed by combining the object motion features of the current frame, and the segmentation process is dynamically adjusted to mitigate error accumulation and enhance adaptability to complex scenes.

Benefits of technology

It improves the accuracy of video segmentation results, enhances the adaptability and robustness of video segmentation models to complex scene changes, and ensures accurate object location and segmentation precision.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118657A_ABST
    Figure CN123118657A_ABST
Patent Text Reader

Abstract

The embodiments of the present specification provide a video segmentation method and a video editing method, wherein the video segmentation method comprises: obtaining a reference mask frame and a target frame of an original video; inputting the reference mask frame and a starting frame into a video segmentation model to obtain a mask frame corresponding to the starting frame; for a current frame, inputting current segmentation prompt information and the current frame into the video segmentation model to obtain a mask frame corresponding to the current frame, wherein the current frame is any frame in the target frame except the starting frame, and the current segmentation prompt information is constructed based on object motion features of segmented frames; and determining a video segmentation result of the original video according to the mask frames corresponding to the target frames. Since the current segmentation prompt information is constructed based on the object motion features of the segmented frames, dynamic adjustment of the current segmentation prompt information is realized as the video frames advance, the adaptability and robustness of the video segmentation model to complex scene changes are enhanced, and the accuracy of the video segmentation result is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments in this specification relate to the field of video processing technology, and in particular to a video segmentation method and a video editing method. Background Technology

[0002] With the development of computer vision technology, video segmentation has become a key task in visual understanding, widely used in various fields such as video editing, content generation, and autonomous driving. Since the release of video segmentation models, interactive video segmentation, which performs accurate object tracking and segmentation based on user-provided visual interaction intentions (such as points, lines, masks, etc.), has rapidly become a research hotspot.

[0003] Currently, the video segmentation intent and the video to be segmented are typically input into a video segmentation model to obtain the segmentation result. However, due to limitations in training data and model capabilities, video segmentation models still suffer from insufficient accuracy when handling interactive video segmentation in some complex scenarios, resulting in poor video segmentation accuracy. Therefore, a more accurate video segmentation scheme is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a video segmentation method. One or more embodiments of this specification also relate to a video editing method, an information processing method based on a video segmentation model, a task platform, a video segmentation device, a video editing device, an information processing device based on a video segmentation model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a video segmentation method is provided, comprising:

[0006] Obtain the reference mask frame and target frame of the original video. The original video includes a reference frame and a target frame. The reference mask frame is used to reflect the region position of the object in the reference frame.

[0007] Input the reference mask frame and the start frame into the video segmentation model to obtain the mask frame corresponding to the start frame, where the start frame is the frame in the target frame that is adjacent to the reference frame;

[0008] For the current frame, the current segmentation prompt information and the current frame are input into the video segmentation model to obtain the mask frame corresponding to the current frame. The current frame is any frame in the target frame except the starting frame. The current segmentation prompt information is constructed based on the object motion features of the segmented frames.

[0009] Based on the mask frame corresponding to the target frame, determine the video segmentation result of the original video.

[0010] According to a second aspect of the embodiments of this specification, a video editing method is provided, comprising:

[0011] Obtain the original video and the video segmentation result of the original video, wherein the video segmentation result is obtained based on the video segmentation method;

[0012] Based on the video segmentation results, object editing is performed on the original video to obtain the video editing result.

[0013] According to a third aspect of the embodiments of this specification, an information processing method based on a video segmentation model is provided, applied to a task platform, comprising:

[0014] Receive model requests sent by terminal devices;

[0015] Based on the model request, a target video segmentation model is determined from multiple video segmentation models. This target video segmentation model is used in the processing of the video segmentation method.

[0016] According to a fourth aspect of the embodiments of this specification, a task platform is provided, including a request interface and a response unit;

[0017] The request interface is used to receive model requests sent by terminal devices. The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters.

[0018] The response unit is used to determine the target video segmentation model from multiple video segmentation models based on the model request. The target video segmentation model is used in the processing of the video segmentation method.

[0019] According to a fifth aspect of the embodiments of this specification, a video segmentation apparatus is provided, comprising:

[0020] The first acquisition module is configured to acquire the reference mask frame and the target frame of the original video, wherein the original video includes the reference frame and the target frame, and the reference mask frame is used to reflect the regional position of the object in the reference frame.

[0021] The first input module is configured to input the reference mask frame and the start frame into the video segmentation model to obtain the mask frame corresponding to the start frame, wherein the start frame is the frame in the target frame that is adjacent to the reference frame;

[0022] The second input module is configured to input the current segmentation prompt information and the current frame into the video segmentation model for the current frame to obtain the mask frame corresponding to the current frame. The current frame is any frame in the target frame other than the starting frame. The current segmentation prompt information is constructed based on the object motion features of the segmented frames.

[0023] The first determining module is configured to determine the video segmentation result of the original video based on the mask frame corresponding to the target frame.

[0024] According to a sixth aspect of the embodiments of this specification, a video editing apparatus is provided, comprising:

[0025] The second acquisition module is configured to acquire the original video and the video segmentation result of the original video, wherein the video segmentation result is obtained based on the video segmentation method.

[0026] The editing module is configured to perform object editing on the original video based on the video segmentation results, and obtain the video editing results.

[0027] According to a seventh aspect of the embodiments of this specification, an information processing apparatus based on a video segmentation model is provided, applied to a task platform, comprising:

[0028] The receiving module is configured to receive model requests sent by the terminal device;

[0029] The second determination module is configured to determine a target video segmentation model from multiple video segmentation models based on a model request, wherein the target video segmentation model is used in the processing of the video segmentation method.

[0030] According to an eighth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0031] Memory and processor;

[0032] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0033] According to page nine of the embodiments of this specification, an electronic device is provided, comprising:

[0034] The memory and processor are connected via a bus;

[0035] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0036] According to a tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0037] According to an eleventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0038] This specification provides a video segmentation method according to one embodiment, comprising: acquiring a reference mask frame and a target frame of an original video, wherein the original video includes a reference frame and a target frame, and the reference mask frame is used to reflect the regional position of an object in the reference frame; inputting the reference mask frame and a start frame into a video segmentation model to obtain a mask frame corresponding to the start frame, wherein the start frame is a frame in the target frame adjacent to the reference frame; for the current frame, inputting current segmentation prompt information and the current frame into the video segmentation model to obtain a mask frame corresponding to the current frame, wherein the current frame is any frame in the target frame other than the start frame, and the current segmentation prompt information is constructed based on the object motion features of already segmented frames; and determining the video segmentation result of the original video based on the mask frame corresponding to the target frame. By using a reference mask frame as a starting point and combining it with the start frame for preliminary segmentation, the position of the object can be located more accurately, providing a reliable basis for the segmentation of subsequent frames. Since the current segmentation prompts are constructed based on the object motion features of the segmented frames, the current segmentation prompts are dynamically adjusted as the video frames progress, making them better adaptable to changes, movements, or even partial occlusion or temporary disappearance of objects. This alleviates the problem of error accumulation when processing consecutive frames, enhances the adaptability and robustness of the video segmentation model to complex scene changes, and further improves the accuracy of video segmentation results. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a video segmentation method provided in one embodiment of this specification;

[0040] Figure 2 This is a schematic diagram illustrating a candidate mask frame filtering process provided in one embodiment of this specification;

[0041] Figure 3 This is a flowchart illustrating the processing procedure of an encoder training method provided in one embodiment of this specification.

[0042] Figure 4 This is a flowchart illustrating a video editing method provided in one embodiment of this specification;

[0043] Figure 5 This is an architecture diagram of a video segmentation system provided in one embodiment of this specification;

[0044] Figure 6 This is a flowchart illustrating an information processing method based on a video segmentation model, provided in one embodiment of this specification.

[0045] Figure 7 This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0046] Figure 8 This is a schematic diagram of the structure of a video segmentation device provided in one embodiment of this specification;

[0047] Figure 9 This is a schematic diagram of the structure of a video editing device provided in one embodiment of this specification;

[0048] Figure 10 This is a schematic diagram of the structure of an information processing device based on a video segmentation model provided in one embodiment of this specification;

[0049] Figure 11 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0050] Figure 12 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0051] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0052] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0053] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0054] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0055] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0056] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0057] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0058] Video object segmentation is the task of segmenting regions in a video that contain objects of a predefined category.

[0059] Interactive video segmentation refers to the task of automatically segmenting the target region in the first frame of a video into subsequent frames, given the target region to be segmented.

[0060] Interactive segmentation models for images and videos (SAM, Segment Anything Model): Designed to segment any object in images and videos, regardless of whether that object appeared during training. SAM achieves flexible and accurate segmentation results by using a novel cue engineering mechanism that allows users to specify regions or objects of interest through simple interactions (such as clicking or drawing lines). SAM possesses strong generalization capabilities, handling a wide variety of scenes and object categories, performing excellently even in complex or unfamiliar scenarios. SAM2 is an upgraded version of SAM.

[0061] Sparse-level point motion is a computer vision technique used to estimate the motion of a small number of salient feature points in an image sequence (such as video frames). These feature points are typically unique pixel regions in an image, such as corners, edges, or other easily identifiable structures. Sparse optical flow algorithms track and analyze the motion of only these selected feature points, without considering all pixels in the entire image.

[0062] Dense-level optical flow is a computer vision technique used to estimate the motion vector of each pixel in an image sequence. Unlike sparse optical flow, dense optical flow covers all pixels of the entire image, generating a complete two-dimensional vector field that describes the motion of every point in the scene.

[0063] Memory encoder: A module used to extract features from the segmented object region and transform them into memory features.

[0064] This specification provides a video segmentation method, and also relates to a video editing method, an information processing method based on a video segmentation model, a task platform, a video segmentation device, a video editing device, an information processing device based on a video segmentation model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0065] See Figure 1 , Figure 1 This specification illustrates a flowchart of a video segmentation method according to an embodiment, which specifically includes the following steps:

[0066] Step 102: Obtain the reference mask frame and target frame of the original video. The original video includes a reference frame and a target frame. The reference mask frame is used to reflect the region position of the object in the reference frame.

[0067] It's important to note that raw video refers to unsegmented video footage. Raw video consists of a series of consecutive frames (i.e., reference frames and target frames), which include the image data for object segmentation. The goal of segmenting raw video is to determine the mask frames corresponding to each frame. The video segmentation process can be viewed as a video object segmentation process, where the target is the object. The mask frames reflect the location and outline of the object in the raw video. Raw video can be from different scenes, such as animated videos in an animation scene, autonomous driving videos in a traffic scene, surgical videos in a medical image analysis scene, and so on.

[0068] An object refers to a specific entity or region within the original video that needs to be identified, tracked, and segmented. An object can be any element with clearly defined boundaries, shape, location, and other characteristics, and can be distinguished individually across a series of video frames. Examples of objects include virtual characters, people, vehicles, animals, objects, natural elements, etc. In the original video, objects may temporarily disappear or be occluded in some frames.

[0069] The location of an object refers to the precise position and extent of an object within a frame of the original video. This location information is typically represented by a mask that explicitly marks the object's position, shape, and size within that frame. For example, the area containing the object can be labeled as foreground (e.g., white), while the background area is labeled as background (e.g., black).

[0070] A reference mask frame is a frame containing a precise label (mask) of the region where a specified object is located. The mask in the reference mask frame can provide information and guidance for object segmentation in subsequent frames. Reference mask frames are typically obtained by the user labeling the object region location on a reference frame. Therefore, the video segmentation method proposed in the embodiments of this specification supports interaction with the user using reference mask frames and can be considered an interactive video object segmentation method.

[0071] A reference frame is the frame in the original video that corresponds to the reference mask frame. The reference frame is usually the first frame of the original video. The location of objects is not marked in the reference frame, and the video segmentation process does not require object segmentation operations on the reference frame.

[0072] Target frames refer to the set of video frames from which object segmentation operations need to be performed. These frames may contain one or more objects that need to be identified and segmented. For example, suppose the original video consists of four video frames, namely video frame 1, video frame 2, video frame 3, and video frame 4. Video frame 1 is a reference mask frame with the object region locations already marked, and video frames 2, 3, and 4 are the target frames.

[0073] In practical applications, there are various ways to obtain the reference mask frame and target frame of the original video. The specific method chosen depends on the actual situation, and the embodiments in this specification do not impose any limitations on this. In one possible implementation of this specification, the reference mask frame and target frame of the original video can be received from the user through a client. In another possible implementation of this specification, the reference mask frame and target frame of the original video can be read from other data acquisition devices or a database.

[0074] Step 104: Input the reference mask frame and the start frame into the video segmentation model to obtain the mask frame corresponding to the start frame, where the start frame is the frame in the target frame that is adjacent to the reference frame.

[0075] It's important to note that the starting frame refers to the frame in the target frame adjacent to the reference frame, such as the frame following the reference frame. The starting frame serves as the basis for initial segmentation using the reference mask frame, allowing the segmentation results to be extended to subsequent frames. If the reference mask frame is the first frame of the original video, the starting frame is the second frame of the original video. Referring to the example above, if the reference mask frame is video frame 1, and the target frames are video frames 2, 3, and 4, then the starting frame is video frame 2. The mask frame is the resulting frame obtained after processing by the video segmentation model. The mask frame accurately marks the position, shape, and outline of an object within its corresponding starting frame; that is, the mask frame corresponding to the starting frame reflects the region of the object within that starting frame.

[0076] A video segmentation model is a neural network model capable of segmenting objects within a video. It generates a series of mask frames by identifying and extracting the regions containing objects in video frames. Video segmentation models can be large-scale visual models, such as SAM, generative adversarial networks, variational autoencoders, etc., selected based on the specific circumstances. This specification does not impose any limitations on these selections in the embodiments.

[0077] In practical applications, there are multiple ways to input the reference mask frame and the start frame into the video segmentation model to obtain the mask frame corresponding to the start frame. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, the reference mask frame and the start frame can be directly input into the video segmentation model to obtain the mask frame corresponding to the start frame. In another possible implementation, model prompts, the reference mask frame, and the start frame can be input into the video segmentation model to obtain the mask frame corresponding to the start frame. Model prompts refer to information used to guide the video segmentation model in segmenting the input frames. Model prompts can be general prompts for each frame, such as "You are a professional video segmenter; please process the input video frames and output the corresponding mask frames."

[0078] Step 106: For the current frame, input the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame. The current frame is any frame in the target frame other than the starting frame. The current segmentation prompt information is constructed based on the object motion features of the segmented frames.

[0079] It should be noted that the current frame refers to any unsegmented frame in the target frame, excluding the starting frame. Referring to the example above, if the reference mask frame is video frame 1, the target frames are video frames 2, 3, and 4, and the starting frame is video frame 2, then the current frame can be video frame 3 or video frame 4. The mask frame corresponding to the current frame reflects the region location of the object within that frame.

[0080] Since video contains both temporal and spatial information, to ensure the accuracy of video segmentation, the current frame can be determined sequentially from the target frames, excluding the starting frame. Referring to the example above, if the reference frame is video frame 1, the target frames are video frames 2, 3, and 4, and the starting frame is video frame 2, then the current frame could be video frame 3.

[0081] A segmented frame refers to a video frame that has already undergone object segmentation. Since video segmentation of the original video is typically based on video timing, a segmented frame can also be understood as a frame preceding the current frame. Referring to the example above, if the current frame is video frame 3, then the segmented frames include video frames 1 and 2; if the current frame is video frame 4, then the segmented frames include video frames 1, 2, and 3. Optionally, a segmented frame can be all video frames that have undergone object segmentation, or it can be a subset of video frames that have undergone object segmentation.

[0082] Current segmentation cue information refers to the segmentation cue information for the current frame. Different current frames correspond to different current segmentation cue information. Current segmentation cue information can be points, lines, bounding boxes, mask regions, or other forms of data. Current segmentation cue information provides the video segmentation model with cross-frame motion features of objects, helping the model to more accurately locate and segment objects in the current frame. For example, current segmentation cue information can indicate the object's direction of movement, speed, or predict its next possible location. Object motion features are used to describe the changes in an object between consecutive segmented frames. Object motion features include, but are not limited to, changes in the object's position, direction of movement, and changes in the object's shape.

[0083] In practical applications, the implementation method of "inputting the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame" can refer to the above implementation method of "inputting the reference mask frame and the starting frame into the video segmentation model to obtain the mask frame corresponding to the starting frame". The embodiments in this specification will not be described in detail.

[0084] Step 108: Determine the video segmentation result of the original video based on the mask frame corresponding to the target frame.

[0085] It should be noted that the video segmentation result can be a sequence of target mask frames consisting only of the mask frames corresponding to the target frame, or a sequence of video mask frames of the original video composed of both the reference mask frames and the mask frames corresponding to the target frame. The mask frame sequence can record the position and shape of objects in the original video as they change over time.

[0086] In practical applications, taking the video segmentation result as the target mask frame sequence as an example, there are multiple ways to determine the video segmentation result of the original video based on the mask frames corresponding to the target frames. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, each mask frame can be labeled with the temporal information of the corresponding target frame in the original video, and all mask frames corresponding to the target frames can be added to an empty sequence to obtain the video segmentation result. In another possible implementation, based on the temporal information of each mask frame, all mask frames corresponding to the target frames can be sequentially added to an empty sequence to obtain the video segmentation result.

[0087] By applying the scheme of the embodiments in this specification, using a reference mask frame as a starting point and combining it with the initial frame for preliminary segmentation, the position of the object can be located more accurately, providing a reliable basis for the segmentation of subsequent frames. Since the current segmentation prompt information is constructed based on the object motion features of already segmented frames, it achieves dynamic adjustment of the current segmentation prompt information as the video frames progress, better adapting to changes, movements, or even partial occlusion or temporary disappearance of objects. This alleviates the error accumulation problem during continuous frame processing, enhances the adaptability and robustness of the video segmentation model to complex scene changes, and further improves the accuracy of the video segmentation results.

[0088] In one optional embodiment of this specification, in order to improve the segmentation effect of the video segmentation model on the current frame, an auxiliary mask frame can be introduced to guide the model's processing on the basis of the current segmentation prompt information. That is, before inputting the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame, the following steps may be included:

[0089] Select auxiliary mask frames from the segmentation mask frames corresponding to the already segmented frames;

[0090] For the current frame, inputting the current segmentation cue information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame may include the following steps:

[0091] For the current frame, the current segmentation prompt information, the current frame, and the auxiliary mask frame are input into the video segmentation model to obtain the mask frame corresponding to the current frame.

[0092] It should be noted that the segmentation mask frame refers to the mask frame corresponding to the already segmented frame. The auxiliary mask frame refers to one or more keyframes selected from the segmentation mask frames that provide additional guidance for subsequent frame segmentation. Auxiliary mask frames typically contain precise annotations of objects, accurately reflecting their regional locations. They can be used to correct prediction errors in video segmentation models, enhance segmentation accuracy, and help video segmentation models better handle complex scenes (such as occlusion, transient disappearances, etc.).

[0093] In practical applications, there are multiple ways to select auxiliary mask frames from the segmentation mask frames corresponding to the segmented frames. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this. In the first possible implementation of this specification, auxiliary mask frames can be selected from the segmentation mask frames corresponding to the segmented frames based on the time dimension. In the second possible implementation of this specification, auxiliary mask frames can be selected from the segmentation mask frames corresponding to the segmented frames based on the spatial dimension. In the third possible implementation of this specification, auxiliary mask frames can be selected from the segmentation mask frames corresponding to the segmented frames based on both the time and spatial dimensions.

[0094] By applying the solutions in the embodiments of this specification and introducing high-quality auxiliary mask frames, the cumulative errors that may occur in the video segmentation model during long-term segmentation can be corrected, helping the video segmentation model adapt to the dynamic changes of the object and significantly improving the accuracy of the segmentation results.

[0095] In this embodiment, to ensure that the video segmentation model maintains a reliable and effective memory structure throughout the video segmentation process, a dynamic spatiotemporal memory selection mechanism is proposed. Based on the temporal and spatial dimensions, auxiliary mask frames are selected from the segmentation mask frames corresponding to the already segmented frames. After selection in both time and space dimensions, pixel regions with low confidence are filtered out, improving the reliability and robustness of the memorized auxiliary mask frames, which is beneficial for improving the accurate segmentation effect of subsequent frames. The process will now be described in detail.

[0096] In one optional embodiment of this specification, the process of filtering out auxiliary mask frames from the segmented mask frames corresponding to the segmented frames may include the following steps:

[0097] Based on the temporal information of the segmented frames, candidate mask frames are selected from the segmented mask frames corresponding to the segmented frames.

[0098] Based on the pixel information of the candidate mask frames, auxiliary mask frames are selected from the candidate mask frames.

[0099] It should be noted that temporal information refers to the temporal order of the segmented frames. Pixel information refers to the data of each pixel in the candidate mask frame, including but not limited to color values ​​and transparency. Pixel information can be used to measure the quality of the candidate mask frame in a spatial dimension. A candidate mask frame is a keyframe selected from the segmented mask frames based on the temporal information of the segmented frames, which can reflect the dynamic changes of the object or is of high quality.

[0100] In practical applications, there are various ways to select candidate mask frames from the corresponding segmented mask frames based on the temporal information of the segmented frames. The specific method should be selected according to the actual situation, and the embodiments in this specification do not limit this. In the first possible implementation of this specification, candidate mask frames can be uniformly selected from the segmented frames according to a certain time interval based on the temporal information of the segmented frames, ensuring that there are corresponding candidate mask frames for each time period. In the second possible implementation of this specification, multiple segmented mask frames adjacent to the current frame can be determined as candidate mask frames based on the temporal information of the segmented frames. In the third possible implementation of this specification, segmented mask frames corresponding to key time points can be found based on the temporal information of the segmented frames, and these segmented mask frames can be determined as candidate mask frames.

[0101] Furthermore, there are multiple ways to select auxiliary mask frames from the candidate mask frames based on their pixel information. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the pixel parameters of the candidate mask frames can be determined based on their pixel information; auxiliary mask frames can then be selected from the candidate mask frames based on these pixel parameters. In another possible implementation, candidate mask frames that are complete and occupy a reasonable area can be selected as auxiliary mask frames based on their pixel information.

[0102] The scheme implemented in this specification involves first selecting a set of keyframes that reflect the dynamic changes of the object from all segmented mask frames based on the temporal information of the segmented frames. Then, based on the pixel information of the candidate mask frames, frames that provide high-quality segmentation references are further selected as auxiliary mask frames for subsequent video segmentation. This multi-dimensional selection strategy allows for more refined management of the memorized segmentation mask frames, ensuring that the video segmentation model consistently uses high-quality object motion features during video segmentation, thereby improving the accuracy and robustness of the segmentation.

[0103] In one optional embodiment of this specification, the process of selecting candidate mask frames from the segmented mask frames corresponding to the segmented frames based on the timing information of the segmented frames may include the following steps:

[0104] Based on the timing information of the segmented frames, multiple initial mask frames adjacent to the current frame are extracted from the segmentation mask frames corresponding to the segmented frames.

[0105] Based on the quality index of the segmented mask frame, a replacement mask frame is selected from the segmented mask frame. The quality index of the replacement mask frame is greater than the quality index of the other frames in the segmented mask frame.

[0106] Based on the quality index of the initial mask frame, a replacement mask frame is selected from multiple initial mask frames. The quality index of the replacement mask frame is lower than the quality index of the frames other than the replacement mask frame in the initial mask frame. The number of replacement mask frames is the same as the number of replacement mask frames.

[0107] By replacing the mask frame in multiple initial mask frames, candidate mask frames are obtained.

[0108] It should be noted that the initial mask frame is extracted from the segmentation mask frames corresponding to the already segmented frames and consists of multiple mask frames adjacent to the current frame. Quality metrics are used to measure the quality of the segmented mask frames. Quality metrics include, but are not limited to, the Intersection over Union (IoU) and occlusion confidence. IoU measures the segmentation accuracy of the video segmentation model; a higher IoU value indicates higher segmentation accuracy. Occlusion confidence refers to the video segmentation model's confidence level in estimating whether an object is occluded and the degree of occlusion; it is used to evaluate the segmentation performance of the video segmentation model in the presence of occlusion.

[0109] Replacement mask frames are high-quality mask frames selected from all segmented mask frames, with a quality index higher than all other segmented mask frames except for the replacement mask frames. These frames will be used to replace the lower-quality initial mask frames. Mask frames to be replaced are low-quality mask frames selected from multiple initial mask frames, with a quality index lower than all other initial mask frames except for the replacement mask frames. These frames will be replaced by replacement mask frames. Candidate mask frames are the set of mask frames obtained after the replacement operation, where the low-quality initial mask frames have been replaced by high-quality replacement mask frames.

[0110] For example, see Figure 2 , Figure 2This diagram illustrates a candidate mask frame filtering process according to an embodiment of this specification. Specifically, it includes: extracting multiple initial mask frames adjacent to the current frame from the segmented mask frames corresponding to the segmented frames based on the timing information of the segmented frames; specifically, extracting six frames adjacent to the current frame from the segmented mask frames corresponding to the segmented frames based on the timing information of the segmented frames as initial mask frames; and filtering replacement mask frames from the segmented mask frames according to their quality indicators. First, extracting thirty-two frames adjacent to the current frame from the segmented mask frames corresponding to the segmented frames based on the timing information of the segmented frames; then, sampling these thirty-two frames at one-frame intervals to obtain sixteen frames. Then, the target region intersection-over-union (IoU) and occlusion confidence are predicted from the sixteen sampled frames. Based on the thresholds for IoU and occlusion confidence, six frames are selected from the sixteen frames. These six frames are relatively reliable compared to the other unselected frames. Finally, a second selection is performed based on a higher threshold, extracting four replacement mask frames from the six selected frames. These four replacement mask frames are relatively reliable compared to the other unselected frames. When selecting replacement mask frames from multiple initial mask frames based on their quality metrics, the target region IoU and occlusion confidence can be predicted from the six initial mask frames. Based on the thresholds for IoU and occlusion confidence, four replacement mask frames are selected from the six initial mask frames. Finally, the four replacement mask frames are used to replace the four unreplaced mask frames from the six initial mask frames, resulting in six candidate mask frames.

[0111] By applying the solutions described in this specification, high-quality frames can be selected from existing segmentation mask frames as auxiliary information to optimize the processing of the video segmentation model and ensure that the segmentation results are more accurate and reliable.

[0112] In one optional embodiment of this specification, the process of selecting auxiliary mask frames from candidate mask frames based on pixel information may include the following steps:

[0113] Based on the pixel information of the candidate mask frame, determine the pixel index of the candidate mask frame;

[0114] Based on pixel metrics, auxiliary mask frames are selected from the candidate mask frames, wherein the pixel metrics of the auxiliary mask frames are greater than the pixel metrics of the candidate mask frames other than the auxiliary mask frames.

[0115] It should be noted that pixel metrics are used to evaluate the quality of each candidate mask frame. Pixel metrics include, but are not limited to, model confidence, edge sharpness, object integrity, and color and texture consistency. Model confidence measures the level of confidence the video segmentation model has in the candidate mask frame. Edge sharpness measures whether the mask edges are smooth, continuous, and free of blur or noise. Object integrity checks the area proportion of the target object in the mask, ensuring that the object occupies a reasonable area and has intact boundaries. Color and texture consistency compares the consistency of color distribution and texture features in the foreground region within the mask.

[0116] For example, taking model confidence as an example, when determining the pixel metrics of candidate mask frames based on their pixel information, the confidence level of the model in segmenting the current frame corresponding to the candidate mask frame can be parsed from the output of the video segmentation model based on the pixel information of the candidate mask frames. During video segmentation, the video segmentation model classifies each pixel in each frame (e.g., determining whether a pixel belongs to an object). Model confidence provides information about the reliability of these classification decisions. For example, if a pixel is marked as part of an object by the video segmentation model with high confidence, it means that the video segmentation model considers this classification highly reliable; conversely, if the confidence is low, there may be uncertainty or error. When selecting auxiliary mask frames, frames with higher model confidence can be selected as auxiliary mask frames because these frames are more likely to contain high-quality, accurate segmentation results.

[0117] The scheme implemented in this specification calculates the pixel index of the candidate mask frame based on its pixel information, and selects the frame with the highest pixel index as the auxiliary mask frame, ensuring that the selected auxiliary mask frame is a high-quality frame, thereby providing more accurate guidance and support in subsequent video segmentation.

[0118] In this embodiment of the specification, to provide cross-frame segmentation object motion features to the video segmentation model, a motion-guided prompting model is proposed. Object motion features are extracted through sparse-level point movement and / or dense-level optical flow. Then, based on the mask region of the previous frame, the object region position in subsequent frames is estimated using these motion features. Based on the estimated object region position, multiple sparse points within the region and the overall bounding box of the region are extracted and used as prompts for the video segmentation model for object segmentation. The process will now be described in detail.

[0119] In one optional embodiment of this specification, the construction method of the current segmentation prompt information of the current frame is described. That is, before inputting the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame, the following steps may be included:

[0120] For the current frame, obtain the segmentation mask frame corresponding to the segmented frame, where the segmentation mask frame is used to reflect the region position of the object in the segmented frame;

[0121] Based on the segmented mask frame, determine the position information of the object in the segmented frame;

[0122] Based on the location information, extract the object motion features in the segmented frames;

[0123] Based on the object's motion characteristics, predict the object's position information in the current frame and obtain the current segmentation prompt information.

[0124] It should be noted that the location information includes, but is not limited to, the object's bounding box, center point coordinates, and outline, used to accurately describe the object's specific location and extent within the segmented frame. The bounding box is typically a rectangle enclosing all pixels of the object. The center point coordinates refer to the coordinates of the object's geometric center within the frame. Based on the location information, methods for extracting the object's motion features within the segmented frame include, but are not limited to, sparse-level point movement techniques and dense-level optical flow techniques. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on these methods in the embodiments.

[0125] In practical applications, there are multiple ways to determine the position information of an object within a segmented frame based on the segmented mask frame. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, a rectangular region containing all foreground pixels of the object can be calculated. The object's contour is then found by contour detection within this rectangular region, and a bounding box that closely approximates the contour is calculated as the position information. In another possible implementation, for a binarized segmented mask frame, the center point coordinates can be obtained by calculating the average of all foreground pixel coordinates. If the bounding box has already been calculated, its center can be directly used as the object's center point coordinates.

[0126] Furthermore, based on the object's motion characteristics, there are multiple ways to predict the object's position information in the current frame and obtain the current segmentation prompt information. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, the object's position information in the current frame can be predicted by linear extrapolation based on the object's motion characteristics in previously segmented frames. Specifically, the displacement vector between adjacent frames can be calculated. Assuming the object moves at a constant speed, adding the displacement vector to the position in the last frame yields the position information in the current frame, thus obtaining the current segmentation prompt information. In another possible implementation, the object's historical motion trajectory across multiple frames can be used to predict the object's position information in the current frame through curve fitting, thereby obtaining the current segmentation prompt information.

[0127] The scheme implemented in this specification extracts object motion features from the segmentation mask frame corresponding to the segmented frame, and uses these motion features to predict the object's position in the current frame, generating current segmentation hints. This explicitly extracts object motion features and uses them for object region estimation and segmentation hints, effectively solving the problem of insufficient object tracking capabilities in complex scenes. Especially in cases of target occlusion or temporary disappearance, the object's position can be predicted through motion features, resulting in more accurate segmentation. Furthermore, the current segmentation hints provide the video segmentation model with cross-frame object motion features, helping the model fully utilize these features during segmentation to more accurately locate and segment objects in the current frame. This not only improves object tracking accuracy but also enhances the robustness of the video segmentation model in complex scenes.

[0128] In one optional embodiment of this specification, the location information includes at least one of key point location information and pixel location information; the extraction of object motion features in the segmented frame based on the location information may include the following steps:

[0129] Input the key point location information into the sparse motion encoder to obtain the object motion features in the segmented frames;

[0130] The pixel location information is input into the dense motion encoder to obtain the object motion features in the segmented frames.

[0131] It's important to note that keypoint location information refers to the coordinates of the key points representing the object's salient features. Keypoints are typically corner points, edge points, or other feature points that represent the object's shape and structure. For example, keypoints could be feature points on a human face such as eyes, nose, and mouth, or joints in a human skeleton. Pixel location information refers to the coordinates of all foreground pixels (i.e., pixels belonging to the object). For example, pixels could be the outline of the target object or all pixels within its bounding box.

[0132] A sparse motion encoder is a model used to process keypoint location information. It extracts object motion features by analyzing changes in keypoints across consecutive frames. Sparse motion encoders can capture the main motion patterns of objects, such as translation, rotation, and scaling.

[0133] A dense motion encoder is a model used to process pixel position information. It extracts object motion features by analyzing the changes in all foreground pixels across consecutive frames. Dense motion encoders are used to extract dense optical flow information of objects, capturing more detailed motion information, including local deformations and texture changes.

[0134] In practical applications, a suitable encoder can be selected from sparse motion encoders and dense motion encoders based on specific needs, or a combination of both can be used to achieve a balance between computational efficiency and accuracy. When combining sparse and dense motion encoders to obtain the object motion features in a segmented frame, the object motion features obtained by the sparse and dense motion encoders can be weighted separately to obtain the final object motion features.

[0135] By applying the schemes of the embodiments in this specification, by inputting key point location information into a sparse motion encoder and / or inputting pixel location information into a dense motion encoder, the motion features of objects in segmented frames can be extracted from different levels, thereby significantly reducing the error of the model during segmentation, enhancing the resistance of the video segmentation model to interference factors such as noise and occlusion, and improving segmentation accuracy and robustness.

[0136] It is worth noting that the video segmentation scheme proposed in the embodiments of this specification is an interactive video segmentation scheme based on motion perception prior and spatiotemporal memory filtering mechanism. By extracting the motion features of objects and incorporating them into the video segmentation model, motion perception prior is realized. The model's memory structure is dynamically filtered from both temporal and spatial dimensions, which significantly improves the accuracy of object segmentation. Especially when dealing with complex scenarios such as target occlusion and temporary disappearance, more stable object tracking and segmentation can be achieved.

[0137] In one optional embodiment of this specification, the training method of sparse motion encoders and / or dense motion encoders is described. That is, before extracting the object motion features in the segmented frames based on the position information, the following steps may be included:

[0138] Obtain the sample video and the sample video segmentation results;

[0139] Based on the sample video and the sample video segmentation results, the parameters of the sparse motion encoder and / or dense motion encoder are adjusted to obtain the trained sparse motion encoder and / or the trained sparse motion encoder.

[0140] It should be noted that the sample videos are video data used to train the sparse motion encoder and / or dense motion encoder. The sample videos contain information about the positional changes of the sample objects, such as translation, rotation, scaling, and deformation. The sample videos typically have diverse content (e.g., different scenes, lighting conditions, occlusion situations, etc.) to ensure that the sparse motion encoder and / or dense motion encoder can adapt to various complex situations. The sample video segmentation result is a sequence of sample mask frames corresponding to the sample video. This sequence includes the sample mask frames corresponding to each sample frame in the sample video, and the masks in these frames indicate the position and region of the sample objects within those frames.

[0141] In practical applications, there are various ways to obtain sample videos and their segmentation results, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on these methods. In one possible implementation, the sample video and its segmentation results can be received from a user via a client. In another possible implementation, the sample video and its segmentation results can be read from other data acquisition devices or databases.

[0142] Furthermore, based on the sample video and the sample video segmentation results, the parameters of the sparse motion encoder and / or dense motion encoder are adjusted to obtain the trained sparse motion encoder and / or the trained dense motion encoder. There are multiple ways to do this, and the specific method is selected according to the actual situation. This specification does not limit the specific method used in the embodiments. In one possible implementation of this specification, the same process as the video segmentation method can be used to predict the predicted video segmentation result corresponding to the sample video. Based on the sample video segmentation result and the predicted video segmentation result, the parameters of the sparse motion encoder and dense motion encoder are adjusted to obtain the trained sparse motion encoder and the trained dense motion encoder. In another possible implementation of this specification, multiple sample mask frames can be determined based on the sample video segmentation results. Using the same process as the video segmentation method, the predicted mask frames corresponding to each sample video frame are predicted, resulting in multiple sets of sample mask frame-predicted mask frame pairs. A subset of these sample mask frame-predicted mask frame pairs is selected, and the parameters of the sparse motion encoder and dense motion encoder are adjusted using these subsets to obtain the trained sparse motion encoder and dense motion encoder. The selection method for the subset of sample mask frame-predicted mask frame pairs can be random selection or uniform selection based on temporal information, depending on the actual situation. This specification does not impose any limitations on the implementation.

[0143] By applying the schemes of the embodiments in this specification, sample videos and sample video segmentation results are obtained, and the parameters of sparse motion encoders and / or dense motion encoders are adjusted, enabling sparse motion encoders and / or dense motion encoders to capture the motion features of objects more accurately. Furthermore, by utilizing diverse sample videos, sparse motion encoders and / or dense motion encoders can learn object motion patterns in different scenarios, enhancing their generalization ability and enabling them to adapt to various complex scenarios.

[0144] Next, the process of adjusting the parameters of the sparse motion encoder and / or dense motion encoder based on the sample video and sample video segmentation results to obtain the trained sparse motion encoder and / or trained sparse motion encoder is explained. See [link to documentation]. Figure 3 , Figure 3 This document illustrates a flowchart of an encoder training method according to an embodiment of this specification. It is assumed that the sample video includes multiple sample video frames (sample video frame 1, sample video frame 2 to sample video frame N). Sample video frame 1 is the first frame of the sample video, and the region location of the sample object is already marked on sample video frame 1; therefore, sample video frame 1 can be called sample mask frame 1. The encoder includes a sparse motion encoder and a dense motion encoder. The encoder training process will now be described in detail:

[0145] For sample video frame 2: Input sample mask frame 1 and sample video frame 2 into the video segmentation model to obtain sample mask frame 2 of sample video frame 2; Input sample mask frame 1 and sample mask frame 2 into the memory encoder to obtain sample mask feature 1 and sample mask feature 2;

[0146] For sample video frame 3: Determine the sample position information 1 of the sample object in sample mask frame 1 and the sample position information 2 of the sample object in sample mask frame 2; input sample position information 1 and sample position information 2 into a sparse motion encoder for key point feature extraction to obtain the sample object motion feature 1 (sparse motion feature) in sample mask frames 1 and 2; input sample position information 1 and sample position information 2 into a dense motion encoder for dense optical flow feature extraction to obtain the sample object motion feature 2 (dense motion feature) in sample mask frames 1 and 2; based on the sample object motion feature 1 and sample object motion feature 2, perform position prediction to obtain the sample object's position in sample mask frames 1 and 2. The predicted position information in sample video frame 3; based on the predicted position information, construct sample segmentation prompt information for sample video frame 3; based on the temporal information of the segmented frames (sample video frame 1 and sample video frame 2 in the current round), select sample candidate mask frames from the segmentation mask frames corresponding to the segmented frames (sample mask frame 1 and sample mask frame 2 in the current round); based on the pixel information of the sample candidate mask frames, select sample auxiliary mask frames from the sample candidate mask frames; extract the sample auxiliary coding features of the sample auxiliary coding frames from the memory encoder; input the sample auxiliary coding features, sample video frame 3, and sample segmentation prompt information for sample video frame 3 into the video segmentation model to obtain sample mask frame 3 of sample video frame 3;

[0147] It should be noted that for any frame from sample video frame 4 to sample video frame N, the segmentation method is the same as that for sample video frame 3, and will not be repeated in the embodiments of this specification. After all sample video frames of the sample video are segmented, a prediction mask frame sequence consisting of N sample mask frames can be obtained. Then, the sample video segmentation result (sample mask frame sequence) of the sample video can be obtained. Based on the prediction mask frame sequence and the sample mask frame sequence, the parameters of the sparse motion encoder and the dense motion encoder are adjusted to obtain the trained sparse motion encoder and the trained dense motion encoder.

[0148] In practical applications, when adjusting the parameters of sparse motion encoders and dense motion encoders, the parameters of memory encoders and video segmentation models are frozen and remain unchanged; only the parameters of sparse motion encoders and dense motion encoders are adjusted.

[0149] The video segmentation results obtained by the video segmentation method proposed in the embodiments of this specification can be used in subsequent video processing tasks. Video processing tasks include video generation tasks, video editing tasks, etc. Taking video editing tasks as an example, the following description, in conjunction with the appendix... Figure 4 This paper further explains the application of video segmentation results obtained through video segmentation methods in video editing tasks. Figure 4This specification illustrates a flowchart of a video editing method according to an embodiment, which may specifically include the following steps:

[0150] Step 402: Obtain the original video and the video segmentation result of the original video, wherein the video segmentation result is obtained based on the video segmentation method.

[0151] Step 404: Based on the video segmentation results, perform object editing on the original video to obtain the video editing results.

[0152] It's important to note that object editing refers to the process of performing specific operations on objects in the original video based on the video segmentation results. These operations include object deletion, object replacement, adding effects, and color adjustment. Object deletion involves removing an object from the original video and filling the gap using background restoration techniques. Object replacement involves replacing an object with another object while maintaining motion consistency. Adding effects involves adding visual effects (such as highlighting, blurring, and transparency) to an object. Color adjustment involves modifying the object's color attributes (such as brightness, contrast, and hue). The video segmentation results reflect the regional location of objects in the original video. The video editing result is a new video generated after object editing, where the objects have been modified in a specified manner.

[0153] In practical applications, there are various ways to obtain the original video and its video segmentation results. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on these methods. In one possible implementation of this specification, the original video and its video segmentation results can be read from other data acquisition devices or databases. In another possible implementation of this specification, the original video can be obtained and then... Figure 1 The video segmentation method shown is used to segment the original video, resulting in the video segmentation result.

[0154] Furthermore, based on the video segmentation results, object editing is performed on the original video to obtain the video editing result. There are various methods, and the specific method chosen depends on the actual situation. This specification does not limit the specific methods used in the embodiments. In one possible implementation, the video segmentation result and the original video can be input into a video editing model to obtain the video editing result. The video editing model can be a large-scale model in the video domain, or a deep learning model trained based on sample videos, sample video segmentation results of sample videos, and sample video editing results. In another possible implementation, traditional image processing techniques (such as image inpainting, edge detection, color correction, etc.) can be used to edit the original video based on the video segmentation result to obtain the video editing result.

[0155] The solution implemented in this specification has a high accuracy rate in video segmentation results obtained by the video segmentation method. Therefore, the location of an object in the original video can be accurately determined based on the video segmentation results, thereby achieving precise object editing of the original video and improving the accuracy of the video editing results.

[0156] Considering the large number of parameters in video segmentation models, sparse motion encoders, or dense motion encoders, and the limited computing resources of the client, the video segmentation method proposed in the embodiments of this specification can be applied to, for example... Figure 5 The video segmentation system shown is not limited to this. See also... Figure 5 , Figure 5 This specification illustrates an architecture diagram of a video segmentation system provided in one embodiment of the specification. The video segmentation system may include a client 502 and a server 504.

[0157] Client 502 is used to send the reference mask frame and target frame of the original video to server 504. The original video includes the reference frame and the target frame. The reference mask frame is used to reflect the regional position of the object in the reference frame.

[0158] Server 504 is used to input the reference mask frame and the start frame into the video segmentation model to obtain the mask frame corresponding to the start frame, where the start frame is the frame in the target frame adjacent to the reference frame; for the current frame, it inputs the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame, where the current frame is any frame in the target frame other than the start frame, and the current segmentation prompt information is constructed based on the object motion features of the segmented frames; based on the mask frame corresponding to the target frame, it determines the video segmentation result of the original video; and sends the video segmentation result of the original video to client 502.

[0159] Client 502 is also used to receive the video segmentation results of the original video sent by server 504.

[0160] like Figure 5 As shown, the video segmentation model is deployed in server 504. Server 504 can connect to one or more clients 502 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client 502 can include, but is not limited to, smartphones, tablets, laptops, PDAs, personal computers (PCs), smart home devices, and in-vehicle devices. Client 502 can also interact with users through a graphical user interface to invoke the video segmentation model, thereby implementing the video segmentation method provided in the embodiments of this specification.

[0161] It is worth noting that the video segmentation method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the video segmentation model, the client can also have similar functions to the server to execute the video segmentation method provided in the embodiments of this specification. In other embodiments, the video segmentation method provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0162] See Figure 6 , Figure 6 This specification illustrates a flowchart of an information processing method based on a video segmentation model, according to an embodiment of the present invention. This method is applied to a task platform and specifically includes the following steps:

[0163] Step 602: Receive the model request sent by the terminal device.

[0164] Step 604: Based on the model request, determine the target video segmentation model from multiple video segmentation models, whereby the target video segmentation model is used in the processing of the video segmentation method.

[0165] It should be noted that multiple video segmentation models can be video segmentation models with different model specifications and parameters, adapted to different scenarios. The target video segmentation model is the video segmentation model suitable for the target scenario. The target scenario can be different scenarios, such as animation video segmentation scenarios and traffic video segmentation scenarios. The model request includes the scene identifier of the target scenario, the scene input data of the target scenario, and at least one of the model specification parameters. Based on the model request, there are multiple ways to determine the target video segmentation model from multiple video segmentation models, and the specific method is selected according to the actual situation. This specification does not limit this in any way. In the first possible implementation of this specification, the corresponding target video segmentation model can be searched from the video segmentation models included in the first model library based on the scene identifier included in the model request; in the second possible implementation of this specification, the target video segmentation model can be trained and obtained based on the scene input data included in the model request; in the third possible implementation of this specification, the corresponding target video segmentation model can be selected based on the model specification parameters in the model request.

[0166] For example, based on the scene identifier of the target scene, at least one pre-trained video segmentation model can be searched from the first model library. Then, based on the model specification parameters, an intermediate video segmentation model can be selected from the at least one video segmentation model. Finally, based on the scene input data of the target scene, the selected intermediate video segmentation model can be trained to obtain a target video segmentation model suitable for user needs.

[0167] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target video segmentation models, realizing personalized model services and providing users with an efficient, flexible and easy-to-use model service solution, thereby improving user experience.

[0168] In one optional embodiment of this specification, determining the target video segmentation model from multiple video segmentation models based on a model request may include the following steps:

[0169] If the model request includes a scene identifier of the target scene, a target video segmentation model suitable for the target scene is searched from the first model library based on the scene identifier. The first model library stores multiple video segmentation models suitable for different scenes.

[0170] When the model request includes scene input data of the target scene, the video segmentation model to be trained is determined from multiple video segmentation models that are suitable for the target scene, and the video segmentation model to be trained is trained based on the scene input data to obtain the target video segmentation model.

[0171] If the model request includes model specification parameters, the target video segmentation model corresponding to the model specification parameters is searched from the second model library. The second model library stores multiple video segmentation models corresponding to different model specification parameters.

[0172] It should be noted that scene identifiers are unique or specific labels used to distinguish different scenes. The First Model Library is a database for storing and managing various pre-trained deep learning models. Multiple video segmentation models adapted to different scenes cover various application scenarios and needs. The First Model Library allows users to select the appropriate video segmentation model according to their needs, or directly call the appropriate video segmentation model to execute the video segmentation method through the application programming interface. The multiple video segmentation models adapted to different scenes are various models stored in the First Model Library suitable for different scenarios, each optimized for a specific application scenario. For example, based on the scene identifier "animation video segmentation" of the target dialogue scene, a target video segmentation model suitable for the animation video segmentation scene can be found in the First Model Library.

[0173] Different video segmentation models are suitable for different scenarios. For example, video segmentation model 1 is suitable for scenarios 1 and 2, while video segmentation model 2 is suitable for scenarios 2 and 3. The video segmentation model to be trained refers to the model among multiple video segmentation models that is suitable for the target scenario, but whose performance can be further optimized. If the target scenario is scenario 1, then the video segmentation model to be trained is video segmentation model 1, which is suitable for scenario 1. The video segmentation model to be trained may not only be applicable to the target scenario but also to other scenarios, making it a general video segmentation model applicable to different scenarios. The video segmentation model to be trained can be used to perform video segmentation methods for the target scenario, but the results may not be very good. In this case, the video segmentation model to be trained can be optimized based on the scene input data of the target scenario. For example, optimizing the video segmentation model to be trained based on the scene input data of an animation video segmentation scenario can yield a target video segmentation model suitable for animation video segmentation scenarios. The scene input data of the target scenario can be understood as the sample data used to train the video segmentation model. The method for training the video segmentation model to be trained based on the scene input data of the target scenario to obtain the target video segmentation model can be supervised fine-tuning.

[0174] Model specifications refer to the various parameters that define the structure and behavior of a model. These parameters can be broadly categorized into two types: model parameters (learnable parameters) and hyperparameters. Model parameters are those automatically adjusted during model training using the backpropagation algorithm, including but not limited to weight matrices and bias terms. For example, in a simple fully connected layer, the weight matrix is ​​a two-dimensional tensor connecting neurons in the input and output layers; the bias term is a one-dimensional vector providing additional offset values ​​for each output neuron. Hyperparameters are parameters set before model training begins, used to control the model's learning process and architecture. Hyperparameters include, but are not limited to, the learning rate and the number of neurons per layer, selected based on specific requirements.

[0175] The solution implemented in this specification, based on scenario requirements, accurately identifies the target video segmentation model suitable for the corresponding scenario through scenario identification, making the video segmentation method more accurate and better suited to the target scenario. Based on scenario requirements, a general-purpose video segmentation model is further trained using scenario input data to obtain a target video segmentation model adapted to the target scenario, making the target video segmentation model more closely suited to the target scenario, thereby improving user experience and task processing quality. Based on model specification parameters, the corresponding target video segmentation model can be accurately found, ensuring the efficient and stable operation of the target video segmentation model and enhancing user experience.

[0176] In one optional embodiment of this specification, after determining the target video segmentation model from multiple video segmentation models based on model requests, the following steps may be further included:

[0177] Deploy the target video segmentation model and build a video segmentation interface based on the target video segmentation model so that the terminal device can schedule the target video segmentation model to execute the video segmentation method through the video segmentation interface.

[0178] It should be noted that the video segmentation interface is an interactive programming interface for terminal devices to schedule the target video segmentation model to execute video segmentation methods, and it is usually provided in the form of an application programming interface (API). Through the video segmentation interface, users can input the reference mask frame and the target frame of the original video to execute the video segmentation method.

[0179] In practical applications, there are various ways to deploy the target video segmentation model, and the specific method should be chosen based on the actual situation. This specification does not impose any limitations on this approach. One possible implementation of this specification is to deploy the target video segmentation model on cloud-side devices using infrastructure provided by a cloud service provider. Another possible implementation of this specification is to deploy the target video segmentation model on edge devices using a lightweight framework. For example, the target video segmentation model can be deployed on a distributed system, and a video segmentation interface can be built based on the target video segmentation model and provided to terminal devices, enabling the terminal devices to schedule the target video segmentation model to execute video segmentation methods.

[0180] By applying the solutions provided in the embodiments of this specification, deploying the target video segmentation model, and constructing a video segmentation interface based on the target video segmentation model, terminal devices can efficiently call the target video segmentation model, thereby improving the processing quality and response speed of the video segmentation method.

[0181] See Figure 7 , Figure 7 This specification shows a schematic diagram of the structure of a task platform 700 provided in one embodiment of the present specification. The task platform 700 includes a request interface 702 and a response unit 704.

[0182] Request interface 702 is used to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0183] The response unit 704 is used to determine a target video segmentation model from multiple video segmentation models based on a model request, wherein the target video segmentation model is used in the processing of the video segmentation method.

[0184] In one optional embodiment of this specification, the task platform further includes a video segmentation interface, which is constructed based on the target video segmentation model.

[0185] The video segmentation interface is used by terminal devices to schedule and execute video segmentation methods.

[0186] The above is an illustrative scheme of a task platform according to this embodiment. It should be noted that the technical solution of this task platform belongs to the same concept as the technical solution of the information processing method based on the video segmentation model described above. For details not described in detail in the technical solution of the task platform, please refer to the description of the technical solution of the information processing method based on the video segmentation model described above.

[0187] Corresponding to the above-described video segmentation method embodiments, this specification also provides embodiments of a video segmentation apparatus. Figure 8 A schematic diagram of a video segmentation apparatus according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0188] The first acquisition module 802 is configured to acquire a reference mask frame and a target frame of the original video, wherein the original video includes a reference frame and a target frame, and the reference mask frame is used to reflect the regional position of the object in the reference frame.

[0189] The first input module 804 is configured to input the reference mask frame and the start frame into the video segmentation model to obtain the mask frame corresponding to the start frame, wherein the start frame is the frame in the target frame that is adjacent to the reference frame;

[0190] The second input module 806 is configured to input the current segmentation prompt information and the current frame into the video segmentation model for the current frame to obtain the mask frame corresponding to the current frame. The current frame is any frame in the target frame other than the starting frame. The current segmentation prompt information is constructed based on the object motion features of the segmented frames.

[0191] The first determining module 808 is configured to determine the video segmentation result of the original video based on the mask frame corresponding to the target frame.

[0192] Optionally, the device further includes: a third acquisition module, configured to acquire, for the current frame, a segmentation mask frame corresponding to the segmented frame, wherein the segmentation mask frame is used to reflect the region position of the object in the segmented frame; determine the position information of the object in the segmented frame based on the segmentation mask frame; extract the object motion features in the segmented frame based on the position information; predict the position information of the object in the current frame based on the object motion features, and obtain the current segmentation prompt information.

[0193] Optionally, the location information includes at least one of key point location information and pixel location information; the third acquisition module is further configured to input the key point location information into a sparse motion encoder to obtain the object motion features of the object in the segmented frame; and input the pixel location information into a dense motion encoder to obtain the object motion features of the object in the segmented frame.

[0194] Optionally, the device further includes: a fourth acquisition module configured to acquire a sample video and a sample video segmentation result of the sample video; and to adjust the parameters of the sparse motion encoder and / or the dense motion encoder based on the sample video and the sample video segmentation result to obtain a trained sparse motion encoder and / or a trained sparse motion encoder.

[0195] Optionally, the device further includes: a filtering module configured to filter out auxiliary mask frames from the segmentation mask frames corresponding to the segmented frames; and a second input module 806 further configured to input the current segmentation prompt information, the current frame, and the auxiliary mask frames into the video segmentation model for the current frame to obtain the mask frame corresponding to the current frame.

[0196] Optionally, the filtering module is further configured to filter candidate mask frames from the segmentation mask frames corresponding to the segmented frames based on the temporal information of the segmented frames; and to filter auxiliary mask frames from the candidate mask frames based on the pixel information of the candidate mask frames.

[0197] Optionally, the filtering module is further configured to: extract multiple initial mask frames adjacent to the current frame from the segmentation mask frames corresponding to the segmented frames based on the temporal information of the segmented frames; filter replacement mask frames from the segmented mask frames according to the quality index of the segmented mask frames, wherein the quality index of the replacement mask frames is greater than the quality index of the frames in the segmented mask frames other than the replacement mask frames; filter mask frames to be replaced from the multiple initial mask frames according to the quality index of the initial mask frames, wherein the quality index of the mask frames to be replaced is less than the quality index of the frames in the initial mask frames other than the mask frames to be replaced, and the number of mask frames to be replaced is the same as the number of replacement mask frames; and replace the mask frames to be replaced in the multiple initial mask frames using the replacement mask frames to obtain candidate mask frames.

[0198] Optionally, the filtering module is further configured to determine the pixel index of the candidate mask frame based on the pixel information of the candidate mask frame; and to filter out auxiliary mask frames from the candidate mask frames based on the pixel index, wherein the pixel index of the auxiliary mask frame is greater than the pixel index of the candidate mask frames other than the auxiliary mask frame.

[0199] By applying the scheme of the embodiments in this specification, using a reference mask frame as a starting point and combining it with the initial frame for preliminary segmentation, the position of the object can be located more accurately, providing a reliable basis for the segmentation of subsequent frames. Since the current segmentation prompt information is constructed based on the object motion features of already segmented frames, it achieves dynamic adjustment of the current segmentation prompt information as the video frames progress, better adapting to changes, movements, or even partial occlusion or temporary disappearance of objects. This alleviates the error accumulation problem during continuous frame processing, enhances the adaptability and robustness of the video segmentation model to complex scene changes, and further improves the accuracy of the video segmentation results.

[0200] The above is an illustrative scheme of a video segmentation device according to this embodiment. It should be noted that the technical solution of this video segmentation device and the technical solution of the video segmentation method described above belong to the same concept. For details not described in detail in the technical solution of the video segmentation device, please refer to the description of the technical solution of the video segmentation method described above.

[0201] Corresponding to the above-described video editing method embodiments, this specification also provides embodiments of a video editing device. Figure 9 A schematic diagram of a video editing apparatus according to one embodiment of this specification is shown. Figure 9 As shown, the device includes:

[0202] The second acquisition module 902 is configured to acquire the original video and the video segmentation result of the original video, wherein the video segmentation result is obtained based on the video segmentation method.

[0203] Editing module 904 is configured to perform object editing on the original video based on the video segmentation results to obtain the video editing results.

[0204] The solution implemented in this specification has a high accuracy rate in video segmentation results obtained by the video segmentation method. Therefore, the location of an object in the original video can be accurately determined based on the video segmentation results, thereby achieving precise object editing of the original video and improving the accuracy of the video editing results.

[0205] The above is an illustrative scheme of a video editing device according to this embodiment. It should be noted that the technical solution of this video editing device and the technical solution of the video editing method described above belong to the same concept. For details not described in detail in the technical solution of the video editing device, please refer to the description of the technical solution of the video editing method described above.

[0206] Corresponding to the above embodiments of the information processing method based on video segmentation models, this specification also provides embodiments of the information processing apparatus based on video segmentation models. Figure 10A schematic diagram of the structure of an information processing device based on a video segmentation model according to one embodiment of this specification is shown. Figure 10 As shown, the device is applied to a mission platform and includes:

[0207] The receiving module 1002 is configured to receive model requests sent by the terminal device;

[0208] The second determining module 1004 is configured to determine a target video segmentation model from multiple video segmentation models based on a model request, wherein the target video segmentation model is used in the processing of the video segmentation method.

[0209] Optionally, the second determining module 1004 is further configured to: when the model request includes a scene identifier of the target scene, search for a target video segmentation model suitable for the target scene from a first model library based on the scene identifier, wherein the first model library stores multiple video segmentation models suitable for different scenes; when the model request includes scene input data of the target scene, determine a video segmentation model to be trained suitable for the target scene from multiple video segmentation models, and train the video segmentation model to be trained based on the scene input data to obtain the target video segmentation model; when the model request includes model specification parameters, search for a target video segmentation model corresponding to the model specification parameters from a second model library, wherein the second model library stores multiple video segmentation models corresponding to different model specification parameters.

[0210] Optionally, the second determining module 1004 is further configured to deploy the target video segmentation model and, based on the target video segmentation model, construct a video segmentation interface so that the terminal device can schedule the target video segmentation model to execute the video segmentation method through the video segmentation interface.

[0211] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target video segmentation models, realizing personalized model services and providing users with an efficient, flexible and easy-to-use model service solution, thereby improving user experience.

[0212] The above is an illustrative scheme of an information processing device based on a video segmentation model according to this embodiment. It should be noted that the technical solution of this information processing device based on a video segmentation model belongs to the same concept as the technical solution of the information processing method based on a video segmentation model described above. Details not described in detail in the technical solution of the information processing device based on a video segmentation model can be found in the description of the technical solution of the information processing method based on a video segmentation model described above.

[0213] Figure 11 A structural block diagram of a computing device 1100 provided in one embodiment of this specification is shown.

[0214] The computing device 1100 includes:

[0215] Memory 1110 and processor 1120;

[0216] The memory 1110 is used to store computer programs / instructions, and the processor 1120 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1120, they implement the steps of the above-mentioned video segmentation method, video editing method, or information processing method based on video segmentation model.

[0217] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, personal computer, all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0218] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing application programming interface (API) invocation capabilities. Models can be invoked into created applications through the API interface, and application management tools are provided for application management and monitoring.

[0219] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master artificial intelligence technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for artificial intelligence development, training, deployment, and application.

[0220] Figure 12 A structural block diagram of an electronic device 1200 provided according to one embodiment of this specification is shown.

[0221] The memory 1210 and the processor 1220 are connected via a bus 1230;

[0222] The memory 1210 is used to store computer programs / instructions, and the processor 1220 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1220, they implement the steps of the video segmentation method, video editing method, or information processing method based on the video segmentation model.

[0223] Specifically, the components of the electronic device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 and the memory 1210 can be connected via a bus 1230.

[0224] Electronic device 1200 may also include access device 1240, which enables electronic device 1200 to communicate with database 1250 storing data via one or more networks 1260. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 1240 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, Wi-MAX (World Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0225] In one embodiment of this specification, the above-described components of the electronic device 1200 and Figure 12 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 12 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0226] Electronic device 1200 can be any type of stationary or mobile electronic device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or PCs. Electronic device 1200 can also be a mobile or stationary electronic device.

[0227] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the technical solutions of the video segmentation method, video editing method, and information processing method based on video segmentation model described above. For details not described in detail in the technical solution of the electronic device, please refer to the description of the technical solutions of the video segmentation method, video editing method, or information processing method based on video segmentation model described above.

[0228] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the video segmentation method, video editing method, or information processing method based on a video segmentation model described above.

[0229] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the video segmentation method, video editing method, and information processing method based on video segmentation models described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of the video segmentation method, video editing method, or information processing method based on video segmentation models described above.

[0230] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described video segmentation method, video editing method, or information processing method based on a video segmentation model.

[0231] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the video segmentation method, video editing method, and information processing method based on video segmentation models described above. Details not described in detail in the technical solution of the computer program product can be found in the descriptions of the technical solutions of the video segmentation method, video editing method, or information processing method based on video segmentation models described above.

[0232] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0233] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0234] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0235] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0236] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A video segmentation method, comprising: Obtain a reference mask frame and a target frame of the original video, wherein the original video includes a reference frame and the target frame, and the reference mask frame is used to reflect the regional position of the object in the reference frame; The reference mask frame and the start frame are input into the video segmentation model to obtain the mask frame corresponding to the start frame, wherein the start frame is the frame in the target frame that is adjacent to the reference frame; For the current frame, the current segmentation prompt information and the current frame are input into the video segmentation model to obtain the mask frame corresponding to the current frame. The current frame is any frame in the target frame other than the starting frame. The current segmentation prompt information is constructed based on the object motion features of the segmented frames. Based on the mask frame corresponding to the target frame, the video segmentation result of the original video is determined.

2. The method according to claim 1, further comprising, before inputting the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame, the method includes: For the current frame, obtain the segmentation mask frame corresponding to the segmented frame, wherein the segmentation mask frame is used to reflect the region position of the object in the segmented frame; Based on the segmented mask frame, determine the position information of the object in the segmented frame; Based on the location information, extract the object motion features of the object in the segmented frame; Based on the object's motion characteristics, predict the object's position information in the current frame to obtain the current segmentation prompt information.

3. The method according to claim 2, wherein the location information includes at least one of key point location information and pixel location information; The step of extracting the object motion features in the segmented frame based on the location information includes: The key point location information is input into a sparse motion encoder to obtain the object motion features of the object in the segmented frame. The pixel position information is input into a dense motion encoder to obtain the object motion features of the object in the segmented frame.

4. The method according to claim 3, further comprising, before extracting the object motion features in the segmented frame based on the location information: Obtain the sample video and the sample video segmentation result of the sample video; Based on the sample video and the sample video segmentation results, the parameters of the sparse motion encoder and / or the dense motion encoder are adjusted to obtain the trained sparse motion encoder and / or the trained sparse motion encoder.

5. The method according to any one of claims 1 to 4, wherein before inputting the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame, the method further includes: From the segmentation mask frames corresponding to the already segmented frames, auxiliary mask frames are selected; The step of inputting the current segmentation prompt information and the current frame into the video segmentation model to obtain the mask frame corresponding to the current frame includes: For the current frame, the current segmentation prompt information, the current frame, and the auxiliary mask frame are input into the video segmentation model to obtain the mask frame corresponding to the current frame.

6. The method according to claim 5, wherein filtering out auxiliary mask frames from the segmented mask frames corresponding to the segmented frames comprises: Based on the timing information of the segmented frames, candidate mask frames are selected from the segmented mask frames corresponding to the segmented frames. The auxiliary mask frame is selected from the candidate mask frames based on the pixel information of the candidate mask frames.

7. The method according to claim 6, wherein selecting candidate mask frames from the segmentation mask frames corresponding to the segmented frames based on the temporal information of the segmented frames includes: Based on the timing information of the segmented frame, multiple initial mask frames adjacent to the current frame are extracted from the segmentation mask frames corresponding to the segmented frame. Based on the quality index of the segmented mask frame, a replacement mask frame is selected from the segmented mask frame, wherein the quality index of the replacement mask frame is greater than the quality index of the frames in the segmented mask frame other than the replacement mask frame. Based on the quality index of the initial mask frame, a replacement mask frame is selected from the plurality of initial mask frames, wherein the quality index of the replacement mask frame is lower than the quality index of the frames other than the replacement mask frame in the initial mask frames, and the number of replacement mask frames is the same as the number of replacement mask frames. The candidate mask frame is obtained by replacing the mask frame to be replaced in the plurality of initial mask frames using the replacement mask frame.

8. The method according to claim 6, wherein selecting the auxiliary mask frame from the candidate mask frames based on the pixel information of the candidate mask frames comprises: The pixel index of the candidate mask frame is determined based on the pixel information of the candidate mask frame; The auxiliary mask frame is selected from the candidate mask frames based on the pixel index, wherein the pixel index of the auxiliary mask frame is greater than the pixel index of the other candidate mask frames.

9. A video editing method, comprising: Obtain the original video and the video segmentation result of the original video, wherein the video segmentation result is obtained based on the method described in any one of claims 1 to 8; Based on the video segmentation results, the original video is edited to obtain the video editing result.

10. An information processing method based on a video segmentation model, applied to a task platform, comprising: Receive model requests sent by terminal devices; Based on the model request, a target video segmentation model is determined from multiple video segmentation models, wherein the target video segmentation model is used in the processing of the method as described in any one of claims 1 to 8.

11. The method according to claim 10, wherein determining the target video segmentation model from multiple video segmentation models based on the model request comprises: If the model request includes a scene identifier of the target scene, a target video segmentation model suitable for the target scene is searched from the first model library based on the scene identifier. The first model library stores multiple video segmentation models suitable for different scenes. When the model request includes scene input data of the target scene, a video segmentation model to be trained that is suitable for the target scene is determined from the plurality of video segmentation models, and the video segmentation model to be trained is trained based on the scene input data to obtain the target video segmentation model; If the model request includes model specification parameters, the target video segmentation model corresponding to the model specification parameters is searched from the second model library, wherein the second model library stores multiple video segmentation models corresponding to different model specification parameters.

12. The method according to claim 10, further comprising, after determining the target video segmentation model from multiple video segmentation models based on the model request: Deploy the target video segmentation model, and based on the target video segmentation model, construct a video segmentation interface so that the terminal device can schedule the target video segmentation model to execute the video segmentation method as described in any one of claims 1 to 8 through the video segmentation interface.

13. A task platform, comprising a request interface and a response unit; The request interface is used to receive model requests sent by the terminal device, wherein... The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters. The response unit is configured to determine a target video segmentation model from multiple video segmentation models based on the model request, wherein the target video segmentation model is used in the processing of the method as described in any one of claims 1 to 8.

14. The task platform according to claim 13, further comprising a video segmentation interface, wherein the video segmentation interface is constructed based on the target video segmentation model; The video segmentation interface is used for the terminal device to schedule and execute the video segmentation method as described in any one of claims 1 to 8.

15. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

16. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

17. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

18. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.