Video processing method and apparatus, medium, and electronic device

Abstract

The present disclosure provides a video processing method, device, medium and electronic equipment, relating to the field of video processing, the method comprising: performing matting processing on a first target video by a video matting model to obtain a first foreground video and a first background video; the video matting model being trained according to at least a second target video output by a video synthesis model, the video synthesis model performing feature extraction on target video frames of the same moment in a second foreground video and a second background video to obtain first video frame features, and obtaining second video frame features corresponding to historical video frames of the same moment in the second foreground video and the second background video; and synthesizing the second foreground video and the second background video according to the first video frame features and the second video frame features to obtain the second target video. The video synthesis result of the present disclosure is more realistic, and thus the video matting model is trained based on the video synthesis result, so that the accuracy and efficiency of the video matting model are higher.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, medium and electronic device. Background Technology

[0002] In the field of video processing, background removal can improve the quality and visual effects of video content. For example, by removing the foreground from user-uploaded videos or LivePhotos, the extracted video foreground can be placed on top of the lock screen component and blended with the original video background to create a new video lock screen effect. Summary of the Invention

[0003] This disclosure provides a video processing method, apparatus, medium, and electronic device to improve the accuracy of video matting.

[0004] According to a first aspect of the present disclosure, a video processing method is provided, comprising:

[0005] Obtain the first target video to be processed;

[0006] The first target video is processed by a video matting model to obtain a first foreground video and a first background video; the video matting model is trained based on at least the second target video output by a video synthesis model, and the video synthesis model is used to obtain the second target video in the following manner:

[0007] Feature extraction is performed on the target video frame at the same moment in the second foreground video and the second background video to obtain the first video frame feature, and the second video frame feature corresponding to the historical video frame at the same moment in the second foreground video and the second background video is obtained, wherein the historical video frame is located before the target video frame;

[0008] Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized to obtain the second target video.

[0009] Optionally, the second foreground video and the second background video are synthesized based on the features of the first video frame and the features of the second video frame to obtain the second target video, including:

[0010] Based on the features of the first video frame, predict the target position of the target subject in the second foreground video in the second background video;

[0011] Based on the first video frame features, the second video frame features, and the target position, target video frames at the same time in the second foreground video and the second background video are synthesized to obtain a target composite frame.

[0012] The second target video is obtained based on the target composite frame.

[0013] Optionally, the video synthesis model includes a feature extraction module, a storage module, a position prediction module, and a synthesis module. The feature extraction module is used to extract features from target video frames at the same time in the second foreground video and the second background video to obtain the first video frame features. The storage module is used to store the second video frame features. The position prediction module is used to predict the target position of the target subject in the second foreground video in the second background video based on the first video frame features. The synthesis module is used to synthesize the target video frames at the same time in the second foreground video and the second background video based on the first video frame features, the second video frame features, and the target position to obtain the target synthesized frame.

[0014] The training of the video synthesis model includes:

[0015] Keeping the parameters of the feature extraction module unchanged, and updating the parameters of the storage module, the location prediction module, and the synthesis module, the video synthesis model is obtained.

[0016] Optionally, the training of the video synthesis model includes:

[0017] Acquire a first sample foreground video, a first sample background video, and first tag data. The first tag data includes a composite frame of sample video frames from the first sample foreground video and the first sample background video at the same time.

[0018] The parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data.

[0019] Optionally, the sample synthesis frame includes the actual location of the target subject in the first sample foreground video in the first sample background video;

[0020] The parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data, including:

[0021] Based on sample video frames at the same time in the first sample foreground video and the first sample background video, a predicted composite frame is obtained, and the predicted composite frame includes the predicted position of the target subject in the first sample foreground video in the first sample background video.

[0022] Based on the actual location and the predicted location, a first loss value is determined;

[0023] The second loss value is determined based on the sample synthesized frame and the predicted synthesized frame;

[0024] The third loss value is determined based on the synthesized frames of samples from adjacent time points and the time consistency loss function;

[0025] The parameters of the video synthesis model are updated based on the first loss value, the second loss value, and the third loss value.

[0026] Optionally, the parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data, including:

[0027] Random data augmentation is performed on the first sample foreground video to adjust the hue and / or saturation information of the first sample foreground video to obtain the second sample foreground video;

[0028] The parameters of the video synthesis model are updated based on the second sample foreground video, the first sample background video, and the first tag data.

[0029] Optionally, feature extraction is performed on the target video frames at the same time in the second foreground video and the second background video to obtain the features of the first video frame, including:

[0030] The first video frame features are obtained by extracting features from the target video frames at the same time in the second foreground video and the second background video through the matching and scoring module.

[0031] The method further includes:

[0032] The matching and scoring module obtains a matching reference value based on the features of the first video frame. The matching reference value is used to characterize the degree of matching between the second foreground video and the second background video.

[0033] Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized, including:

[0034] When the matching reference value is greater than the first preset reference threshold, the second foreground video and the second background video are synthesized based on the first video frame features and the second video frame features.

[0035] Optionally, the training of the matching scoring model includes:

[0036] Acquire a third sample foreground video, a second sample background video, and second tag data. The second tag data includes a true score that characterizes the degree of matching between the third sample foreground video and the second sample background video.

[0037] The matching scoring model obtains a predicted score based on the third sample foreground video and the second sample background video;

[0038] The parameters of the matching scoring model are updated based on the actual score and the predicted score.

[0039] Optionally, the training of the video matting model further includes:

[0040] The video matting model is trained by alternately using a first sample dataset and a second sample dataset; wherein the first sample dataset includes image segmentation data and / or video segmentation data, and the second sample dataset includes image matting data and / or video matting data.

[0041] Optionally, the image matting data is obtained in the following way:

[0042] Based on the image segmentation data, a tricolor map is generated, which is used to indicate the foreground region, the background region, and the transition region between the foreground region and the background region in the image segmentation data;

[0043] The image matting data is obtained based on the tri-color image and the image diffusion model. The image diffusion model is used to gradually add noise to the tri-color image until it becomes a pure noise image, and then gradually remove noise from the pure noise image until the image matting data is obtained.

[0044] Optionally, the image matting data is obtained in the following way:

[0045] A mask is generated based on the image segmentation data, and the mask is used to indicate the foreground and background regions in the image segmentation data;

[0046] The image matting data is obtained based on the mask image and the preset image matting model. The image matting model is used to extract the global and local features of the image segmentation data, and the image matting data is obtained through the global and local features.

[0047] Optionally, the video segmentation data is obtained in the following manner:

[0048] Obtain a first sample video to be segmented, wherein the first sample video includes at least one video frame labeled with the target subject;

[0049] The video segmentation data is obtained by performing forward and / or backward propagation based on the first sample video and the at least one video frame using a preset video segmentation model.

[0050] Optionally, the video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The video matting data is obtained in the following way:

[0051] Obtain the second sample video to be cut out;

[0052] The video matting model obtains initial video matting data based on the second sample video, and the difficulty prediction module obtains a difficulty reference value based on the second sample video. The difficulty reference value is used to characterize the ease or difficulty of performing video matting on the second sample video.

[0053] The video cutout data is determined based on the initial video cutout data, the difficulty reference value, and the second preset reference threshold.

[0054] Optionally, the video matting data is determined based on the initial video matting data, the difficulty reference value, and the second preset reference threshold, including:

[0055] When the difficulty reference value is less than or equal to the second preset reference threshold, the initial video cutout data is used as the video cutout data;

[0056] When the difficulty reference value is greater than the second preset reference threshold, the initial video cutout data is adjusted to obtain the video cutout data.

[0057] Optionally, the method further includes:

[0058] Repeat the steps of obtaining initial video matting data based on the second sample video using the video matting model, and obtaining a difficulty reference value based on the second sample video using the difficulty prediction module;

[0059] When the number of repeated executions exceeds a preset number, the second preset reference threshold is adjusted to obtain a third preset reference threshold;

[0060] The steps of obtaining initial video matting data based on the second sample video using the video matting model and obtaining a difficulty reference value based on the second sample video using the difficulty prediction module are executed again. The video matting data is then determined based on the initial video matting data, the difficulty reference value, and the third preset reference threshold.

[0061] Optionally, the video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The difficulty is used to assist the video matting model in determining whether to perform matting processing on the first target video. The training of the video matting model includes:

[0062] Obtain a third sample dataset, which includes target data whose difficulty labeling value exceeds a third preset reference threshold, and the proportion of the number of target data in the third sample dataset is greater than a preset proportion.

[0063] Based on the third sample dataset, update the parameters of the difficulty prediction module, while keeping the parameters of other modules in the video matting model unchanged.

[0064] Optionally, the video matting model further includes a classification module, which is used to identify the type of the target subject in the first target video. The type is used to assist the video matting model in matting the first target video. The training of the video matting model further includes:

[0065] Obtain a fourth sample dataset, which includes data of multiple subject types, and the proportion of each subject type in the fourth sample dataset is the same.

[0066] Based on the fourth sample dataset, update the parameters of the classification module, while keeping the parameters of other modules in the video matting model unchanged.

[0067] Optionally, the method further includes:

[0068] The first foreground video, the first background video, and the target component are synthesized to obtain a third target video for display on the lock screen or desktop interface, wherein the target component is a lock screen component or a desktop component.

[0069] According to a second aspect of the present disclosure, a video processing apparatus is provided, comprising:

[0070] The acquisition module is configured to acquire the first target video to be processed.

[0071] The acquisition module is configured to perform image matting processing on the first target video using a video matting model to obtain a first foreground video and a first background video; the video matting model is trained at least on a second target video output by a video synthesis model, and the video synthesis model is used to obtain the second target video in the following manner:

[0072] Feature extraction is performed on the target video frame at the same moment in the second foreground video and the second background video to obtain the first video frame feature, and the second video frame feature corresponding to the historical video frame at the same moment in the second foreground video and the second background video is obtained, wherein the historical video frame is located before the target video frame;

[0073] Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized to obtain the second target video.

[0074] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the video processing method provided in the first aspect of the present disclosure.

[0075] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0076] Storage device for storing computer programs;

[0077] An execution device is used to execute the computer program to implement the video processing method provided in the first aspect of the embodiments of this disclosure.

[0078] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0079] This disclosure uses a video matting model to perform matting processing on a first target video, obtaining a first foreground video and a first background video. The video matting model is trained at least on a second target video output by a video synthesis model. The video synthesis model extracts features from target video frames at the same time in the second foreground and second background videos to obtain first video frame features, and acquires second video frame features corresponding to historical video frames at the same time in the second foreground and second background videos. Based on the first and second video frame features, the second foreground and second background videos are synthesized to obtain the second target video. The video synthesis model synthesizes the current video frame based on the features of historical video frames, considering the temporal information between video frames, thus avoiding flickering and discontinuities between video frames, resulting in a more realistic video synthesis result. Therefore, training the video matting model based on the video synthesis result enhances its generalization and understanding capabilities for real video processing, resulting in higher accuracy and efficiency.

[0080] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0081] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0082] Figure 1 This is a flowchart illustrating a video processing method according to an exemplary embodiment.

[0083] Figure 2 This is a flowchart illustrating a video synthesis method according to an exemplary embodiment.

[0084] Figure 3 This is an architecture diagram of a matching and scoring module according to an exemplary embodiment.

[0085] Figure 4 This is a block diagram illustrating a video processing apparatus according to an exemplary embodiment.

[0086] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0087] Figure 6 This is a block diagram illustrating an apparatus for video processing according to an exemplary embodiment. Detailed Implementation

[0088] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0089] In related technologies, video matting is commonly used for portrait video matting. Video matting models can be trained based on video segmentation data or video matting data to extract the foreground from the video. Video segmentation data has many categories but coarse edges. While video matting data has high accuracy, it is limited in quantity and has high annotation costs, with only people as a possible category. For some specific user scenarios, dynamic objects such as animals and vehicles can be extracted as foreground objects, as can static objects such as tables and teacups that move relative to the camera as the video moves.

[0090] In training video matting models, publicly available datasets can be used. These datasets offer high annotation accuracy, achieving hair-level edge precision, which significantly improves the model's ability to handle hair edges. However, the datasets are often limited in quantity, the scenes are simple, and the data is mostly synthetic video. Synthetic video data differs greatly from real-world scenes, which can negatively impact model training.

[0091] In terms of synthetic video data, most of them are based on image synthesis models. Using such models to synthesize videos will cause obvious flickering, resulting in poor realism of the synthesized videos.

[0092] Reference Figure 1 , Figure 1This is a flowchart illustrating a video processing method according to an exemplary embodiment, such as... Figure 1 As shown, the video processing method used in the terminal includes the following steps.

[0093] In step S101, the first target video to be processed is acquired.

[0094] In step S102, the first target video is processed by a video matting model to obtain a first foreground video and a first background video.

[0095] The video matting model is trained based on at least the second target video output by the video synthesis model. The video synthesis model is used to obtain the second target video in the following way: extracting features from the target video frames at the same time in the second foreground video and the second background video to obtain the first video frame features, and obtaining the second video frame features corresponding to the historical video frames at the same time in the second foreground video and the second background video, wherein the historical video frames are located before the target video frames; and synthesizing the second foreground video and the second background video based on the first video frame features and the second video frame features to obtain the second target video.

[0096] This disclosure uses a video matting model to perform matting processing on a first target video, obtaining a first foreground video and a first background video. The video matting model is trained at least on a second target video output by a video synthesis model. The video synthesis model extracts features from target video frames at the same time in the second foreground and second background videos to obtain first video frame features, and acquires second video frame features corresponding to historical video frames at the same time in the second foreground and second background videos. Based on the first and second video frame features, the second foreground and second background videos are synthesized to obtain the second target video. The video synthesis model synthesizes the current video frame based on the features of historical video frames, considering the temporal information between video frames, thus avoiding flickering and discontinuities between video frames, resulting in a more realistic video synthesis result. Therefore, training the video matting model based on the video synthesis result enhances its generalization and understanding capabilities for real video processing, resulting in higher accuracy and efficiency.

[0097] For example, the first target video is the video to be processed. The foreground and background videos can be separated from the first target video using a video matting model. The first target video can be obtained from any video stored in the storage module, or it can be obtained from the data stream acquired by the terminal in real time. It can be understood that the image data stream corresponding to a dynamic photo can also be used as the first target video for matting processing.

[0098] For example, the first foreground video is the foreground portion extracted from the first target video after matting. The first background video is the background portion extracted from the first target video after matting. The video matting model is a deep learning model that can be used to separate the foreground and background in a video. This disclosure does not limit the specific type of video matting model.

[0099] For example, the second target video is a video generated by the video synthesis model based on a second foreground video and a second background video. This second target video can be used to train the video matting model. Historical video frames are frames preceding the target video frame. The features of the second video frames corresponding to these historical frames can be used to assist in synthesizing the target video frame at the same moment in the second foreground video and the second background video. The historical video frame features can include temporal information between videos, serving as a reference for synthesizing the current video frame and avoiding inter-frame flickering or discontinuity in the synthesized video.

[0100] For example, feature extraction of a target video frame at the same moment in both the second foreground video and the second background video can be performed by: extracting features from each video frame in the second foreground video based on a foreground encoder, extracting features from each video frame in the second background video based on a background encoder, and then concatenating the features of the target video frames at the same moment in the second foreground video and the second background video to obtain the first video frame features. Alternatively, feature extraction can be performed on the target video frame at a specific moment in the second foreground video based on a foreground encoder, and on the target video frame at that moment in the second background video based on a background encoder, and then concatenating the two to obtain the first video frame features.

[0101] For example, the second video frame feature is the feature of a historical video frame in the second foreground video and the second background video. The historical video frame is located before the target video frame. When synthesizing the target video frame in the second foreground video and the second background video, the target video frames corresponding to the second foreground video and the second background video can be synthesized based on the second video frame feature corresponding to one or more historical video frames.

[0102] For example, after obtaining the first video frame feature corresponding to the current moment, the video synthesis model can save the first video frame feature as the second video frame feature corresponding to the next moment. For the first video frame in the second foreground video and the second background video, video matting methods from related technologies can be used to synthesize the first video frame in the second foreground video and the second background video, obtaining the synthesized result corresponding to the first video frame.

[0103] For example, the durations of the second foreground video and the second background video can be the same or different. If the durations of the second foreground video and the second background video are different, the duration of the final synthesized second target video can be based on the shortest video duration between the two. The duration of the second target video can also be preset. For example, if the durations of the second foreground video and the second background video are 30 seconds, the duration of the second target video could be 10 seconds. In this case, a first duration range for the second foreground video and a second duration range for the second background video can be determined according to the user's needs. As in the example above, the first duration range for the second foreground video can be determined to be 5-15 seconds, and the second duration range for the second background video can be determined to be 20-30 seconds.

[0104] For example, the initial foreground video and initial background video can be pre-trimmed to unify their durations, resulting in a second foreground video and a second background video. The original durations of the initial foreground video and initial background video can be different. For example, the original duration of the initial foreground video could be 15 seconds, and the original duration of the initial background video could be 20 seconds. Furthermore, different time-series positions of the same duration from the initial foreground video and initial background video can be extracted to obtain the second foreground video and the second background video. For example, if the original durations of both the initial foreground video and initial background video are 20 seconds, frames from 0-15 seconds of the initial foreground video can be extracted as the second foreground video, and frames from 2-17 seconds of the initial background video can be extracted as the second background video.

[0105] For example, by compositing each video frame in the second foreground video and the second background video, a composite second target video can be obtained. This allows the foreground in the video to be dynamically replaced, providing more possibilities for video editing. The foreground and background videos separated by the video matting model can also support real-time video processing applications, such as video conferencing, live streaming, and gaming.

[0106] For example, by processing the features of the first video frame corresponding to the current video frame based on the features of the second video frame corresponding to historical video frames, the temporal information between video frames is taken into account, making the second target video closer to the actually acquired video. Using the second target video to train the video matting model, and then applying the trained video matting model to actual video matting scenarios, can improve the video matting model's ability to understand real-world scenes when handling complex scenes and details, resulting in more accurate first foreground and first background videos.

[0107] As an optional embodiment, the second target video is obtained by synthesizing the second foreground video and the second background video based on the features of the first video frame and the features of the second video frame, including:

[0108] Based on the features of the first video frame, predict the target position of the target in the second foreground video in the second background video;

[0109] Based on the features of the first video frame, the features of the second video frame, and the target position, the target video frames at the same time in the second foreground video and the second background video are synthesized to obtain the target composite frame.

[0110] The second target video is obtained based on the target composite frame.

[0111] For example, the target subject is an object identified and tracked in a video frame, such as a person, vehicle, animal, or other salient object. The target location is the expected location of the target subject in the second background video, which can be predicted based on features of the first video frame. The target composite frame is a single video frame synthesized from the target subject in the second foreground video and the background in the second background video. The second target video is the composite result of the second foreground video and the second background video. The second target video can be obtained based on the target composite frames corresponding to multiple consecutive time points.

[0112] For example, the target position of a subject in the second foreground video can be predicted in the second background video based on the features of the first video frame. However, the position of the target subject in the second foreground video may not perfectly correspond to its actual position in the second background video. For instance, people, vehicles, and animals are generally located on the ground, while a teacup may be placed on a table or windowsill. Without predicting the target position in the second background video, unreasonable situations may arise, such as animals floating on the ground while running.

[0113] For example, each video frame in the foreground video includes information such as the position and action of the target subject in the video interface after the background has been extracted. This can also be understood as the foreground video including pixel information of the area where the target subject is located and the background area, where the background area contains no pixel information or consists entirely of black pixels. Correspondingly, the background video may include dynamically changing pixel information of the background area, as well as the area where the target subject is located, where the target subject area contains no pixel information or consists entirely of black pixels.

[0114] For example, the target subject in the foreground video may change as the video data stream of the foreground video increases. For instance, at the first moment, the target subject in the foreground video might be a person, while at the second moment, 5 seconds later, the target subject might switch to a dog. Therefore, the target subject may be different at different moments corresponding to the same second foreground video. This can be identified and classified based on the classification module in the video matting model. The implementation principles of the classification module are explained in the following embodiments.

[0115] Based on this, the video synthesis model disclosed herein can improve the quality of synthesized videos through feature extraction and target subject localization, making the content of the synthesized videos closer to the content of real-world video scenes. When used to train a video matting model, it can improve the accuracy of video matting.

[0116] As an optional embodiment, the video synthesis model includes a feature extraction module, a storage module, a position prediction module, and a synthesis module. The feature extraction module is used to extract features from target video frames at the same time in the second foreground video and the second background video to obtain first video frame features. The storage module is used to store second video frame features. The position prediction module is used to predict the target position of the target subject in the second foreground video in the second background video based on the first video frame features. The synthesis module is used to synthesize the target video frames at the same time in the second foreground video and the second background video based on the first video frame features, the second video frame features, and the target position to obtain a target synthesized frame.

[0117] Training the video synthesis model includes:

[0118] Keeping the parameters of the feature extraction module unchanged, and updating the parameters of the storage module, location prediction module, and synthesis module, a video synthesis model is obtained.

[0119] For example, a video synthesis model includes multiple modules for implementing different functions. These modules may include a feature extraction module, a storage module, a position prediction module, and a synthesis module. The feature extraction module extracts features from target video frames at the same time in both the second foreground video and the second background video to obtain the features of the first video frame. The storage module stores the features of the second video frame; after extracting features from the target video frame, it can also store the features of the first video frame at the current time as the features of the second video frame at the next time. The position prediction module predicts the target position of the subject in the second foreground video in the second background video based on the features of the first video frame. The synthesis module synthesizes the target video frames at the same time in both the second foreground video and the second background video based on the features of the first video frame, the features of the second video frame, and the target position to obtain the synthesized target frame.

[0120] For example, the storage module can persistently store the second video frame features corresponding to multiple historical video frames extracted by the feature extraction module from the foreground and background videos, to assist in the synthesis of the current target video frame in the foreground and background videos. By sharing the features of historical video frames with the current video frame, the storage module can avoid flickering and discontinuities between video frames in the synthesized second target video. The storage module can store the features of all video frames and all target synthesized frames. In the video, some video frames may experience jitter or drift; the storage module can correct these jitter or drift phenomena through global memory.

[0121] like Figure 2 As shown, the feature extraction module is connected to the storage module and the location prediction module, and both the storage module and the location prediction module are connected to the synthesis module. Specifically, when performing video synthesis tasks using the video synthesis model, the feature extraction module includes a foreground encoder and a background encoder. The foreground encoder extracts features from the video frames of the foreground video, and the background encoder extracts features from the video frames of the background video. When training the video synthesis model, in addition to the foreground encoder and background encoder, the feature extraction module may also include an attention module. The foreground video during training includes the transparency channel information of the foreground video, and the attention module can extract features based on the transparency channel information to determine the location of the target subject in the foreground video that requires special attention.

[0122] See also Figure 2 The first video frame feature is obtained by concatenating the features extracted by the foreground encoder, background encoder, and attention module. This first video frame feature is stored in a storage module. Based on the first video frame feature, a position prediction module predicts the target position of the object in the second foreground video within the second background video. Then, a synthesis module synthesizes the target video frame at the current moment in both the second foreground and background videos based on the target position output by the position prediction module, the first video frame feature stored in the storage module, and the second video frame feature stored in the storage module, resulting in a synthesized target frame. Finally, based on the synthesized target frames corresponding to multiple consecutive moments, a second target video is obtained.

[0123] For example, the feature extraction module in a video synthesis model can be a general feature extraction module, or it can be a feature extraction module associated with other tasks. When training the video synthesis module, the feature extraction module may have already been trained. To reduce computation and training time, the parameters of the feature extraction module can be kept unchanged while the parameters of other modules are updated. The video synthesis model can also be a pre-trained model. In a pre-trained model, the performance of the feature extraction module can be considered very stable and sufficient to meet the accuracy requirements of feature extraction. Therefore, the parameters of the feature extraction module can also be kept unchanged while other modules are trained.

[0124] As an optional implementation, training the video synthesis model includes:

[0125] Acquire the first sample foreground video, the first sample background video, and the first label data. The first label data includes a composite frame of sample video frames from the first sample foreground video and the first sample background video at the same time.

[0126] The parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first label data.

[0127] For example, the first sample foreground video and the first sample background video are videos used to train the video synthesis module. The first sample foreground video includes the target subject. (See reference...) Figure 2 The first sample foreground video may also include transparency channel data, i.e., alpha channel data, to assist the video synthesis model in identifying the region where the target subject is located in the first sample foreground video. The first label data is used for supervised training of the video synthesis model to serve as a reference for adjusting the parameters of the video synthesis model and optimizing its performance.

[0128] For example, a sample video frame is a single video frame from the same moment in both the first sample foreground video and the first sample background video. A sample composite frame is the desired composite result of the sample video frames from the first sample foreground video and the first sample background video at the same moment. Specifically, sample video frames corresponding to a specific moment can be extracted beforehand from the first sample foreground video and the first sample background video. These frames can be pre-processed based on model recognition or manually labeled to obtain sample composite frames that conform to the actual scene. These sample composite frames are then used as label data to train the video synthesis model.

[0129] For example, a loss function can be calculated based on the difference between the sample synthesized frame and the synthesized frame predicted by the model. The parameters of the video synthesis model are then updated using a backpropagation algorithm. Alternatively, the parameters of the feature extraction model can be updated only, along with those of the storage module, location prediction module, and synthesis module, to minimize the loss function. This process continues until the parameters of the video synthesis model converge, i.e., the difference between the synthesized frame predicted by the model and the sample synthesized frame is minimized, or until a preset number of iterations is reached to obtain a trained video synthesis model.

[0130] As an optional embodiment, the sample synthesis frame includes the actual location of the target subject in the first sample foreground video in the first sample background video;

[0131] Based on the first sample foreground video, the first sample background video, and the first label data, the parameters of the video synthesis model are updated, including:

[0132] Based on sample video frames at the same time in the first sample foreground video and the first sample background video, a predicted composite frame is obtained. The predicted composite frame includes the predicted position of the target subject in the first sample foreground video in the first sample background video.

[0133] Determine the first loss value based on the actual location and the predicted location;

[0134] The second loss value is determined based on the sample synthesized frame and the predicted synthesized frame;

[0135] The third loss value is determined based on the synthesized frames of samples from adjacent time points and the time consistency loss function;

[0136] The parameters of the video synthesis model are updated based on the first, second, and third loss values.

[0137] For example, a sample composite frame is a frame synthesized from video frames at the same time in the first sample foreground video and the first sample background video. The sample composite frame is a synthesized result that conforms to actual expectations, where the target subject is located in the expected real position in the sample composite frame. The real position is the actual position of the target subject in the first sample background video, provided as label data. A predicted composite frame is a frame synthesized by the video synthesis model based on sample video frames in the first sample foreground video and the first sample background video, where the target subject is located at the position predicted by the model.

[0138] For example, the first loss value is obtained based on the difference between the actual and predicted positions of the target object. The second loss value is calculated based on the difference between the sampled composite frame and the predicted composite frame. The third loss value is calculated based on the sampled composite frames at adjacent time points and the temporal consistency loss function, which can be used to maintain the temporal continuity between video frames.

[0139] For example, the first loss value L pos It can be obtained based on the following formula:

[0140]

[0141] Where T represents duration, () pred Indicates the predicted position; where, in the preset coordinate system, x t It can represent the x-coordinate of the upper left corner of the rectangular region at time t, y t w can represent the ordinate of the top-left corner of the rectangular region at time t. t h can represent the width of the rectangular region at time t. t It can represent the height of the rectangular region at time t; () gt The actual location can be pre-labeled; ||·||1 represents the L1 norm.

[0142] For example, the second loss value L vgg The VGG feature extractor can be used to calculate the perceptual difference between the sample synthesized frame and the predicted synthesized frame.

[0143] For example, Temporal Consistency Loss is a loss function used in video processing tasks such as video matting, video compositing, and video harmonization. Using Temporal Consistency Loss ensures that transitions between video frames are natural and smooth, without abrupt changes, thereby improving the naturalness and realism of the video.

[0144] For example, the temporal consistency loss function can calculate the differences between adjacent composite frames in a video sequence. This can be done by calculating pixel-level differences between adjacent composite frames; if the pixel differences between two frames are too large, it indicates poor continuity between video frames, and the temporal consistency loss function will penalize this. Alternatively, it can calculate the differences between the feature representations of the composite frames. Here, pre-trained CNNs (Convolutional Neural Networks) can be used to extract the feature representations of adjacent composite frames, focusing on semantic-level temporal consistency. Furthermore, recurrent neural networks (RNNs) such as LSTM or GRU can be used as temporal encoders to encode the temporal information of video frames, and the temporal consistency between video frames can be evaluated by comparing the outputs of the temporal encoders for adjacent composite frames.

[0145] For example, the third loss value L temp It can be obtained based on the following formula:

[0146]

[0147] Where, d pred This represents the temporal information of the synthesized frames of samples from adjacent time points in the prediction; d gt The temporal information of the sample composite frame representing the actual adjacent time points can be obtained by pre-annotation; ||·||2 represents the L2 norm.

[0148] For example, the total loss value can be calculated based on the first loss value, the second loss value, and the third loss value. The total loss value L can be obtained based on the following formula. total :

[0149] L total =λ1L pos +λ2L vgg +λ3L temp

[0150] Wherein, λ1 is the weight parameter corresponding to the first loss value, λ2 is the weight parameter corresponding to the second loss value, and λ3 is the weight parameter corresponding to the third loss value, all of which can be preset according to the actual situation.

[0151] For example, the parameters of the video synthesis model can be updated using optimization algorithms such as gradient descent based on the total loss value corresponding to the first, second, and third loss values, in order to train the video synthesis model.

[0152] As an optional embodiment, the parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data, including:

[0153] Random data augmentation is performed on the first sample foreground video to adjust the hue and / or saturation information of the first sample foreground video to obtain the second sample foreground video;

[0154] The parameters of the video synthesis model are updated based on the foreground video of the second sample, the background video of the first sample, and the first label data.

[0155] For example, the second foreground video sample is a foreground video sample obtained by random data augmentation of the first foreground video sample. Random data augmentation can process existing data in a random manner to generate new data samples. Here, through random data augmentation, the tone information and / or saturation information of the image or video frame are randomly changed, which can produce a significant difference from the first background video sample, simulating different lighting and color conditions, thereby increasing the diversity of the training data.

[0156] For example, the parameters of the video synthesis model are trained and updated using the augmented second foreground video, the original first background video, and the first label data. The process of randomly augmenting the first foreground video and updating the parameters of the video synthesis model can be repeated until the performance of the video synthesis model reaches the desired synthesis effect.

[0157] Based on this, through random data augmentation, the model can synthesize the second sample foreground video and the first sample background video under different hue and saturation conditions, improving the model's adaptability and robustness to different environments, enhancing the video synthesis model's generalization ability to synthesize foreground and background videos with different color features, and improving the quality of synthesis.

[0158] In addition, data augmentation can effectively increase the number of training samples, reduce the risk of overfitting in video synthesis models, and make video synthesis models more stable in practical applications.

[0159] As an optional embodiment, feature extraction is performed on target video frames at the same time in the second foreground video and the second background video to obtain the features of the first video frame, including:

[0160] The first video frame features are obtained by extracting features from the target video frames at the same time in the second foreground video and the second background video through the matching and scoring module.

[0161] The method also includes:

[0162] The matching and scoring module obtains a matching reference value based on the features of the first video frame. The matching reference value is used to characterize the degree of matching between the second foreground video and the second background video.

[0163] Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized, including:

[0164] When the matching reference value is greater than the first preset reference threshold, the second foreground video and the second background video are synthesized based on the features of the first video frame and the features of the second video frame.

[0165] For example, the matching scoring module is used to evaluate the degree of matching between the second foreground video and the second background video, and to extract features of the target video frame at the same time in both the second foreground video and the second background video. The matching reference value is a value generated by the matching scoring module, used to quantify the degree of matching between the second foreground video and the second background video frame.

[0166] Here, by matching the features extracted by the scoring module and the matching reference values, the matching degree between the foreground video and the background video can be evaluated more accurately, thereby improving the quality of the synthesized video.

[0167] For example, when the matching reference value is greater than the first preset reference threshold, the matching degree of the target video frames at the same time in the second foreground video and the second background video is high, and the synthesis effect is good. The second foreground video and the second background video can continue to be synthesized based on the features of the first and second video frames. When the matching reference value is less than or equal to the first preset reference threshold, the matching degree of the target video frames at the same time in the second foreground video and the second background video is low, and the synthesis effect may not meet expectations. The second foreground video and the second background video can no longer be synthesized.

[0168] Here, using a first preset reference threshold to determine whether to continue synthesizing the second foreground video and the second background video can reduce unnecessary calculations and improve the efficiency of the video synthesis process.

[0169] Based on this, the second foreground video and the second background video are synthesized only when the matching reference value is greater than the first preset reference threshold, so that the video synthesis effect is better, that is, the video effect of the second target video is more realistic. In this way, the video matting model is trained based on the real second target video, and the resulting video matting model can increase its understanding and generalization ability of real scenes, thereby improving the effect of video matting.

[0170] As an optional implementation, training the matching scoring model includes:

[0171] Acquire the third sample foreground video, the second sample background video, and the second label data. The second label data includes the true score used to characterize the degree of matching between the third sample foreground video and the second sample background video.

[0172] The predicted score is obtained by matching the scoring model based on the foreground video of the third sample and the background video of the second sample.

[0173] The parameters of the matching scoring model are updated based on the actual and predicted scores.

[0174] For example, the third foreground video and the second background video can be used to train the matching scoring model. The second label data includes the true score, which characterizes the degree of matching between the third foreground video and the second background video, and can be used to supervise the training of the matching scoring model. The predicted score is the score predicted by the matching scoring model based on the third foreground video and the second background video.

[0175] For example, the second label data can be preset. For instance, if the third sample foreground video and the second sample background video are foreground and background videos separated from the same video, then the third sample foreground video and the second sample background video can be considered a perfect match, and the true score can be set to 1. For instance, if the third sample foreground video and the second sample background video are random and unrelated foreground and background videos, then the third sample foreground video and the second sample background video can be considered a complete mismatch, and the true score can be set to 0. For instance, if the second sample background video is a scene video similar to the original background video of the third sample foreground video, then the true score can be set to any value between 0.3 and 0.9, depending on the actual situation; wherein, based on the original video, image matting can be performed to obtain the third sample foreground video and the original background video, where the second sample background video has similar colors and lighting conditions to the original background video but a different background.

[0176] For example, a loss value is calculated based on the true score and the predicted score. The loss function can be the mean squared error (MSE) or the mean absolute error (MAE). Using the loss value between the true and predicted scores, the parameters of the matching scoring model are updated using an optimization algorithm such as gradient descent to reduce the difference between the predicted and true scores until the parameters of the matching scoring model converge.

[0177] Here, the parameters of the matching scoring model are updated based on the true scores in the second label data to make the predicted scores output by the matching scoring model closer to the true scores, thereby improving the accuracy of the matching scoring model. Accurate matching scores can be used to select foreground and background videos with high matching degrees during video synthesis, thereby improving the realism of the synthesized video. Therefore, training the video matting model based on the real second target video results in a video matting model with increased understanding and generalization ability of real scenes, improving the effect of video matting.

[0178] In related technologies, video matting models can be trained using either video segmentation data or video matting data. However, video segmentation data suffers from poor edge detail, while video matting data is limited in quantity, has high annotation costs, and suffers from limited category diversity.

[0179] As an optional implementation, the training of the video matting model also includes:

[0180] The video matting model is trained by alternating between a first sample dataset and a second sample dataset; wherein the first sample dataset includes image segmentation data and / or video segmentation data, and the second sample dataset includes image matting data and / or video matting data.

[0181] For example, the first sample dataset contains image segmentation data and / or video segmentation data, and the second sample dataset contains image matting data and / or video matting data. With the first sample dataset, the video matting model can learn how to identify and separate the basic structures of the foreground and background. With the second sample dataset, the video matting model can learn how to accurately handle the edges and details of the target subject.

[0182] For example, annotations can be based on real-world video scenes to increase the generalization ability of the video matting model. The data annotation tools should vary depending on the data type and application scenario. The sample data used to train the video matting model includes image segmentation data, image matting data, video segmentation data, and video matting data. The annotation cost and difficulty of image segmentation data, image matting data, video segmentation data, and video matting data increase progressively.

[0183] During training, the simple matting data may lead to overfitting of the trained video matting model due to its limited scope, small data volume, and the presence of a large amount of poorly rendered synthetic data. To avoid overfitting, the first and second sample datasets are used alternately during training to enhance the model's semantic understanding capabilities, enabling it to adapt to more diverse scenarios and conditions, and improving its generalization ability and robustness.

[0184] The image segmentation and / or video segmentation data in the first sample dataset can make the training and testing data more representative, thereby improving the model's generalization ability. The image matting and / or video matting data in the second sample dataset can separate the target subject from the background data, reduce the interference of background noise on the video matting model, and more accurately identify the edges and details of the foreground object, thus improving the matting quality.

[0185] As an optional embodiment, a preset image segmentation model can be used to output image segmentation data corresponding to the target image to be segmented.

[0186] For example, for a target image containing only a single-category subject, the Grounded SAM vision application can be used to obtain image segmentation data, where the image segmentation data is alpha channel data. That is, by inputting a prompt word corresponding to the category of the target subject, such as "prompt" (person, cat, dog, car), into the SAM-HQ model, the model can automatically obtain image segmentation data. This method is simple, efficient, and highly accurate for single-category targets.

[0187] For example, if the target image contains multiple categories of subjects, such as vehicles and people in a single frame, the accuracy of automatically extracting the target subject from multiple categories by inputting instructions is not high. In such cases, a high-resolution binary segmentation network such as Birefnet or MVANet can be used for image segmentation. High-resolution binary segmentation networks are not sensitive to subject categories and can automatically extract the target subject.

[0188] For example, for image data with simple backgrounds and easily distinguishable subjects, a fully automated image segmentation model can be used followed by manual selection. This allows for the manual filtering out of inaccurate image segmentation data, resulting in lower costs.

[0189] For example, for image data with complex backgrounds and subjects that are difficult to distinguish, the subject outline can be determined by manually adjusting the segmented data using a click-based method. The corresponding image segmentation model can be the SAM model, which ensures semantic accuracy.

[0190] As an optional embodiment, the image matting data is obtained in the following way:

[0191] Based on the image segmentation data, a tricolor map is generated. The tricolor map is used to indicate the foreground region, background region, and transition region between the foreground and background regions in the image segmentation data.

[0192] Based on the tri-color image and the image diffusion model, image matting data is obtained. The image diffusion model is used to gradually add noise to the tri-color image until it becomes a pure noise image, and then gradually remove noise from the pure noise image until the image matting data is obtained.

[0193] For example, image segmentation data refers to the separation data of foreground and background obtained through image segmentation methods, typically containing the contour and location information of the target subject in the foreground. Trimaps can be used to assist in image matting, containing defined foreground regions, defined background regions, and uncertain transition regions. Image diffusion models generate images with different noise levels by simulating the noise diffusion process in an image, or recover the original image from a noisy image.

[0194] For example, even with detailed image segmentation data processing, the precision cannot be very high. Image matting data includes the foreground and background, as well as the transition area between them. Specifically, image matting data can be matted data from images with dense hair, such as human hair strands or images of cats and dogs with dense fur.

[0195] For example, image segmentation data can be used to generate a corresponding tri-color image. In the tri-color image, white is typically used to represent a defined foreground region, black to represent a defined background region, and gray to represent the transition region between the foreground and background. Then, using an image diffusion model, noise is gradually added to the tri-color image to simulate the process from a clear image to a purely noisy image. The noise in the purely noisy image is then gradually removed using the image diffusion model to recover clear image matting data.

[0196] As an example, for data with a lot of hair, a tri-color image can be generated, and then inference can be performed using an image diffusion model to obtain image matting data. The image diffusion model can be the DiffMatte model, which progressively adds Gaussian noise to the tri-color image until a purely noisy image is obtained. Then, by progressively removing the Gaussian noise from the purely noisy image, clear image matting data containing alpha channel data is generated.

[0197] Therefore, image matting data obtained through the image diffusion model can capture image details, resulting in high precision. Consequently, video matting models trained based on this image matting data exhibit high accuracy.

[0198] As an optional embodiment, the image matting data is obtained in the following way:

[0199] Based on the image segmentation data, a mask map is generated. The mask map is used to indicate the foreground and background regions in the image segmentation data.

[0200] Based on the mask image and the preset image matting model, image matting data is obtained. The image matting model is used to extract global and local features of the image segmentation data, and the image matting data is obtained through the global and local features.

[0201] For example, a mask is a binary image that can be used to assist in image matting. The mask indicates the foreground and background regions in the image segmentation data. Image matting models can be used to extract global and local features from image segmentation data and generate matted image data.

[0202] For example, for data where hair features are not obvious, a mask-based approach can be used, and image matting data can be generated through a preset image matting model. This method is simple and has a high success rate.

[0203] For example, image segmentation data can be used to generate a corresponding mask. Each pixel in the mask is labeled as belonging to the foreground or background, typically represented by 0 and 1 (or black and white). Using a pre-defined image matting model, such as the CascadePSP model, global and local features can be extracted from the mask and image segmentation data. Global features may include the image's color distribution, texture, etc., while local features may include edge information, local texture, etc. Combining global and local features can generate accurate image matting data, which may include the precise outline of the foreground, transparency information, etc.

[0204] As an optional implementation, the video segmentation data is obtained in the following manner:

[0205] Obtain the first sample video to be segmented, which includes at least one video frame labeled with the target subject;

[0206] Video segmentation data is obtained by performing forward and / or backward propagation based on the first sample video and at least one video frame using a pre-defined video segmentation model.

[0207] For example, the first sample video is the raw video data used for the video segmentation task, which contains the target subject. The target subject is the object that has been marked or labeled in the video frame, which can be done manually or automatically detected and labeled based on image and video algorithms. The video segmentation data is the segmentation data from the first sample video, including foreground and background information. A preset video segmentation model can be used to automatically segment the target subject and background from the video frame.

[0208] For example, at least one frame containing the labeled target subject can be obtained from the first sample video. This frame or several frames containing the labeled target subject can be used to guide the segmentation process of the entire video. Video segmentation data can be obtained by performing forward and / or backward propagation based on the first sample video and at least one video frame using a preset video segmentation model. For example, forward or backward propagation can be performed using models such as XMEM, Cuite, and SAM2. For video frames whose segmentation results do not meet expectations, manual adjustments can be made in real time, and forward or backward propagation can be performed again until all video frames of the video are labeled. Although the edges of the labeled video may not be fine, the semantic information is strong and the continuity is good. Moreover, the parameters of the video segmentation model are not large, which greatly improves the labeling speed and reduces the cost of manual input.

[0209] As an optional embodiment, the video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The video matting data is obtained in the following way:

[0210] Obtain the second sample video to be cut out;

[0211] The initial video matting data is obtained based on the second sample video using the video matting model, and the difficulty prediction module is used to obtain the difficulty reference value based on the second sample video. The difficulty reference value is used to characterize the difficulty of performing video matting on the second sample video.

[0212] Based on the initial video cutout data, the difficulty reference value, and the second preset reference threshold, the video cutout data is determined.

[0213] For example, the second sample video is the original video data used for the video matting task. Based on the matting processing of the second sample video, video matting data can be obtained for training the video matting model. The video matting model used to process the second sample video can be the same as or different from the video matting model trained using the video matting data.

[0214] For example, the difficulty prediction module is used to determine the difficulty of video matting. Based on the difficulty prediction module's assessment, it can be determined whether the video matting model should continue. The difficulty reference value is the output of the difficulty prediction module, used to quantify the difficulty of the second sample video matting task. For instance, if the difficulty prediction module determines that the current video matting is difficult (i.e., the difficulty reference value is greater than a threshold), it can output a prompt message, suggesting to the user that the matting effect may be poor, or directly recommending that the user not perform the matting process.

[0215] For example, the initial video matting data is a preliminary matting result obtained from a second sample video using a video matting model. The second preset reference threshold can be a threshold set according to the actual situation, used to determine whether the initial video matting data can be used as the final video matting data based on a difficulty reference value.

[0216] As an optional embodiment, video matting data is determined based on initial video matting data, a difficulty reference value, and a second preset reference threshold, including:

[0217] When the difficulty reference value is less than or equal to the second preset reference threshold, the initial video cutout data will be used as the video cutout data.

[0218] When the difficulty reference value is greater than the second preset reference threshold, the initial video cutout data is adjusted to obtain the video cutout data.

[0219] For example, the second sample video can be processed by a video matting model to obtain initial video matting data, and the second sample video can be analyzed by a difficulty prediction module to output a difficulty reference value, which reflects the difficulty of the video matting task.

[0220] For example, a difficulty reference value is compared with a second preset reference threshold to determine whether further processing or optimization of the initial video matting data is needed. If the difficulty reference value is lower than or equal to the second preset reference threshold, the initial video matting data is likely of good quality and can be directly used as training data for the video matting model. If the difficulty reference value is higher than the second preset reference threshold, the initial video matting data is likely of poor quality and may require further adjustment, such as manual adjustment or the use of more complex matting techniques.

[0221] Therefore, by considering the difficulty of video matting, the matting results for high-difficulty videos can be adjusted to improve the quality of the final matting data, achieving high-precision and consistent video matting results. The difficulty prediction module enables the video matting model to adapt to video matting tasks of varying difficulty, improving its flexibility and adaptability.

[0222] As an optional embodiment, the method further includes:

[0223] Repeat the steps of obtaining initial video matting data based on the second sample video using the video matting model, and obtaining a difficulty reference value based on the second sample video using the difficulty prediction module;

[0224] When the number of repeated executions exceeds the preset number, the second preset reference threshold is adjusted to obtain the third preset reference threshold;

[0225] The process is repeated to obtain initial video matting data based on the second sample video using the video matting model, and to obtain a difficulty reference value based on the second sample video using the difficulty prediction module. Finally, the video matting data is determined based on the initial video matting data, the difficulty reference value, and the third preset reference threshold.

[0226] For example, the preset number of repetitions can be set according to the actual situation. The third preset reference threshold is a new threshold obtained by adjusting the second preset reference threshold after the number of repetitions exceeds the preset number.

[0227] For example, after the number of repetitions exceeds a preset number, the reference threshold corresponding to the difficulty can be gradually increased. Treating the difficulty judgment result corresponding to each reference threshold as a stage allows for differentiation between the difficulty judgment results of different stages. Repeating the above steps, in each stage, video matting data with a difficulty reference value less than or equal to the reference threshold is retained, while video matting data with a difficulty reference value greater than the reference threshold is re-labeled. After multiple executions, the difficulty of the video matting data will gradually increase, enabling the video matting model to handle difficult video data and cope with complex scenes. This method not only expands the training dataset but also improves model performance through self-supervised training and augmentation strategies, achieving a virtuous cycle between the model and the data.

[0228] As an optional embodiment, the video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The difficulty is used to assist the video matting model in determining whether to perform matting processing on the first target video. The training of the video matting model includes:

[0229] Obtain the third sample dataset, which includes target data whose difficulty labeling value exceeds the third preset reference threshold, and the proportion of the number of target data in the third sample dataset is greater than the preset proportion;

[0230] Based on the third sample dataset, update the parameters of the difficulty prediction module, while keeping the parameters of other modules in the video matting model unchanged.

[0231] For example, the third sample dataset is used to train the difficulty prediction module. It contains multiple video samples and their corresponding difficulty annotation values. The difficulty values ​​in the third sample dataset can be manually annotated beforehand to train the difficulty prediction module. The preset ratio can be set according to actual needs; for example, the preset ratio can be 8:2 or 7:3.

[0232] For example, the proportion of high-difficulty samples in the third sample dataset is greater than a preset proportion to provide sufficient high-difficulty samples for training. During training, the parameters of other modules in the video matting model, except for the difficulty prediction module, are kept unchanged. The parameters of the difficulty prediction module are updated using the third sample dataset to focus on optimizing the difficulty prediction capability, which can improve the prediction accuracy of the difficulty prediction module.

[0233] For example, the difficulty prediction module can perform detailed analysis of the prediction results for video frames to evaluate whether the model can accurately separate the foreground and background, especially for scenes with blurred edges, severe occlusion, or complex backgrounds, where image matting is often more difficult. The difficulty score comprehensively considers the accuracy and completeness of the matting results, particularly in detail processing and edge sharpness. However, the subject extraction dataset contains only data with clearly defined subjects, making it impossible to supervise the difficulty branch.

[0234] For example, by retraining the difficulty prediction module, the dataset used, in addition to the subject extraction dataset, also includes a lot of high-difficulty data, such as crowds, that the video matting model might voluntarily abandon. High-difficulty data cannot be filtered by category judgment, but it can be filtered by the difficulty prediction module. However, obtaining mask labels for such complex backgrounds is more difficult than for ordinary scenes. Correspondingly, the mask labels for high-difficulty data can be set to completely black images, which satisfies the training task of the difficulty prediction module without any annotation cost.

[0235] Based on this, the difficulty prediction module is trained with high-difficulty samples to improve its prediction accuracy, enabling the video matting model to better adapt to video matting tasks of varying difficulty and improve matting quality.

[0236] As an optional embodiment, the video matting model further includes a classification module, which is used to identify the type of target subject in the first target video. The type is used to assist the video matting model in matting the first target video. The training of the video matting model also includes:

[0237] Obtain the fourth sample dataset, which includes data of multiple subject types, with each subject type having the same proportion of data in the fourth sample dataset;

[0238] Based on the fourth sample dataset, update the parameters of the classification module, while keeping the parameters of other modules in the video matting model unchanged.

[0239] For example, the classification module is used to identify the type of target subject in the video. Subject type is the category of the target subject in the video, such as people, animals, vehicles, etc. The fourth sample dataset is the dataset used to train the classification module, containing video samples of multiple different types of subjects, and the proportion of video samples corresponding to each type of subject in the fourth sample dataset is the same. For example, the proportion of people, cats, dogs, cars, and other types is 1:1:1:1:1.

[0240] For example, during the training of a video matting model, the parameters of all modules except the classification module can be kept unchanged, while the parameters of the classification module are updated to focus on optimizing the classification module's ability to recognize different subject types. Specifically, the classification module is trained using a fourth sample dataset, where the proportion of video samples corresponding to each subject type in the fourth sample dataset is equal. This avoids bias in the classification module towards any particular type of data, thereby improving the accuracy of recognizing different subject types.

[0241] For example, when training a video matting model, the ratio of data for the human, cat, dog, and vehicle categories to data for other categories in the training data is 9:1. This will severely bias the trained model towards the human, cat, dog, and vehicle categories, increasing false positives. To improve the accuracy of the classification module, it can be trained in one step. In the fourth sample dataset, the ratio of each subject type is approximately 1:1:1:1:1, and subject extraction is unnecessary. Therefore, there is no need to set mask labels for other subject categories; the mask labels can be set to completely black images, significantly reducing annotation costs. Furthermore, the subcategories within other categories can be enriched as much as possible, including background data, various objects, other animals, and images with special shapes.

[0242] As an optional embodiment, the method further includes:

[0243] The first foreground video, the first background video, and the target component are synthesized to obtain a third target video for display on the lock screen or desktop interface, wherein the target component is a lock screen component or a desktop component.

[0244] For example, in the case of a dynamic depth-of-field wallpaper scenario, that is, the user-uploaded video or live photo is cut out, the extracted video foreground is set on the top layer of the lock screen component, and the original video background is merged to form a new video lock screen effect. The first foreground video and the first background video obtained by the video cutout model disclosed herein are combined with the lock screen component or desktop component expected by the user to synthesize a third target video displayed on the lock screen interface or desktop interface.

[0245] Based on this, after training the video matting model with a more realistic second target video obtained by the video synthesis model, the matting accuracy of the video matting model is higher. As a result, the third target video synthesized based on the first foreground video, the first background video, and the lock screen component or desktop component can better meet the user's expectations and improve the user experience.

[0246] As a concrete example, extracting the foreground from videos recorded in front of a green screen is relatively easy, so most keying data only includes the original video and alpha channel data. To suit practical video keying models, the foreground of these green screen videos can be re-composited onto our subsequently captured background video while maintaining realism.

[0247] See Figure 3 The diagram illustrates the architecture of the matching scoring model. The matching scoring module establishes a one-to-one relationship between the foreground and background videos to be synthesized by scoring the matching degree between the training foreground and background videos. The processing flow of the matching scoring module includes video content analysis, image completion, and supervised training.

[0248] When training the matching and scoring module, real-world scene matting or segmentation data can be used to extract foreground video as input to the matching score model. Specifically, based on the original video and alpha channel data, the foreground region of each frame in the original video is hollowed out, and an image completion model, such as LaMa or DeepFillv2, is used to complete the foreground region, generating a new, subjectless, completely background video. This completely background video is then used as input to the matching and scoring module. Additionally, the alpha channel data is also used as input to guide the matching and scoring module in focusing on the region where the target subject is located for scoring. The final output is a value between 0 and 1, representing the degree of matching between the foreground and background videos.

[0249] In this system, the foreground and background videos are input into different multi-scale encoders. Alpha is used as a reference image and is added to the encoded features through an attention module. After feature extraction, the images are fed into a fully connected layer to output the final matching reference value. During training, preset real labels can be used to supervise the results. The loss is the mean squared error loss, and the diversity of the data is enhanced through color perturbation and video frame editing.

[0250] After the matching and scoring module is trained, it can be used to perform pairwise inference and scoring on all the foreground and background videos that need to be synthesized.

[0251] In another example, the matching and scoring module can also be image-based, i.e., for matching foreground and background videos, the matching and scoring module can score video frames in the foreground and background videos at the same time. After obtaining the score result of each video frame, the average of multiple score results is calculated as the matching score result of the foreground and background videos. Finally, the background video with the highest matching degree with the foreground video is selected and input into the video synthesis model for video synthesis. The result synthesized by the video synthesis model is then used for training the video segmentation model.

[0252] Reference Figure 4 , Figure 4 This is a block diagram illustrating a video processing apparatus 400 according to an exemplary embodiment. Figure 4 As shown, the video processing device 400 includes an acquisition module 401 and an acquisition module 402.

[0253] The acquisition module 401 is configured to acquire the first target video to be processed.

[0254] The module 402 is configured to perform matting processing on the first target video using a video matting model to obtain a first foreground video and a first background video; the video matting model is trained at least based on a second target video output by a video synthesis model, and the video synthesis model is used to obtain the second target video in the following manner:

[0255] Feature extraction is performed on the target video frame at the same moment in the second foreground video and the second background video to obtain the first video frame feature, and the second video frame feature corresponding to the historical video frame at the same moment in the second foreground video and the second background video is obtained, wherein the historical video frame is located before the target video frame;

[0256] Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized to obtain the second target video.

[0257] As an optional embodiment, the second target video is obtained by synthesizing the second foreground video and the second background video based on the features of the first video frame and the features of the second video frame, including:

[0258] Based on the features of the first video frame, predict the target position of the target in the second foreground video in the second background video;

[0259] Based on the features of the first video frame, the features of the second video frame, and the target position, the target video frames at the same time in the second foreground video and the second background video are synthesized to obtain the target composite frame.

[0260] The second target video is obtained based on the target composite frame.

[0261] As an optional embodiment, the video synthesis model includes a feature extraction module, a storage module, a position prediction module, and a synthesis module. The feature extraction module is used to extract features from target video frames at the same time in the second foreground video and the second background video to obtain first video frame features. The storage module is used to store second video frame features. The position prediction module is used to predict the target position of the target subject in the second foreground video in the second background video based on the first video frame features. The synthesis module is used to synthesize the target video frames at the same time in the second foreground video and the second background video based on the first video frame features, the second video frame features, and the target position to obtain a target synthesized frame.

[0262] Training the video synthesis model includes:

[0263] Keeping the parameters of the feature extraction module unchanged, and updating the parameters of the storage module, location prediction module, and synthesis module, a video synthesis model is obtained.

[0264] As an optional implementation, training the video synthesis model includes:

[0265] Acquire the first sample foreground video, the first sample background video, and the first label data. The first label data includes a composite frame of sample video frames from the first sample foreground video and the first sample background video at the same time.

[0266] The parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first label data.

[0267] As an optional embodiment, the sample synthesis frame includes the actual location of the target subject in the first sample foreground video in the first sample background video;

[0268] Based on the first sample foreground video, the first sample background video, and the first label data, the parameters of the video synthesis model are updated, including:

[0269] Based on sample video frames at the same time in the first sample foreground video and the first sample background video, a predicted composite frame is obtained. The predicted composite frame includes the predicted position of the target subject in the first sample foreground video in the first sample background video.

[0270] Determine the first loss value based on the actual location and the predicted location;

[0271] The second loss value is determined based on the sample synthesized frame and the predicted synthesized frame;

[0272] The third loss value is determined based on the synthesized frames of samples from adjacent time points and the time consistency loss function;

[0273] The parameters of the video synthesis model are updated based on the first, second, and third loss values.

[0274] As an optional embodiment, the parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data, including:

[0275] Random data augmentation is performed on the first sample foreground video to adjust the hue and / or saturation information of the first sample foreground video to obtain the second sample foreground video;

[0276] The parameters of the video synthesis model are updated based on the foreground video of the second sample, the background video of the first sample, and the first label data.

[0277] As an optional embodiment, feature extraction is performed on target video frames at the same time in the second foreground video and the second background video to obtain the features of the first video frame, including:

[0278] The first video frame features are obtained by extracting features from the target video frames at the same time in the second foreground video and the second background video through the matching and scoring module.

[0279] The video processing device 400 is also configured to:

[0280] The matching and scoring module obtains a matching reference value based on the features of the first video frame. The matching reference value is used to characterize the degree of matching between the second foreground video and the second background video.

[0281] Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized, including:

[0282] When the matching reference value is greater than the first preset reference threshold, the second foreground video and the second background video are synthesized based on the features of the first video frame and the features of the second video frame.

[0283] As an optional implementation, training the matching scoring model includes:

[0284] Acquire the third sample foreground video, the second sample background video, and the second label data. The second label data includes the true score used to characterize the degree of matching between the third sample foreground video and the second sample background video.

[0285] The predicted score is obtained by matching the scoring model based on the foreground video of the third sample and the background video of the second sample.

[0286] The parameters of the matching scoring model are updated based on the actual and predicted scores.

[0287] As an optional implementation, the training of the video matting model also includes:

[0288] The video matting model is trained by alternating between a first sample dataset and a second sample dataset; wherein the first sample dataset includes image segmentation data and / or video segmentation data, and the second sample dataset includes image matting data and / or video matting data.

[0289] As an optional embodiment, the image matting data is obtained in the following way:

[0290] Based on the image segmentation data, a tricolor map is generated. The tricolor map is used to indicate the foreground region, background region, and transition region between the foreground and background regions in the image segmentation data.

[0291] Based on the tri-color image and the image diffusion model, image matting data is obtained. The image diffusion model is used to gradually add noise to the tri-color image until it becomes a pure noise image, and then gradually remove noise from the pure noise image until the image matting data is obtained.

[0292] As an optional embodiment, the image matting data is obtained in the following way:

[0293] Based on the image segmentation data, a mask map is generated. The mask map is used to indicate the foreground and background regions in the image segmentation data.

[0294] Based on the mask image and the preset image matting model, image matting data is obtained. The image matting model is used to extract global and local features of the image segmentation data, and the image matting data is obtained through the global and local features.

[0295] As an optional implementation, the video segmentation data is obtained in the following manner:

[0296] Obtain the first sample video to be segmented, which includes at least one video frame labeled with the target subject;

[0297] Video segmentation data is obtained by performing forward and / or backward propagation based on the first sample video and at least one video frame using a pre-defined video segmentation model.

[0298] As an optional embodiment, the video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The video matting data is obtained in the following way:

[0299] Obtain the second sample video to be cut out;

[0300] The initial video matting data is obtained based on the second sample video using the video matting model, and the difficulty prediction module is used to obtain the difficulty reference value based on the second sample video. The difficulty reference value is used to characterize the difficulty of performing video matting on the second sample video.

[0301] Based on the initial video cutout data, the difficulty reference value, and the second preset reference threshold, the video cutout data is determined.

[0302] As an optional embodiment, video matting data is determined based on initial video matting data, a difficulty reference value, and a second preset reference threshold, including:

[0303] When the difficulty reference value is less than or equal to the second preset reference threshold, the initial video cutout data will be used as the video cutout data.

[0304] When the difficulty reference value is greater than the second preset reference threshold, the initial video cutout data is adjusted to obtain the video cutout data.

[0305] As an optional embodiment, the video processing device 400 is also configured to:

[0306] Repeat the steps of obtaining initial video matting data based on the second sample video using the video matting model, and obtaining a difficulty reference value based on the second sample video using the difficulty prediction module;

[0307] When the number of repeated executions exceeds the preset number, the second preset reference threshold is adjusted to obtain the third preset reference threshold;

[0308] The process is repeated to obtain initial video matting data based on the second sample video using the video matting model, and to obtain a difficulty reference value based on the second sample video using the difficulty prediction module. Finally, the video matting data is determined based on the initial video matting data, the difficulty reference value, and the third preset reference threshold.

[0309] As an optional embodiment, the video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The difficulty is used to assist the video matting model in determining whether to perform matting processing on the first target video. The training of the video matting model includes:

[0310] Obtain the third sample dataset, which includes target data whose difficulty labeling value exceeds the third preset reference threshold, and the proportion of the number of target data in the third sample dataset is greater than the preset proportion;

[0311] Based on the third sample dataset, update the parameters of the difficulty prediction module, while keeping the parameters of other modules in the video matting model unchanged.

[0312] As an optional embodiment, the video matting model further includes a classification module, which is used to identify the type of target subject in the first target video. The type is used to assist the video matting model in matting the first target video. The training of the video matting model also includes:

[0313] Obtain the fourth sample dataset, which includes data of multiple subject types, with each subject type having the same proportion of data in the fourth sample dataset;

[0314] Based on the fourth sample dataset, update the parameters of the classification module, while keeping the parameters of other modules in the video matting model unchanged.

[0315] As an optional embodiment, the video processing device 400 is also configured to:

[0316] The first foreground video, the first background video, and the target component are synthesized to obtain a third target video for display on the lock screen or desktop interface, wherein the target component is a lock screen component or a desktop component.

[0317] Regarding the video processing apparatus 400 in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the video processing method, and will not be elaborated upon here.

[0318] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the video processing method provided in this disclosure.

[0319] Based on the same inventive concept, this disclosure also provides an electronic device, comprising:

[0320] Storage device for storing computer programs;

[0321] An execution device is used to execute the computer program to implement the video processing method provided in this disclosure.

[0322] Figure 5 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0323] Reference Figure 5 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output interface 812, sensor component 814, and communication component 816.

[0324] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the video processing method described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0325] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0326] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0327] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0328] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0329] Input / output interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0330] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0331] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0332] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the video processing method described above.

[0333] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to complete the video processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0334] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the video processing method described above when executed by the programmable device.

[0335] Figure 6 This is a block diagram illustrating an apparatus 1900 for video processing according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server. (Refer to...) Figure 6 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the aforementioned video processing method.

[0336] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958. Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0337] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0338] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, include: Obtain the first target video to be processed; The first target video is processed by a video matting model to obtain a first foreground video and a first background video. The video matting model is trained based on at least the second target video output by the video synthesis model, and the video synthesis model is used to obtain the second target video in the following manner: Feature extraction is performed on the target video frame at the same moment in the second foreground video and the second background video to obtain the first video frame feature, and the second video frame feature corresponding to the historical video frame at the same moment in the second foreground video and the second background video is obtained, wherein the historical video frame is located before the target video frame; Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized to obtain the second target video.

2. The method according to claim 1, characterized in that, Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized to obtain the second target video, including: Based on the features of the first video frame, predict the target position of the target subject in the second foreground video in the second background video; Based on the first video frame features, the second video frame features, and the target position, target video frames at the same time in the second foreground video and the second background video are synthesized to obtain a target composite frame. The second target video is obtained based on the target composite frame.

3. The method according to claim 2, characterized in that, The video synthesis model includes a feature extraction module, a storage module, a position prediction module, and a synthesis module. The feature extraction module is used to extract features from target video frames at the same time in the second foreground video and the second background video to obtain the features of the first video frame. The storage module is used to store the features of the second video frame. The position prediction module is used to predict the target position of the target subject in the second foreground video in the second background video based on the features of the first video frame. The synthesis module is used to synthesize the target video frames at the same time in the second foreground video and the second background video based on the features of the first video frame, the features of the second video frame, and the target position to obtain the target synthesized frame. The training of the video synthesis model includes: Keeping the parameters of the feature extraction module unchanged, and updating the parameters of the storage module, the location prediction module, and the synthesis module, the video synthesis model is obtained.

4. The method according to claim 1, characterized in that, The training of the video synthesis model includes: Acquire a first sample foreground video, a first sample background video, and first tag data. The first tag data includes a composite frame of sample video frames from the first sample foreground video and the first sample background video at the same time. The parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data.

5. The method according to claim 4, characterized in that, The sample composite frame includes the actual location of the target subject in the first sample foreground video in the first sample background video; The parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data, including: Based on sample video frames at the same time in the first sample foreground video and the first sample background video, a predicted composite frame is obtained, and the predicted composite frame includes the predicted position of the target subject in the first sample foreground video in the first sample background video. Based on the actual location and the predicted location, a first loss value is determined; The second loss value is determined based on the sample synthesized frame and the predicted synthesized frame; The third loss value is determined based on the synthesized frames of samples from adjacent time points and the time consistency loss function; The parameters of the video synthesis model are updated based on the first loss value, the second loss value, and the third loss value.

6. The method according to claim 4, characterized in that, The parameters of the video synthesis model are updated based on the first sample foreground video, the first sample background video, and the first tag data, including: Random data augmentation is performed on the first sample foreground video to adjust the hue and / or saturation information of the first sample foreground video to obtain the second sample foreground video; The parameters of the video synthesis model are updated based on the second sample foreground video, the first sample background video, and the first tag data.

7. The method according to claim 1, characterized in that, Feature extraction is performed on the target video frames at the same time in the second foreground video and the second background video to obtain the features of the first video frame, including: The first video frame features are obtained by extracting features from the target video frames at the same time in the second foreground video and the second background video through the matching and scoring module. The method further includes: The matching and scoring module obtains a matching reference value based on the features of the first video frame. The matching reference value is used to characterize the degree of matching between the second foreground video and the second background video. Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized, including: When the matching reference value is greater than the first preset reference threshold, the second foreground video and the second background video are synthesized based on the first video frame features and the second video frame features.

8. The method according to claim 7, characterized in that, The training of the matching scoring model includes: Acquire a third sample foreground video, a second sample background video, and second tag data. The second tag data includes a true score that characterizes the degree of matching between the third sample foreground video and the second sample background video. The matching scoring model obtains a predicted score based on the third sample foreground video and the second sample background video; The parameters of the matching scoring model are updated based on the actual score and the predicted score.

9. The method according to any one of claims 1-8, characterized in that, The training of the video matting model also includes: The video matting model is trained by alternately using a first sample dataset and a second sample dataset; wherein the first sample dataset includes image segmentation data and / or video segmentation data, and the second sample dataset includes image matting data and / or video matting data.

10. The method according to claim 9, characterized in that, The image matting data is obtained in the following way: Based on the image segmentation data, a tricolor map is generated, which is used to indicate the foreground region, the background region, and the transition region between the foreground region and the background region in the image segmentation data; The image matting data is obtained based on the tri-color image and the image diffusion model. The image diffusion model is used to gradually add noise to the tri-color image until it becomes a pure noise image, and then gradually remove noise from the pure noise image until the image matting data is obtained.

11. The method according to claim 9, characterized in that, The image matting data is obtained in the following way: A mask is generated based on the image segmentation data, and the mask is used to indicate the foreground and background regions in the image segmentation data; The image matting data is obtained based on the mask image and the preset image matting model. The image matting model is used to extract the global and local features of the image segmentation data, and the image matting data is obtained through the global and local features.

12. The method according to claim 9, characterized in that, The video segmentation data is obtained in the following way: Obtain a first sample video to be segmented, wherein the first sample video includes at least one video frame labeled with the target subject; The video segmentation data is obtained by performing forward and / or backward propagation based on the first sample video and the at least one video frame using a preset video segmentation model.

13. The method according to claim 9, characterized in that, The video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The video matting data is obtained in the following way: Obtain the second sample video to be cut out; The video matting model obtains initial video matting data based on the second sample video, and the difficulty prediction module obtains a difficulty reference value based on the second sample video. The difficulty reference value is used to characterize the ease or difficulty of performing video matting on the second sample video. The video cutout data is determined based on the initial video cutout data, the difficulty reference value, and the second preset reference threshold.

14. The method according to claim 13, characterized in that, The video cutout data is determined based on the initial video cutout data, the difficulty reference value, and the second preset reference threshold, including: When the difficulty reference value is less than or equal to the second preset reference threshold, the initial video cutout data is used as the video cutout data; When the difficulty reference value is greater than the second preset reference threshold, the initial video cutout data is adjusted to obtain the video cutout data.

15. The method according to claim 13, characterized in that, The method further includes: Repeat the steps of obtaining initial video matting data based on the second sample video using the video matting model, and obtaining a difficulty reference value based on the second sample video using the difficulty prediction module; When the number of repeated executions exceeds a preset number, the second preset reference threshold is adjusted to obtain a third preset reference threshold; The steps of obtaining initial video matting data based on the second sample video using the video matting model and obtaining a difficulty reference value based on the second sample video using the difficulty prediction module are executed again. The video matting data is then determined based on the initial video matting data, the difficulty reference value, and the third preset reference threshold.

16. The method according to any one of claims 1-8, characterized in that, The video matting model includes a difficulty prediction module, which is used to determine the difficulty of video matting. The difficulty is used to assist the video matting model in determining whether to perform matting processing on the first target video. The training of the video matting model includes: Obtain a third sample dataset, which includes target data whose difficulty labeling value exceeds a third preset reference threshold, and the proportion of the number of target data in the third sample dataset is greater than a preset proportion. Based on the third sample dataset, update the parameters of the difficulty prediction module, while keeping the parameters of other modules in the video matting model unchanged.

17. The method according to any one of claims 1-8, characterized in that, The video matting model further includes a classification module, which is used to identify the type of target subject in the first target video. The type is used to assist the video matting model in matting the first target video. The training of the video matting model also includes: Obtain a fourth sample dataset, which includes data of multiple subject types, and the proportion of each subject type in the fourth sample dataset is the same. Based on the fourth sample dataset, update the parameters of the classification module, while keeping the parameters of other modules in the video matting model unchanged.

18. The method according to any one of claims 1-8, characterized in that, The method further includes: The first foreground video, the first background video, and the target component are synthesized to obtain a third target video for display on the lock screen or desktop interface, wherein the target component is a lock screen component or a desktop component.

19. A video processing apparatus, characterized in that, include: The acquisition module is configured to acquire the first target video to be processed. The acquisition module is configured to perform image matting processing on the first target video using a video matting model to obtain a first foreground video and a first background video; The video matting model is trained based on at least the second target video output by the video synthesis model, and the video synthesis model is used to obtain the second target video in the following manner: Feature extraction is performed on the target video frame at the same moment in the second foreground video and the second background video to obtain the first video frame feature, and the second video frame feature corresponding to the historical video frame at the same moment in the second foreground video and the second background video is obtained, wherein the historical video frame is located before the target video frame; Based on the features of the first video frame and the features of the second video frame, the second foreground video and the second background video are synthesized to obtain the second target video.

20. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the video processing method according to any one of claims 1-18.

21. An electronic device, characterized in that, include: Storage device for storing computer programs; An execution device is used to execute the computer program to implement the video processing method according to any one of claims 1-18.

Images