Character background editing method and device, storage medium and computer equipment

Abstract

According to the character background editing method and device, the storage medium and the computer equipment provided by the invention, after the original video data with the character and the target background data without the character are obtained, the target video can be firstly determined to generate the large model; according to the target video generation large model, sample video data with figures are taken as sample labels, sample reference backgrounds after portrait elimination in the sample video data and sample reference data after synthesis of portrait masks extracted from the sample video data and sample background data without figures are taken as training samples; the target video generation large model is obtained after training, which is equivalent to that expected output is real and unmodified data when the target video generation large model is trained, so that after the background data in the original video data is replaced by the target background data through the target video generation large model, relatively real target video data can be obtained; and thus, the character background editing effect in the video creation scene is effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method, device, storage medium, and computer device for editing a person's background. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of AI technology in the field of video editing is becoming more and more extensive, and more and more self-media creators or ordinary users are trying to use AI technology for video creation. At present, in the field of video creation, replacing the background of a person has become an extremely common and frequent requirement.

[0003] The current mainstream video person background replacement technologies can be roughly divided into two categories: traditional color domain matte extraction and deep learning model matte extraction. The traditional color domain matte extraction has a high usage threshold, requiring users to have certain image processing capabilities, and there are many software parameters, which requires a lot of manpower and material resources to achieve matte extraction. And the person background editing technology based on deep learning usually needs to first use a semantic segmentation model to identify and extract the person mask, and then use a generative adversarial network to fuse the person with the new background, or directly fuse the person with the new background at the pixel level without using a model. Such technologies often have difficulty achieving high-quality fusion between the person and the new background. Especially when there are problems such as inconsistent proportions and lighting differences between the person and the target background, the existing algorithms often perform poorly and are difficult to meet the requirements of complex scenarios. Summary of the Invention

[0004] The purpose of this application aims to solve at least one of the above technical defects, especially the technical defect that the existing video person background replacement technology in the prior art cannot have both a low usage threshold and a better fusion effect.

[0005] This application provides a method for editing a person's background, and the method includes:

[0006] Obtain the original video data with a person and the target background data without a person;

[0007] Determine a target video generation large model, where the target video generation large model uses the sample video data with a person as a sample label, and uses the sample reference background after removing the person in the sample video data, and the sample reference data synthesized by combining the person mask extracted from the sample video data and the sample background data without a person as training samples, and is obtained after training;

[0008] Replace the background data in the original video data with the target background data through the target video generation large model to obtain the target video data.

[0009] Optionally, the training process of the target video generation large model includes:

[0010] Obtain sample video data with people and sample background data without people;

[0011] Extract the human figure mask from the sample video data, and eliminate the human figure mask from the sample video data to obtain a sample reference background after eliminating the human figure;

[0012] Synthesize the human figure mask and the sample background data to obtain sample reference data;

[0013] Replace the background data in the sample reference data with the sample reference background through a preset initial video generation large model to obtain predicted video data;

[0014] Take the predicted video data approaching the sample video data as the goal, and train the initial video generation large model until a preset end condition is reached;

[0015] Use the trained initial video generation large model as the target video generation large model.

[0016] Optionally, the extracting the human figure mask from the sample video data includes:

[0017] Obtain a human figure matting model;

[0018] Use the human figure matting model to extract the human figure mask from the sample video data.

[0019] Optionally, the eliminating the human figure mask from the sample video data to obtain a sample reference background after eliminating the human figure includes:

[0020] Obtain an image inpainting model;

[0021] Eliminate the human figure corresponding to the human figure mask in the sample video data through the image inpainting model to obtain a sample reference background after eliminating the human figure.

[0022] Optionally, the initial video generation large model includes an initial encoding module, an initial feature fusion module, and an initial decoding module;

[0023] The replacing the background data in the sample reference data with the sample reference background through a preset initial video generation large model to obtain predicted video data includes:

[0024] Input the sample reference data and the sample reference background into the initial encoding module to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data output by the initial encoding module, and the predicted background features corresponding to the sample reference background;

[0025] Fuse the predicted spatio-temporal semantic features and the predicted background features through the initial feature fusion module to obtain predicted fusion features;

[0026] Decode the predicted fusion features into predicted video data through the initial decoding module.

[0027] Optionally, the initial encoding module includes an initial background encoding module and an initial spatio-temporal semantic encoding module;

[0028] The step of inputting the sample reference data and the sample reference background into the initial encoding module to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data and the predicted background features corresponding to the sample reference background output by the initial encoding module includes:

[0029] Input the sample reference data into the initial spatio-temporal semantic encoding module to obtain the predicted spatio-temporal semantic features output by the initial spatio-temporal semantic encoding module;

[0030] Input the sample reference background into the initial background encoding module to obtain the predicted background features output by the initial background encoding module.

[0031] Optionally, the target video generation large model includes a target encoding module, a target feature fusion module, and a target decoding module;

[0032] The step of replacing the background data in the original video data with the target background data through the target video generation large model to obtain the target video data includes:

[0033] Input the original video data and the target background data into the target encoding module to obtain the target spatio-temporal semantic features corresponding to the original video data and the target background features corresponding to the target background data output by the target encoding module;

[0034] Fuse the target spatio-temporal semantic features and the target background features through the target feature fusion module to obtain target fusion features;

[0035] Decode the target fusion features into target video data through the target decoding module.

[0036] This application also provides a character background editing device, including:

[0037] A data acquisition module for acquiring the original video data with characters and the target background data without characters;

[0038] A model determination module, configured to determine a target video generation large model, where the target video generation large model uses sample video data with people as sample labels, and uses the sample reference background obtained by removing the human figures from the sample video data, and the sample reference data obtained by synthesizing the human figure masks extracted from the sample video data and the sample background data without people as training samples, and is obtained after training;

[0039] A video generation module, configured to replace the background data in the original video data with the target background data through the target video generation large model to obtain target video data.

[0040] This application also provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the person-background editing method according to any one of the above embodiments.

[0041] This application also provides a computer device, including: one or more processors, and a memory;

[0042] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the person-background editing method according to any one of the above embodiments are executed.

[0043] From the above technical solutions, it can be seen that the embodiments of this application have the following advantages:

[0044] For the person-background editing method, device, storage medium, and computer device provided by this application, after obtaining the original video data with people and the target background data without people, this application can first determine the target video generation large model. Since the target video generation large model uses sample video data with people as sample labels, and uses the sample reference background obtained by removing the human figures from the sample video data, and the sample reference data obtained by synthesizing the human figure masks extracted from the sample video data and the sample background data without people as training samples, and is obtained after training, it is equivalent to the expected output during the training of the target video generation large model being real and unmodified data, so that the target video generation large model can simulate real video data. Therefore, after this application replaces the background data in the original video data with the target background data through the target video generation large model, relatively real target video data can be obtained, thereby effectively improving the person-background editing effect in the video creation scenario. And in this process, the user only needs to input the original video data with people and the target background data without people to obtain the target video data, thus greatly reducing the usage threshold. Description of the Drawings

[0045] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0046] Figure 1 A flowchart of a method for editing a person's background provided by an embodiment of the present application;

[0047] Figure 2 A process display diagram for eliminating the human figure in the sample video data provided by an embodiment of the present application;

[0048] Figure 3 A process display diagram for synthesizing a human figure mask with sample background data provided by an embodiment of the present application;

[0049] Figure 4 A model architecture diagram when using the Wan2.1 model for training provided by an embodiment of the present application;

[0050] Figure 5 A structural diagram of a device for editing a person's background provided by an embodiment of the present application;

[0051] Figure 6 An internal structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0053] In one embodiment, as Figure 1 shown, Figure 1 A flowchart of a method for editing a person's background provided by an embodiment of the present application; The present application provides a method for editing a person's background, and the method may include:

[0054] S110: Obtain the original video data with a person and the target background data without a person.

[0055] In this step, when replacing the background of the person in the video data, the original video data with a person and the target background data without a person can be obtained first, so as to replace the background in the original video data with the target background data.

[0056] Among them, this application can collect and upload the original video data and the target background data through the Internet, or can shoot and upload the original video data and the target background data through a shooting device. The specific acquisition methods of the video data and the background data can be selected according to the actual situation and are not limited here. The characters in the original video data with characters obtained by this application have clear and continuous textures, and the characters in the original video data can be one or multiple, which can be specifically selected according to the actual situation and are not limited here either. The target background data without characters obtained by this application refers to videos or images different from the background in the original video data, which can be specifically selected according to the actual situation and are not limited here either.

[0057] S120: Determine the target video generation large model. Among them, the target video generation large model uses the sample video data with characters as the sample label, and uses the sample reference background after removing the human figure in the sample video data, and the sample reference data obtained by synthesizing the human figure mask extracted from the sample video data and the sample background data without characters as the training samples, and is obtained after training.

[0058] In this step, after obtaining the original video data with characters and the target background data without characters through S110, the target video generation large model can also be determined. The target video generation large model is a model pre-trained and saved locally. Using this model for character background replacement can obtain the target video data.

[0059] In a specific implementation manner, during the training stage of the target video generation large model of this application, the sample video data with characters can be collected as the sample label. The sample video data can be collected through the Internet or shot through a shooting device, which can be specifically selected according to the actual situation and are not limited here. The characters in the sample video data with characters obtained by this application have clear and continuous textures, and the characters in the sample video data can be one or multiple, which can be specifically selected according to the actual situation and are not limited here either.

[0060] This application can also collect background videos or images without characters through the Internet, or shoot background videos or images without characters through a shooting device, and synthesize them with the human figure mask extracted from the sample video data to obtain the sample reference data, while the video data after removing the human figure in the sample video data can be used as the sample reference background. During training, this application can use the sample reference data and the sample reference background as the training samples, and use the training samples and the sample label to train the initial video generation large model to obtain the target video generation large model.

[0061] Furthermore, in order to improve the quality of video generation in this application, open-source video generation large models such as Wan2.1 and HunyanVideo can be used as the base model, that is, the initial video generation large model. Then, the base model is fine-tuned by combining the pre-acquired training samples and sample labels, so as to train the target video generation large model that can be used for video character background editing. It can be understood that open-source video generation large models such as Wan2.1 and HunyanVideo all adopt the diffusion model or its variant as the core framework, generate high-quality video content by gradually denoising, and support multi-modal inputs such as text and images, thus realizing the generation process from text to video or from image to video. In addition, during the video generation process, mechanisms such as 3D convolution and spatio-temporal attention can be used to jointly model the temporal coherence and spatial details of the video, ensuring the dynamic naturalness of the generated video. Therefore, using this type of video generation large model as the base model in this application can significantly improve the video generation quality and efficiency, and lower the threshold of video creation.

[0062] S130: Replace the background data in the original video data with the target background data through the target video generation large model to obtain the target video data.

[0063] In this step, after determining the target video generation large model through S120, this application can use the target video generation large model to replace the background data in the original video data with the target background data, so as to obtain the target video data.

[0064] Among them, since the target video generation large model of this application uses the sample video data with characters as the sample label, and the sample reference background after removing the human figures from the sample video data, and the sample reference data synthesized by combining the human figure mask extracted from the sample video data and the sample background data without characters as the training samples, and is obtained after training, therefore, this application can not only realize the video character background editing ability by relying on the powerful generation ability of the target video generation large model, but also make the generated target video data more realistic and natural, thus effectively improving the editing effect of the character background in the video creation scenario.

[0065] In the above embodiments, after obtaining the original video data with people and the target background data without people, the present application can first determine the target video generation large model. Since the target video generation large model uses the sample video data with people as the sample label, and the sample reference background obtained by removing the human figures from the sample video data, and the sample reference data synthesized by combining the human figure masks extracted from the sample video data with the sample background data without people as the training samples, and is obtained after training. It is equivalent to expecting the output during the training of the target video generation large model to be real and unmodified data, so that the target video generation large model can simulate real video data. Therefore, after the present application replaces the background data in the original video data with the target background data through the target video generation large model, relatively real target video data can be obtained, thereby effectively improving the character background editing effect in the video creation scenario. And in this process, the user only needs to input the original video data with people and the target background data without people to obtain the target video data, thus greatly reducing the usage threshold.

[0066] In one embodiment, the training process of the target video generation large model may include:

[0067] S210: Obtain sample video data with people and sample background data without people.

[0068] S211: Extract the human figure mask from the sample video data, and remove the human figure mask from the sample video data to obtain the sample reference background after removing the human figure.

[0069] S212: Combine the human figure mask with the sample background data to obtain sample reference data.

[0070] S213: Replace the background data in the sample reference data with the sample reference background through a preset initial video generation large model to obtain predicted video data.

[0071] S214: Take the predicted video data approaching the sample video data as the goal, and train the initial video generation large model until the preset end condition is reached.

[0072] S215: Use the trained initial video generation large model as the target video generation large model.

[0073] In this embodiment, when training the target video generation large model, a batch of sample video data with people and sample background data without people can be collected by collecting through the Internet or shooting with a shooting device. Then, the human figure mask in the sample video data is extracted, and after eliminating the human figure corresponding to the human figure mask from the sample video data, the sample reference background after eliminating the human figure is obtained. Next, the present application can synthesize the human figure mask and the sample background data to obtain the sample reference data. The present application can input the sample video data, the sample background data, and the sample reference data into the initial video generation large model together. After replacing the background data in the sample reference data with the sample reference background through the initial video generation large model, the predicted video data is obtained. Then, aiming at making the predicted video data approach the sample video data, the initial video generation large model is trained until the preset end condition is reached, so as to obtain the trained initial video generation large model. The present application can use the trained initial video generation large model as the target video generation large model, so that the background in the original video data input by the user can be replaced through the target video generation large model.

[0074] Schematically, as Figure 2 , 3 shown, Figure 2 is a process display diagram for eliminating the human figure in the sample video data provided by the embodiment of the present application, Figure 3 is a process display diagram for synthesizing the human figure mask and the sample background data provided by the embodiment of the present application; from Figure 2 , Figure 3 it can be seen that the present application can perform a human figure extraction operation on the sample video data (i.e., the original human figure video in Figure 3 ), and after obtaining the human figure mask, synthesize it with the sample background data (i.e., the background image / video in Figure 3 ) to obtain the sample reference data (i.e., the synthesized reference image in Figure 3 ). Next, the present application can input the sample video data, the sample background data, and the sample reference data into the initial video generation large model together for training, so as to obtain the target video generation large model.

[0075] Among them, the preset end conditions of the present application can include that the number of training times reaches a preset threshold, the training loss converges to a preset range, etc., which can be specifically set according to the actual situation and are not limited here. Through the above training process, the present application can ensure that the target video generation large model can learn how to accurately separate and synthesize people and the background, so as to generate high-quality target video data.

[0076] In addition, it is worth noting that during the process of character background editing, it is also necessary to consider issues such as the consistency of lighting, shadows, and perspective between the character and the background. To solve these problems, the present application can introduce corresponding loss functions during the training stage of the target video generation large model to constrain the generated video data to be consistent with the real video data in these aspects. For example, the present application can introduce a lighting consistency loss function to constrain the continuity of lighting between the characters and the background in the generated video, thereby avoiding obvious lighting inconsistencies in the generated video, and further improving the accuracy and naturalness of character background editing, bringing a more convenient and efficient solution to the field of video creation.

[0077] In one embodiment, extracting the portrait mask from the sample video data in S211 may include:

[0078] S2111: Obtain a portrait matting model.

[0079] S2112: Use the portrait matting model to extract the portrait mask from the sample video data.

[0080] In this embodiment, when extracting the portrait mask from the sample video data, a mainstream or the latest portrait matting model can be obtained first, and then the portrait matting model can be used to extract the portrait mask from the sample video data, and the portrait mask can be saved separately.

[0081] Specifically, the portrait matting model of the present application can be constructed using deep learning techniques, such as based on architectures such as convolutional neural networks (CNNs) or generative adversarial networks (GANs). By training a large amount of image data containing characters and backgrounds, the portrait matting model can learn the feature differences between the characters and the backgrounds, so as to accurately extract the portrait mask. And during the extraction process, the portrait matting model will process each frame of the sample video data and extract the portrait mask frame by frame, so as to ensure the continuity and accuracy of the extraction results. The extracted portrait mask can be used in the subsequent background replacement process to be synthesized with new background data to generate high-quality video data.

[0082] In one embodiment, eliminating the portrait mask from the sample video data in S211 to obtain a sample reference background after eliminating the portrait may include:

[0083] S2113: Obtain an image inpainting model.

[0084] S2114: Use the image inpainting model to eliminate the portrait corresponding to the portrait mask in the sample video data to obtain a sample reference background after eliminating the portrait.

[0085] In this embodiment, when eliminating the human figures in the sample video data, the mainstream or latest image restoration model can be obtained first, and then the image restoration model is used to eliminate the human figures corresponding to the human figure masks in the sample video data, so as to obtain the sample reference background after eliminating the human figures.

[0086] Specifically, the image restoration model of the present application can also be constructed using deep learning techniques, such as architectures based on convolutional neural networks (CNNs) or generative adversarial networks (GANs). The image restoration model is trained on a large amount of image data containing missing regions to learn the methods for restoring the missing regions in the images, so that it can accurately eliminate the human figures corresponding to the human figure masks in the sample video data and generate a coherent and natural background. The generated sample reference background after eliminating the human figures can be used in the subsequent background replacement process to be synthesized with new background data to generate high-quality video data. In this way, the present application can efficiently implement human background editing and provide users with a more convenient and efficient video creation experience.

[0087] In one embodiment, the initial video generation large model may include an initial encoding module, an initial feature fusion module, and an initial decoding module.

[0088] In S213, replacing the background data in the sample reference data with the sample reference background through a preset initial video generation large model to obtain predicted video data may include:

[0089] S2131: Inputting the sample reference data and the sample reference background into the initial encoding module to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data output by the initial encoding module, and the predicted background features corresponding to the sample reference background.

[0090] S2132: Fusing the predicted spatio-temporal semantic features with the predicted background features through the initial feature fusion module to obtain predicted fusion features.

[0091] S2133: Decoding the predicted fusion features into predicted video data through the initial decoding module.

[0092] In this embodiment, the initial video generation large model may include multiple modules, specifically an initial encoding module, an initial feature fusion module, and an initial decoding module. After training the initial video generation large model in the present application, the initial video generation large model learns the features of the sample labels, that is, learns the features of the sample video data with human figures, and then uses the learned features to adjust the parameters in the initial encoding module, the initial feature fusion module, and the initial decoding module, so as to improve the video generation quality.

[0093] Specifically, in this application, the sample reference data and the sample reference background can be first input into the initial encoding module. The initial encoding module will encode the input sample reference data and the sample reference background to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data and the predicted background features corresponding to the sample reference background respectively. Then, this application will input the obtained predicted spatio-temporal semantic features and the predicted background features into the initial feature fusion module. The initial feature fusion module will perform a fusion process on these two features to obtain the predicted fusion features. Finally, this application will input the predicted fusion features into the initial decoding module. The initial decoding module will decode the predicted fusion features to obtain the predicted video data.

[0094] The predicted video data is generated by the initial video generation large model based on the input sample reference data and the sample reference background. Its background has been replaced with the sample reference background, while the characters retain the character features in the sample reference data. Therefore, through the above process, this application continuously trains and adjusts the initial video generation large model, and finally obtains a target video generation large model that can generate high-quality target video data.

[0095] In one embodiment, the initial encoding module may include an initial background encoding module and an initial spatio-temporal semantic encoding module.

[0096] In S2131, inputting the sample reference data and the sample reference background into the initial encoding module to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data and the predicted background features corresponding to the sample reference background output by the initial encoding module may include:

[0097] S21311: Input the sample reference data into the initial spatio-temporal semantic encoding module to obtain the predicted spatio-temporal semantic features output by the initial spatio-temporal semantic encoding module.

[0098] S21312: Input the sample reference background into the initial background encoding module to obtain the predicted background features output by the initial background encoding module.

[0099] In this embodiment, after inputting the sample reference data and the sample reference background into the initial encoding module, the initial encoding module will encode the input sample reference data and the sample reference background to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data and the predicted background features corresponding to the sample reference background respectively.

[0100] Specifically, the initial encoding module of the present application may include an initial background encoding module and an initial spatio-temporal semantic encoding module. Among them, the initial spatio-temporal semantic encoding module is responsible for encoding the sample reference data to extract the spatio-temporal semantic information of the people therein, including the actions, positions of the people, and the relative relationships with the background, etc., so as to obtain the predicted spatio-temporal semantic features. The initial background encoding module is responsible for encoding the sample reference background to extract the spatial structure and texture features of the background, so as to obtain the predicted background features. In this way, the present application can ensure that in the subsequent feature fusion process, the features of people and the background can be effectively combined, thereby generating high-quality target video data.

[0101] In a specific implementation manner, the initial video generation large model of the present application may be the Wan2.1 model. When the present application uses the Wan2.1 model as the basic model, its specific model architecture is as Figure 4 shown, Figure 4 which is the model architecture diagram when using the Wan2.1 model for training provided by the embodiments of the present application; Figure 4 The reference background image / video in is the sample reference background of the present application, the reference image is the sample reference data in the present application, and the expected generation result is the sample video data of the present application.

[0102] Figure 4 In, the initial background encoding module is responsible for extracting the static features of the reference background image (such as scene layout, hue), and outputting the low-dimensional latent variable bg_latent. The initial spatio-temporal semantic encoding module is responsible for compressing the input video frame (or single-frame image) into the spatio-temporal latent variable video_latent and retaining the temporal dynamic information. The initial feature fusion module completes the denoising (noise prediction) of the spatio-temporal latent variable through the Transformer architecture and supports long-sequence modeling and multi-condition fusion. Specifically, the initial feature fusion module can receive bg_latent (background latent variable) from the initial background encoding module and video_latent (noisy spatio-temporal latent variable) from the initial spatio-temporal semantic encoding module, and achieve multi-condition control through cross-attention or concatenation. The initial decoding module can decode the denoised low-dimensional spatio-temporal latent variable of the initial feature fusion module into a video frame in the pixel space, and finally generate the predicted video data.

[0103] Furthermore, the design of the initial background encoding module usually focuses on processing static background images. This application can be extended technically to support video background input. For example, this application can extract the background features of dynamic backgrounds through key frame extraction + static encoding, or can also achieve the feature extraction of dynamic backgrounds through methods such as temporal background encoders or dynamic and static background separation training. Specifically, it can be set according to the actual situation and is not limited here.

[0104] In one embodiment, the target video generation large model may include a target encoding module, a target feature fusion module, and a target decoding module.

[0105] Replacing the background data in the original video data with the target background data through the target video generation large model in S130 to obtain target video data may include:

[0106] S131: Input the original video data and the target background data into the target encoding module to obtain the target spatio-temporal semantic features corresponding to the original video data output by the target encoding module, and the target background features corresponding to the target background data.

[0107] S132: Fuse the target spatio-temporal semantic features and the target background features through the target feature fusion module to obtain target fusion features.

[0108] S133: Decode the target fusion features into target video data through the target decoding module.

[0109] In this embodiment, the target video generation large model can also include multiple modules, specifically a target encoding module, a target feature fusion module, and a target decoding module. After training the target video generation large model, this application can use this target video generation large model to replace the background of the original video data, thereby generating target video data.

[0110] Specifically, this application can first input the original video data and the target background data into the target encoding module. The target encoding module will encode the input original video data and target background data to obtain the target spatio-temporal semantic features corresponding to the original video data and the target background features corresponding to the target background data respectively. Then, this application can input the obtained target spatio-temporal semantic features and target background features into the target feature fusion module. The target feature fusion module will perform fusion processing on these two features to obtain target fusion features. Finally, this application inputs the target fusion features into the target decoding module. The target decoding module will decode the target fusion features to obtain target video data.

[0111] It can be understood that the target video data of this application is generated by the target video generation large model based on the input original video data and target background data. Its background has been replaced with the target background data, while the characters retain the character features in the original video data. In this way, this application can efficiently achieve character background editing, providing users with a more convenient and efficient video creation experience. At the same time, since the target video generation large model is trained on a large number of sample data, it can learn the complex relationship between characters and backgrounds, thereby generating more natural and realistic target video data.

[0112] The following describes the character background editing device provided by the embodiments of this application. The character background editing device described below can be mutually corresponding and referred to with the character background editing method described above.

[0113] In one embodiment, as Figure 5 shown, Figure 5 is a schematic structural diagram of a character background editing device provided by an embodiment of this application; this application also provides a character background editing device, which may specifically include a data acquisition module 210, a model determination module 220, and a video generation module 230, specifically as follows:

[0114] The data acquisition module 210 is used to acquire the original video data with characters and the target background data without characters.

[0115] The model determination module 220 is used to determine the target video generation large model. Among them, the target video generation large model uses the sample video data with characters as the sample label, and uses the sample reference background after removing the human portrait in the sample video data, and the sample reference data synthesized by the human portrait mask extracted from the sample video data and the sample background data without characters as the training samples, and is obtained after training.

[0116] The video generation module 230 is used to replace the background data in the original video data with the target background data through the target video generation large model to obtain the target video data.

[0117] In the above embodiments, after obtaining the original video data with people and the target background data without people, the present application can first determine the target video generation large model. Since the target video generation large model uses the sample video data with people as the sample label, and the sample reference background after removing the human figures in the sample video data, and the sample reference data synthesized by the human figure masks extracted from the sample video data and the sample background data without people as the training samples, and is obtained after training. It is equivalent to expecting the output during the training of the target video generation large model to be real and unmodified data, so that the target video generation large model can simulate real video data. Therefore, after the present application replaces the background data in the original video data with the target background data through the target video generation large model, relatively real target video data can be obtained, thereby effectively improving the character background editing effect in the video creation scenario. And this process only requires the user to input the original video data with people and the target background data without people to obtain the target video data, thus greatly reducing the usage threshold.

[0118] In one embodiment, the present application further provides a computer-readable storage medium. Computer-readable instructions are stored in the computer-readable storage medium. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the character background editing method according to any one of the above embodiments.

[0119] In one embodiment, the present application further provides a computer device, including: one or more processors, and a memory.

[0120] Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the one or more processors, the steps of the character background editing method according to any one of the above embodiments are executed.

[0121] Schematically, as Figure 6 shown, Figure 6 is an internal structure diagram of a computer device provided by an embodiment of the present application. The computer device 300 can be provided as a server. Referring to Figure 6 , the computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by a memory 301 for storing instructions executable by the processing component 302, such as application programs. The application programs stored in the memory 301 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the character background editing method of any of the above embodiments.

[0122] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS XTM, Unix TM, Linux TM, Free BSDTM, or the like.

[0123] Those skilled in the art can understand that Figure 6 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0124] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0125] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0126] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for editing a character background, characterized in that, The method includes: Obtaining original video data with people and target background data without people; Determining a target video generation large model, where the target video generation large model uses sample video data with people as sample labels, and uses the sample reference background after removing the human figures from the sample video data, and the sample reference data synthesized by combining the human figure masks extracted from the sample video data with the sample background data without people as training samples, and is obtained after training; Replacing the background data in the original video data with the target background data through the target video generation large model to obtain target video data.

2. The method for editing a character background according to claim 1, wherein The training process of the target video generation large model includes: Obtaining sample video data with people and sample background data without people; Extracting the human figure mask from the sample video data, and removing the human figure mask from the sample video data to obtain a sample reference background after removing the human figure; Combining the human figure mask with the sample background data to obtain sample reference data; Replacing the background data in the sample reference data with the sample reference background through a preset initial video generation large model to obtain predicted video data; Taking the predicted video data approaching the sample video data as the goal, training the initial video generation large model until a preset end condition is reached; Taking the trained initial video generation large model as the target video generation large model.

3. The method for editing a character background according to claim 2, wherein The extracting of the human figure mask from the sample video data includes: Obtaining a human figure matting model; Using the human figure matting model to extract the human figure mask from the sample video data.

4. The method for editing a character background according to claim 2, wherein The removing of the human figure mask from the sample video data to obtain a sample reference background after removing the human figure includes: Obtaining an image restoration model; Removing the human figure corresponding to the human figure mask in the sample video data through the image restoration model to obtain a sample reference background after removing the human figure.

5. The method for editing a character background according to claim 2, wherein, The initial video generation large model includes an initial encoding module, an initial feature fusion module, and an initial decoding module; The replacing of the background data in the sample reference data with the sample reference background through a preset initial video generation large model to obtain predicted video data includes: Inputting the sample reference data and the sample reference background into the initial encoding module to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data output by the initial encoding module, and the predicted background features corresponding to the sample reference background; Fusing the predicted spatio-temporal semantic features with the predicted background features through the initial feature fusion module to obtain predicted fusion features; Decoding the predicted fusion features into predicted video data through the initial decoding module.

6. The method for editing a character background according to claim 5, wherein The initial encoding module includes an initial background encoding module and an initial spatio-temporal semantic encoding module; The inputting of the sample reference data and the sample reference background into the initial encoding module to obtain the predicted spatio-temporal semantic features corresponding to the sample reference data output by the initial encoding module, and the predicted background features corresponding to the sample reference background includes: Input the sample reference data into the initial spatio-temporal semantic encoding module to obtain the predicted spatio-temporal semantic features output by the initial spatio-temporal semantic encoding module; Input the sample reference background into the initial background encoding module to obtain the predicted background features output by the initial background encoding module.

7. The method for editing a character background according to any one of claims 1-6, characterized in that The target video generation large model includes a target encoding module, a target feature fusion module, and a target decoding module; Replacing the background data in the original video data with the target background data through the target video generation large model includes: Input the original video data and the target background data into the target encoding module to obtain the target spatio-temporal semantic features corresponding to the original video data and the target background features corresponding to the target background data output by the target encoding module; Fuse the target spatio-temporal semantic features and the target background features through the target feature fusion module to obtain target fusion features; Decode the target fusion features into target video data through the target decoding module.

8. A character background editing device, characterized in that, Including: A data acquisition module for acquiring original video data with people and target background data without people; A model determination module for determining a target video generation large model, where the target video generation large model uses sample video data with people as sample labels, the sample reference background after removing the human figure in the sample video data, and the sample reference data obtained by synthesizing the human figure mask extracted from the sample video data and the sample background data without people as training samples and is obtained after training; A video generation module for replacing the background data in the original video data with the target background data through the target video generation large model to obtain target video data.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the person-background editing method according to any one of claims 1 to 7.

10. A computer device, characterized in that, Including: One or more processors and a memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the person-background editing method according to any one of claims 1 to 7 are executed.

Images