Video processing method, model training method, electronic device and storage medium

Abstract

The present disclosure relates to the technical field of large models and the technical field of video processing. Disclosed are a video processing method, a model training method, an electronic device and a storage medium. The method comprises: acquiring an original video to be edited and video content to be used for replacement, wherein said video content is used for replacing a foreground target and / or a background target displayed in the original video; on the basis of said video content, performing spatial representation decomposition and encoding on the original video to obtain encoding results; and combining the encoding results to generate a target video. The present disclosure solves the technical problem in the prior art whereby videos generated for replacement content in video processing solutions exhibit poor scene adaptability and low rendering realism and accuracy.

Description

Video processing method, model training method, electronic device and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of large model and the technical field of video processing, in particular, to a video processing method, a model training method, an electronic device and a storage medium. BACKGROUND

[0002] Video processing technology (in particular, video generation, video replacement, etc.) is one of the hot research directions in the related technical field. Video processing technology has a wide range of applications in scenarios such as film and television, interactive entertainment, media creation, etc. The existing technology usually only generates a video that meets the user's input text description, that is, text-to-video. However, the above text-to-video scheme has the following technical defects: it relies on text description to generate the overall content of the video, cannot support more concrete video generation requirement description forms, and has insufficient controllability; it can only generate videos showing objects performing simple movements, and has limitations in complex action generation; it is difficult to solve the video generation of complex scene pictures (such as scene pictures with foreground occlusion).

[0003] Therefore, how to improve the user controllability and complex scene adaptability of the video processing scheme, and enhance the rendering fidelity and accuracy of the generated video has become one of the important technical problems in the related technical field. In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0004] The embodiments of the present disclosure provide a video processing method, a model training method, an electronic device and a storage medium to at least solve the technical problem that the video generated by the video processing scheme in the prior art has poor scene adaptability, rendering fidelity and accuracy.

[0005] According to an aspect of an embodiment of the present disclosure, a video processing method is provided, comprising: obtaining an original video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace a foreground target and / or a background target displayed by the original video; performing spatial representation decomposition coding on the original video based on the video content to be replaced, to obtain a coding result; and combining the coding result to generate a target video.

[0006] According to another aspect of an embodiment of the present disclosure, a video processing method is also provided, comprising: obtaining an original character video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace an original character displayed by the original character video with a target character; performing spatial representation decomposition coding on the original character video based on the video content to be replaced, to obtain a coding result; and combining the coding result to generate a target character video.

[0007] According to another aspect of the embodiments of the present disclosure, a video processing method is also provided, including: obtaining a video processing request through a first application programming interface, wherein the request data carried in the video processing request includes: an original video to be edited and video content to be replaced, the video content to be replaced being used to replace a foreground target and / or a background target shown in the original video; and returning a video processing response through a second application programming interface, wherein the response data carried in the video processing response includes: a target video, the target video being generated by combining an encoding result, the encoding result being obtained by performing spatial representation decomposition coding on the original video based on the video content to be replaced.

[0008] According to another aspect of the embodiments of the present disclosure, a video processing method is also provided, including: selecting an original video to be edited in response to a first touch operation performed on a graphical user interface; uploading video content to be replaced in response to a second touch operation performed on the graphical user interface, the video content to be replaced being used to replace a foreground target and / or a background target shown in the original video; generating a target video in response to a third touch operation performed on the graphical user interface, the target video being generated by combining an encoding result, the encoding result being obtained by performing spatial representation decomposition coding on the original video based on the video content to be replaced; and displaying the target video in the graphical user interface.

[0009] According to another aspect of the embodiments of the present disclosure, a model training method is also provided, including: obtaining a sample video; and training an initial content reconstruction model using the sample video to obtain a target content reconstruction model, wherein the initial content reconstruction model is used to obtain appearance coding corresponding to the sample video, motion coding corresponding to the sample video, scene and occlusion coding corresponding to the sample video, and to combine the motion coding, the appearance coding and the scene and occlusion coding to obtain prediction noise.

[0010] According to another aspect of the embodiments of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor configured to execute the program, wherein the program, when executed, performs any of the video processing methods described above.

[0011] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is also provided, including a stored executable program, wherein the executable program, when executed, controls a device in which the computer readable storage medium is located to perform any of the video processing methods described above.

[0012] According to another aspect of the embodiments of the present disclosure, a computer program product is also provided, including a computer program, the computer program, when executed by a processor, implementing any of the video processing methods described above.

[0013] In the embodiments of the present disclosure, the original video to be edited and the video content to be replaced are obtained, wherein the video content to be replaced is used to replace the foreground target and / or background target shown in the original video; the original video is spatially characterized and decomposed and coded based on the video content to be replaced, to obtain a coding result; and the coding result is combined to generate a target video. Thus, based on the original video and the video content to be replaced, the technical concept of spatial characterization decomposition coding combined with combined video generation is adopted to process and edit the video data in the coding dimension, which not only can replace the complex content in the more complex application scenarios and generate the content, but also can enhance the adaptability between the video content to be replaced and the non-replaced part of the original video. In addition, since the above-mentioned video content to be replaced is not limited to text description for replacing the foreground target and / or background target, the present disclosure can support the user to describe the video data processing requirement in a more concrete form, thereby enhancing the controllability of the scheme. Based on this, the present disclosure achieves the purpose of generating a target video with higher accuracy and stronger realism, thereby realizing the technical effects of improving the scene adaptability, rendering realism and accuracy of video processing, and further solving the technical problems of poor scene adaptability, poor rendering realism and accuracy of the video generated by the video processing scheme in the prior art for the content to be replaced.

[0014] It is easy to note that the general description above and the detailed description below are merely for exemplifying and explaining the present disclosure, and do not constitute a limitation on the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0015] The drawings described herein are used to provide further understanding of the present disclosure, and form a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions serve to explain the present disclosure, and do not constitute an improper limitation on the present disclosure. In the drawings:

[0016] FIG. 1 is a schematic diagram of an application scenario of a video processing method according to an embodiment of the present disclosure;

[0017] FIG. 2 is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0018] FIG. 3 is a schematic diagram of an optional video generation framework according to an embodiment of the present disclosure;

[0019] FIG. 4 is a flowchart of another video processing method according to an embodiment of the present disclosure;

[0020] FIG. 5 is a flowchart of still another video processing method according to an embodiment of the present disclosure;

[0021] FIG. 6 is a flowchart of still another video processing method according to an embodiment of the present disclosure;

[0022] FIG. 7 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0023] FIG. 8 is a structural block diagram of a video processing apparatus according to an embodiment of the present disclosure;

[0024] FIG. 9 is a structural block diagram of another video processing apparatus according to an embodiment of the present disclosure;

[0025] FIG. 10 is a structural block diagram of yet another video processing apparatus according to an embodiment of the present disclosure;

[0026] FIG. 11 is a structural block diagram of yet another video processing apparatus according to an embodiment of the present disclosure;

[0027] FIG. 12 is a structural block diagram of a model training apparatus according to an embodiment of the present disclosure;

[0028] FIG. 13 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to enable persons skilled in the art to better understand the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present disclosure.

[0030] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0031] The technical solution provided by the present disclosure is mainly implemented by using a large model technology. Here, the large model refers to a deep learning model with a large number of model parameters, which can generally include hundreds of millions, billions, tens of billions, hundreds of billions, or even more than ten trillion model parameters. The large model can also be referred to as a foundation model. A large-scale unlabeled corpus is used to pre-train the large model, and a pre-training model with more than one hundred million parameters is output. Such a model can adapt to a wide range of downstream tasks, and the model has good generalization ability. For example, a large-scale language model (LLM) and a multi-modal pre-training model.

[0032] It should be noted that the large model can be fine-tuned by using a small number of samples in actual application, so that the large model can be applied to different tasks. For example, the large model can be widely applied to the fields of natural language processing (NLP), computer vision, and speech processing. Specifically, the large model can be applied to computer vision field tasks such as visual question answering (VQA), image captioning (IC), and image generation. The large model can also be widely applied to natural language processing field tasks such as text-based sentiment classification, text summarization generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, and the like. In the embodiments of the present disclosure, video processing by using a large language model or a pre-training model is taken as an example for explanation and description.

[0033] First, some nouns or terms appearing in the description of the embodiments of the present disclosure are applicable to the following explanations.

[0034] Pre-training model: refers to a model that is pre-trained on a large-scale dataset. The pre-training model does not have to be trained separately for a single scenario, but can be used as a basis for a new task, that is, the pre-training model is fine-tuned based on the new task to obtain a model for the new task. By applying the pre-training model technology, the model performance in a specific scenario is improved, and the model generalization and stability are enhanced.

[0035] Camera parameters: used to specify the internal characteristics (such as focal length, aperture, etc.) and external characteristics (such as camera position, camera direction, etc.) of a camera. The camera parameters are key parameters in the process of projecting a three-dimensional scene to a two-dimensional scene.

[0036] Diffusion generative models are a class of deep learning models that generate data by simulating a stochastic diffusion process. Diffusion generative models treat data as the result of a random diffusion process starting from a simple distribution (such as a Gaussian noise distribution),

[0037] Variational autoencoders (VAEs) are a type of generative model that learn a latent representation of data and generate new data that resembles the input data. They are commonly used in data dimensionality reduction and data generation scenarios.

[0038] Differentiable renderers are tools used to render three-dimensional models by generating two-dimensional images. They calculate the gradient of each pixel with respect to the model parameters during the rendering process, and then use optimization methods such as gradient descent to improve the rendering results.

[0039] UV mapping is a technique used to map two-dimensional texture images onto the surface of a three-dimensional model. Each point on the surface of the three-dimensional model corresponds to a set of UV coordinates, which specify the corresponding pixel on the two-dimensional texture image.

[0040] Vertices are the basic units that make up the mesh elements (such as triangular mesh elements, quadrilateral mesh elements, etc.) of a three-dimensional model. The more vertices a three-dimensional model has, the more detailed it is.

[0041] Skinned Multi-Person Linear Model (SMPL) is a parametric human three-dimensional model. SMPL can be used to accurately represent different shapes and poses of the human body. The extended version of SMPL can accurately represent different shapes and poses of multiple human bodies (such as hand movements and facial expressions, etc.) with higher performance, which is called SMPLX.

[0042] Linear Blend Skinning (LBS) is a technique used to simulate the deformation of the skin of a three-dimensional model in different poses. It determines the position of the model vertices by applying bone movements to the model vertices and using linear blending techniques.

[0043] According to the embodiments of the present disclosure, a video processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0044] In view of the large amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned video processing method provided by the embodiments of the present disclosure can be applied to the application scenario as shown in FIG. 1, but is not limited thereto. In the application scenario as shown in FIG. 1, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 can include, but is not limited to, a smartphone, a tablet computer, a notebook computer, a palm computer, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to call the large model, thereby realizing the method provided by the embodiments of the present disclosure.

[0045] Under the above running environment, the present disclosure provides a video processing method as shown in FIG. 2. FIG. 2 is a flowchart of a video processing method according to an embodiment of the present disclosure. As shown in FIG. 2, the video processing method includes:

[0046] Step S21, obtaining an original video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace a foreground target and / or a background target displayed in the original video;

[0047] Step S22, performing spatial feature decomposition coding on the original video based on the video content to be replaced to obtain a coding result;

[0048] Step S23, combining the coding result to generate a target video.

[0049] The above-mentioned original video can include a plurality of original video frames. For example, in an exemplary application scenario, the original video at least displays a person target. The above-mentioned foreground target at least includes the person target, a posture target corresponding to the person target, and / or an action target. The above-mentioned background target can include a scene model displayed in the picture background of the original video. In addition, the original video can also display an animal target, an article target, a virtual digital object target, etc.

[0050] Based on the above-mentioned to-be-replaced video content, spatial representation decomposition coding can be performed on the original video on a large amount of video data to obtain a coding result. The specific implementation manner of the spatial representation decomposition coding can be that, according to the spatial attribute information (such as depth information), the original video is spatially represented and decomposed, thereby the three-dimensional spatial representation shown in the two-dimensional original video frame can be fully understood, and further, the decomposition result obtained by the spatial representation decomposition is coded based on the to-be-replaced video content at the feature level to obtain a coding result. The coding result can correspond to a plurality of coding dimensions, and the plurality of coding dimensions can correspond to a plurality of decomposition layers included in the decomposition result, and each coding dimension corresponds to a sub-coding result. For example, the coding result includes at least one of appearance coding, action coding, scene coding, model coding, and the like.

[0051] Further, in the coding process, the coding corresponding to part or all of the content in the original video can be replaced according to the specific editing requirements of the original video. Based on this, the coding result is subjected to a combined video generation process to obtain the above-mentioned target video.

[0052] The above-mentioned method steps provided by the embodiments of the present disclosure can support replacing part of the content in the original video based on the to-be-replaced video content to obtain a new target video. For example, the above-mentioned to-be-replaced video content can include a character role image, based on which the above-mentioned scheme can replace the original character displayed in the original video with the target character displayed in the character role image. For another example, the above-mentioned to-be-replaced video content can include a target scene image, based on which the above-mentioned scheme can replace the original video background displayed in the original video with the target scene displayed in the target scene image. The above-mentioned to-be-replaced video content can also include action sequence data, based on which the above-mentioned scheme can replace the original action of the character model displayed in the original video with the target action corresponding to the action sequence data.

[0053] The above-mentioned video processing method can be applied to provide video processing services, especially video editing services, for a preset application scenario to obtain a target video in which part of the content in the original video is replaced as specified. The above-mentioned preset application scenario can include but is not limited to scenarios involving video processing, video playing, video editing, etc. in the fields of e-commerce, education, medical treatment, conference, social network, financial product, logistics and navigation, etc.

[0054] In the embodiments of the present disclosure, the original video to be edited and the video content to be replaced are obtained, wherein the video content to be replaced is used to replace the foreground target and / or the background target displayed in the original video; the original video is spatially characterized and decomposed and coded based on the video content to be replaced, to obtain a coding result; and the coding result is combined to generate a target video. Thus, based on the original video and the video content to be replaced, the technical concept of spatial characterization decomposition coding combined with combined video generation is adopted to process and edit the video data in the coding dimension, to enhance the adaptability between the video content to be replaced and the non-replaced part of the original video, to achieve the purpose of generating a target video that is more accurate and has stronger realism, thereby realizing the technical effects of improving the scene adaptability, rendering realism and accuracy of video processing, and further solving the technical problems in the related art that the video generation model is trained based on network data, resulting in a video generated by the trained video generation model having poor quality and not meeting user expectations.

[0055] The above method steps provided by the embodiments of the present disclosure are further described below in combination with application scenarios.

[0056] The embodiments of the present disclosure provide a video processing method with higher controllability, and construct a video generation framework based on spatial characterization decomposition modeling as shown in FIG. 3. Based on the video generation framework, the scheme provided by the present disclosure can support the user to replace the content of the corresponding part to be replaced in the original video frame after inputting the video content to be replaced, and then quickly realize video generation. In this process, the user needs to perform simple operations but can achieve strong controllability of original video editing.

[0057] The embodiments of the present disclosure provide a video processing method based on spatial characterization modeling, which has strong controllability and can effectively improve the controllability in multiple dimensions in video processing, and can also enhance the performance of character interaction and complex spatial motion in video processing.

[0058] In an optional embodiment, in step S22, the original video is spatially characterized and decomposed and coded based on the video content to be replaced, to obtain a coding result, including the following method steps:

[0059] In step S221, the original video is spatially hierarchically decomposed according to the depth information of the original video, to obtain a plurality of decomposition layers, wherein the plurality of decomposition layers include a scene layer, an object layer and a front occlusion layer.

[0060] In step S222, the plurality of decomposition layers are feature-coded based on the video content to be replaced, to obtain a coding result.

[0061] In the optional embodiment described above, the depth information of the original video can include the picture depth data of each original video frame in the original video. The original video frame displays foreground objects and background objects, for example, the picture depth value corresponding to the foreground object is smaller than the picture depth value of the background object. Based on this, the embodiments of the present disclosure can perform spatial hierarchical decomposition on the original video frame based on the depth information of the two-dimensional original video frame, to obtain a plurality of decomposition layers corresponding to a plurality of different depth ranges.

[0062] For example, the plurality of decomposition layers include, in order from deep to shallow, a scene layer, an object layer, and a front occlusion layer.

[0063] Further, based on the to-be-replaced video content, the to-be-encoded decomposition layer in the plurality of decomposition layers is feature-encoded to obtain an encoding result. For example, the to-be-replaced video content is used to determine to replace the scene in the original video, at this time, the scene layer is determined as the to-be-encoded layer, and the scene layer is specifically scene feature-encoded to obtain the encoding result.

[0064] The depth information is used to provide a three-dimensional perspective of the spatial layout of the scene corresponding to the original video. The depth information can be the distance between each pixel point and the camera. Through the depth information, the relative positions and distances of different objects displayed in the video frame can be determined, so that spatial hierarchical analysis can be realized. Through the depth information, the original video is spatially hierarchically decomposed to identify and separate different visual elements, which are classified into different layers for subsequent processing.

[0065] The spatial hierarchical decomposition is an image or video processing technology, which analyzes the depth information to layer the video content according to the front-back relationship and depth distance, thereby constructing a hierarchical visual structure. This helps subsequent feature encoding, especially when replacing objects, which can more accurately locate and process objects.

[0066] The to-be-replaced video content can be a video segment that a user wants to replace a specific content in the original video, such as a different foreground object, a background environment, or the appearance and action of a specific object.

[0067] The feature encoding refers to the process of converting visual features (such as color, texture, shape, action, etc.) into numerical representation, which is usually performed through a neural network model. The encoding result is a compact, mathematical representation that can be used for subsequent video generation or modification. Based on this, each decomposition layer is feature-encoded to obtain an automatic encoding result corresponding to the decomposition layer. That is, the encoding result includes a plurality of sub-encoding results.

[0068] Through the steps S221 to S222, the embodiment of the disclosure can fully consider the spatial information of the original video display content in the process of encoding the original two-dimensional video frame by introducing a spatial hierarchical decomposition scheme based on depth information, thereby ensuring that the to-be-encoded decomposition layer can be determined in a targeted manner when feature coding is performed based on the to-be-replaced video content, and the accuracy of the coding result is improved.

[0069] In an optional embodiment, in step S221, the original video is spatially hierarchically decomposed according to the depth information of the original video to obtain a plurality of decomposition layers, including the following method steps:

[0070] Step S2211, using an instance segmentation model to decompose an object layer from the original video;

[0071] Step S2212, using a depth detection model to extract a depth value from the original video to obtain a front occlusion layer;

[0072] Step S2213, determining the layers other than the object layer and the front occlusion layer as a scene layer.

[0073] Instance segmentation is a deep learning technique that can identify and segment different object instances in a video. Using an instance segmentation model to decompose an object layer from the original video, that is, separating the target object in the original video from the background to form an object layer.

[0074] The depth detection model is used to extract the depth value of each pixel in the video. The front occlusion layer refers to the layer in the foreground of the scene that may occlude other objects, such as trees, fences, etc. Through the depth value, it can be determined which objects are in front of other objects, thereby constructing this layer.

[0075] After removing the object layer and the front occlusion layer, the remaining part mainly represents the background environment of the video, which is called a scene layer. The scene layer contains parts of the video that are not directly related to specific target objects, such as the sky, the ground, buildings, etc.

[0076] In the above optional embodiment, the instance segmentation model can be a pre-trained model. For example, the instance segmentation model can be a Sequential Associative Memory (SAM) model, a K-nearest neighbor model, a naive Bayes model, a neural network model, a hidden Markov model, and a principal component analysis model, etc. The object detection bounding box can be specified by the technical personnel according to the specific application scenario, or can be predicted by the neural network model corresponding to the specific application scenario. The depth detection model can be a pre-trained model. For example, the depth detection model can be a Depth-Anything model, etc.

[0077] In the process of decomposing the object layer from the original video by using the instance segmentation model, the object layer is extracted from the multiple layers corresponding to the original video according to the object detection bounding box.

[0078] In an application scenario, as shown in FIG. 3, the original video frame is spatially hierarchically decomposed to obtain multiple decomposed layers. Specifically, the original video can be a binary mask video, which contains instance segmentation regions in time sequence. For each original video frame in the original video, the display content of the original video frame is multi-instance decomposed in combination with the depth information of the original video frame. For example, according to the picture depth value, the original video frame is decomposed into a scene layer, an object layer (which can be a person layer), and a front occlusion layer.

[0079] Through the above steps S2211 to S2213, the original video is spatially hierarchically decomposed by the instance segmentation, object detection based on the bounding box, and depth value extraction combined scheme to obtain multiple decomposed layers, so as to be subsequently processed differently according to the video content to be replaced.

[0080] In an optional embodiment, in step S2211, the object layer is decomposed from the original video by using the instance segmentation model and the object detection bounding box, including the following method steps:

[0081] Step S2214, the target object in the original video is instance segmented by using the instance segmentation model to obtain a segmentation result.

[0082] Step S2215, the object mask extraction is performed on the segmentation result according to the object detection bounding box to obtain the object layer.

[0083] The instance segmentation model is a neural network model for recognizing and segmenting specific targets in images and videos by using computer vision technology. Unlike traditional object detection (only giving the bounding box of the object) and semantic segmentation (only giving the category to which the object belongs), the instance segmentation model can simultaneously recognize different object instances in an image and generate accurate pixel-level masks for each instance, i.e., each pixel of an object is classified, and then the segmentation result is obtained. Using the instance segmentation model can improve the accuracy of the disclosed embodiment in processing the person, animal, vehicle, and other target objects in the video.

[0084] The segmentation result is the direct output of the instance segmentation model after processing the target object, which is usually an image or a video frame containing pixel masks of each instance. These pixel masks can be understood as a black and white image with the same size as the original video frame, where the white pixels correspond to the pixels of the target object, and the black pixels correspond to the pixels of the background or other non-target objects.

[0085] The object detection bounding box is used to roughly frame the region of the target object in the video frame. The object detection bounding box can be generated by a separate object detection model or can be part of the output of the instance segmentation model. The object detection bounding box provides a bounding box of the target object, which is used for further refinement of the segmentation result processing to ensure the accuracy and efficiency of the segmentation.

[0086] Further, based on the object detection bounding box, the pixel mask (i.e., object mask) of the target object is accurately extracted from the segmentation result, completing the object mask extraction. The process of the object mask extraction usually involves alignment and cropping of the pixel mask and the object detection bounding box to ensure that the extracted object layer contains information of the target object and excludes the background and other irrelevant objects.

[0087] Still taking the above application scenario as an example, a pre-trained instance segmentation model is used to perform multi-instance decomposition on a given original video containing n original video frames. Specifically, based on the object detection bounding box (such as a human body detection bounding box), an object mask is extracted from the original video frame to obtain an object layer. The object layer can be regarded as a subset of the original video frame, which is used for subsequent feature encoding and replacement operations.

[0088] Through the steps S2214 to S2215, the pixel-level information of the target object is accurately extracted from the original video, and the object detection bounding box and the instance segmentation model are used to determine the object layer from the original video, which can accurately identify and segment the object in the video frame, provide detailed information about the shape and contour of the object, and has a high processing speed, providing key data support for subsequent video editing (such as replacing the foreground or background).

[0089] In an optional embodiment, in step S2212, a depth detection model is used to extract depth values from the original video to obtain a front occlusion layer, including the following method steps:

[0090] In step S2216, a depth detection model is used to extract depth values from the original video to obtain a depth threshold, wherein the depth threshold is a frame-by-frame average depth value of the object layer.

[0091] In step S2217, the depth threshold is used to find a to-be-fused layer, wherein the depth of the to-be-fused layer is lower than the depth threshold.

[0092] In step S2218, the to-be-fused layer is fused to obtain a front occlusion layer.

[0093] The depth detection model is used to estimate the depth information of each pixel in each video frame in the original time-frequency. The depth detection model can be a monocular image-based depth estimation model, or a model based on stereo images or optical flow information. In the field of video processing, the depth detection model can help understand the three-dimensional structure of the scene.

[0094] The depth detection model is applied to each video frame of the original video. The depth detection model outputs a depth map corresponding to each video frame. The value of each pixel point in the depth map represents the depth value of the point. In one application scenario, each video frame corresponds to a depth threshold, which can be the average depth value of multiple pixel points in the depth map corresponding to the video frame. The depth threshold is used as a basis for searching the to-be-fused layer corresponding to the video frame. In another application scenario, the average depth values of multiple depth maps corresponding to multiple video frames are averaged to obtain a total average depth value as a depth threshold. The depth threshold is used as a basis for searching the to-be-fused layer in multiple decomposition layers of the entire original video.

[0095] In the process of searching for the to-be-fused layer, the depth threshold is used to determine which layers in the multiple decomposition layers are close to the camera (i.e., foreground layers) and which layers are far from the camera (i.e., background layers). These foreground layers contain elements closer to the foreground and may contain details of partially occluded target objects (such as branches, window railings, etc.). The foreground layer is not the main layer in the video processing process, but is determined as the to-be-fused layer.

[0096] Further, the to-be-fused layer (which can include multiple foreground layers) is fused into one layer to obtain a front occlusion layer. The layer fusion process can include pixel-level weighted averaging, maximum value selection or other fusion strategies to ensure that the front occlusion layer correctly reflects all occlusion details close to the camera. The generation of the front occlusion layer helps to more realistically simulate the occlusion effect of the foreground object in the video, especially when replacing the foreground object or performing depth synthesis.

[0097] Still taking the above application scenario as an example, a pre-trained depth detection model (such as depth-anything) is used to extract the depth value of the original video frame to determine the average depth value of each frame of the object layer of the multiple original video frames as the depth threshold. Further, when the original video frame is decomposed into layers, the part of the original video frame with a depth value less than the depth threshold is determined as the front occlusion layer, and the part of the original video frame excluding the object layer and the front occlusion layer is determined as the scene layer.

[0098] By the steps S2216 to S2218, the depth value of the original video is determined by the depth detection model, and the depth threshold is further determined, so that the object layer and the front occlusion layer can be accurately distinguished, and the accuracy and efficiency of subsequent content replacement of different layers in the video are improved.

[0099] In an optional embodiment, in step S222, the plurality of decomposed layers are feature encoded based on the video content to be replaced, to obtain an encoding result, including at least one of the following method steps:

[0100] In step S223, the motion feature of the object layer is feature encoded based on the video content to be replaced, to obtain motion encoding.

[0101] In step S224, the appearance feature of the object layer is feature encoded based on the video content to be replaced, to obtain appearance encoding.

[0102] In step S225, the scene feature of the scene layer and the occlusion feature of the front occlusion layer are feature encoded based on the video content to be replaced, to obtain scene and occlusion encoding.

[0103] As still shown in FIG. 3, based on the plurality of decomposed layers obtained by spatial hierarchical decomposition, the corresponding object video, scene video and front occlusion video can be obtained; the object video, scene video and front occlusion video are respectively modeled and feature encoded, to obtain an encoding result (including scene and occlusion encoding); and the encoding result is input into the diffusion generator according to a preset rule, to generate a target video, which includes a plurality of synthesized video frames.

[0104] Specifically, in the object modeling of the object video, the motion feature is feature encoded based on the video content to be replaced, to obtain motion encoding, wherein the motion feature is obtained by spatial motion representation rendering of the object video. The appearance feature is feature encoded based on the video content to be replaced, to obtain appearance encoding, wherein the appearance feature is obtained by normalized identity decoupling (i.e., ID decoupling) of the object video and the video content to be replaced.

[0105] By the steps S223 to S225, the plurality of decomposed layers obtained by spatial hierarchical decomposition are feature encoded, to obtain multiple dimensions of encoding, so as to generate the target video in multiple dimensions.

[0106] In the embodiment of the disclosure, when the object modeling is performed, the appearance feature and the motion feature of the decomposed object video are decoupled, to obtain motion encoding and appearance encoding, respectively.

[0107] In an optional embodiment, in step S223, the action features of the object layer are feature-encoded based on the video content to be replaced, to obtain an action code, including the following method steps:

[0108] In step S2231, the torso posture estimation and the hand posture estimation are performed on the object layer, to obtain the posture parameters and the camera parameters corresponding to the target object;

[0109] In step S2232, the model mesh corresponding to the target object is deformed, to obtain a deformation result, wherein the deformation result is used to determine the spatial position of the model surface layer code corresponding to the target object under the posture parameters, and the model surface layer code is a surface layer code pre-anchored on the model surface layer vertex corresponding to the target object.

[0110] In step S2233, the action code is generated based on the camera parameters and the deformation result.

[0111] The torso posture estimation and the hand posture estimation on the object layer include: performing the torso posture estimation and the hand posture estimation on the target object in the object layer. When the target object is a human object, in the process of performing the human posture estimation on the human object, the basic posture estimation is performed on the main stem part of the human body (including the head, the chest, the waist and the legs) to complete the torso posture estimation; and the detailed posture estimation is performed on the hands of the human body to complete the hand posture estimation.

[0112] It should be noted that in the field of computer vision, the key point positions (such as the positions of multiple points of the shoulder, the waist, the hip and the like) of the human body are detected and estimated from the image or the video by using a deep learning model (such as an algorithm based on a convolutional neural network), to complete the torso posture estimation, so as to infer the posture of the torso. Due to the high degree of freedom and the complex posture change of the hands, a more fine model algorithm is needed to complete the hand posture estimation.

[0113] Through the torso posture estimation and the hand posture estimation, the posture parameters and the camera parameters corresponding to the target object are obtained. The posture parameters refer to the numerical values describing the posture of the target object obtained from the torso posture estimation and the hand posture estimation, such as joint angles, position coordinates and the like. The camera parameters involve the imaging characteristics of the camera, including the focal length, the optical center position, the image sensor size and the like, as well as the pose (position and posture) of the camera relative to the scene, which are used to understand the perspective relationship of the image.

[0114] The model mesh corresponding to the target object can be a structure composed of vertices, edges and faces, used to represent the shape of the three-dimensional model corresponding to the target object. Further, by applying the pose parameters to the model mesh, the positions of the vertices in the model mesh are changed, so that the shape of the three-dimensional model of the target object conforms to the pose described by the pose parameters, thereby obtaining a deformed model mesh. As a deformation result, the deformed model mesh can determine the spatial position of the model surface layer code of the target object under the pose parameters. Through the deformation of the model mesh, the pose information can be converted into the geometric change of the model surface, providing a basis for subsequent coding.

[0115] Further, based on the camera parameters and the deformation result, an action code is generated, which is to convert the action features of the target object into a numerical representation understandable by a computer, usually through a neural network model (action encoder). The action code can efficiently represent the action information for subsequent action replacement in video processing.

[0116] In an optional embodiment, in step S2233, based on the camera parameters and the deformation result, an action code is generated, including the following method steps:

[0117] Step S2234, mapping the deformation result to a two-dimensional plane based on the camera parameters to obtain a mapping result;

[0118] Step S2235, texture interpolation calculation is performed on the mapping result to obtain an initial motion representation corresponding to the target object;

[0119] Step S2236, replacing the initial motion representation with a target motion representation according to the video content to be replaced;

[0120] Step S2237, merging the target motion representation in the time dimension to obtain a merging result;

[0121] Step S2238, encoding the merging result using an action encoder to obtain an action code.

[0122] The deformation result of the model mesh corresponding to the three-dimensional model is mapped to a two-dimensional image coordinate system using the camera parameters to obtain a mapping result. The above mapping process can be realized by a projection algorithm, and the mapping result is recorded as the projection result of the deformation result on a two-dimensional plane. Through two-dimensional mapping, the three-dimensional information in the deformation result can be converted into a two-dimensional form suitable for image processing and video processing.

[0123] Further, texture interpolation calculation is performed on the mapping result to fill in the gaps or discontinuous texture areas that may be caused by projection, to generate an initial motion representation of the target object. The above texture interpolation calculation can be realized by various algorithms, such as bilinear interpolation, cubic spline interpolation, etc.

[0124] According to the to-be-replaced video content, the initial motion representation is adjusted to be more matched with the motion of the target object, and the target motion representation is generated. The motion representation replacement process can involve fine-tuning the motion of the target object to ensure that the replaced motion is consistent with the environment and motion in the target video.

[0125] Further, the target motion representation is merged in the time axis (temporal dimension) to generate a continuous motion sequence, obtain a merging result, and ensure the continuity of the motion in time. Then, the merging result is input into a motion encoder to generate a compact and efficient motion code. The motion encoder is a neural network model for extracting and encoding motion features, and the output of the motion encoder can be used as input for subsequent video processing stages.

[0126] As can be seen from the above, the embodiments of the present disclosure can provide a complex but efficient process for extracting motion information of a target object from an original video and generating new, matched, and encoded motion features based on to-be-replaced video content, thereby providing technical support for video content modification and innovation.

[0127] According to the above execution steps, in the application scenario corresponding to the video generation framework as shown in FIG. 3, a set of surface codes Z is first defined, the set of surface codes Z includes a plurality of features (such as 10475), and the plurality of features are anchored on a plurality of structured human body model surface vertices (such as 10475) of SMPLX. Then, a three-dimensional human pose estimation algorithm is used to perform torso pose estimation and hand pose estimation on video frames in the object video frame by frame to obtain pose parameters of SMPLX under the three-dimensional human body model and corresponding camera parameters; a linear blend skinning algorithm (LBS) is used to deform the human body model grid, thereby obtaining the spatial position of the surface code Z under the pose parameters; further, a differentiable rasterizer is used to map the deformed surface code Z to a two-dimensional plane based on the camera parameters, and a UV texture coordinate interpolation algorithm is used to obtain a numerical continuous rendering feature map as an initial motion representation.

[0128] At this time, if the to-be-replaced video content is used to determine the target motion representation, the initial motion representation is replaced by the target motion representation according to the to-be-replaced video content; further, the target motion representation is merged in the time sequence dimension to obtain a merging result, and the merging result is input into a motion encoder constructed by three-dimensional convolution to output a motion code.

[0129] By the steps S2231 to S2237, the motion representation of the human model displayed in the original video is extracted based on the structured surface layer coding, and the mapping relationship between the two-dimensional rendering result and the surface dense points of the underlying three-dimensional human model is established, so that the three-dimensional rotation distortion and the occlusion relationship between the skeletal muscles of the human model in the space movement are effectively expressed, and the model can more deeply understand the spatial movement of the human model in the video processing process.

[0130] In an optional embodiment, in step S224, the appearance features of the object layer are feature coded based on the video content to be replaced, to obtain appearance coding, including the following method steps:

[0131] Step S2241, obtaining the original appearance representation of the object layer;

[0132] Step S2242, replacing the original appearance representation with a target appearance representation according to the video content to be replaced;

[0133] Step S2243, performing pose transformation on the target appearance representation using a pose transformation model to obtain a transformation result, wherein the pose transformation model is used to map any pose displayed in the target frame to a target pose space;

[0134] Step S2244, encoding the transformation result using an appearance encoder to obtain appearance coding.

[0135] The original appearance representation of the object layer can be obtained by quality evaluation and screening of the object layer.

[0136] In the process of obtaining appearance coding based on the decomposed object video when modeling the object, first, the object layer of the original video frame is quality evaluated and filtered and screened to obtain an original appearance representation. The original appearance representation can correspond to a representative frame (which can also be regarded as a partial frame to be replaced) obtained by screening. Based on this, the original appearance representation is replaced with a target appearance representation according to the video content to be replaced. Further, a pre-trained human pose transformation model is used to map any pose of the human model to a standard pose space to obtain a transformation result. The human model in the standard pose space is in an A-shaped distribution, i.e., the transformation result at least includes a standardized human model image.

[0137] Further, the transformation result is input into a pre-trained appearance encoder to obtain corresponding appearance coding. The appearance encoder can be composed of a U-shaped network.

[0138] By the steps S2241 to S2244, in the process of encoding the object layer (or the corresponding object video), the normalization decoupling manner between the appearance feature and the motion feature introduced by the embodiment of the disclosure can effectively reduce the interference of the appearance information of the object and the running representation learning, so that the model can better learn the mapping relationship between the motion feature and the target pose, reduce the information loss in the motion encoding, and further enhance the authenticity and accuracy of the generated target video.

[0139] In an optional embodiment, in step S225, the scene features of the scene layer and the occlusion features of the front occlusion layer are feature encoded based on the to-be-replaced video content to obtain scene and occlusion encoding, including the following method steps:

[0140] Step S2251, replacing the original scene features of the scene layer with target scene features based on the to-be-replaced video content;

[0141] Step S2252, encoding the target scene features using a variational autoencoder to obtain scene encoding, and feature encoding the occlusion features of the front occlusion layer using a variational autoencoder to obtain occlusion encoding;

[0142] Step S2253, connecting the scene encoding and the occlusion encoding in feature dimension to obtain scene and occlusion encoding.

[0143] In the above optional embodiment, as still shown in FIG. 3, in the process of scene modeling for the plurality of decomposition layers obtained by spatial hierarchical decomposition, first, the scene features of the original scene (i.e., original scene features) exhibited in the scene layer are replaced with the scene features of the target scene (i.e., target scene features) exhibited in the to-be-replaced video content based on the to-be-replaced video content, and second, the target scene features are input into a pre-trained scene encoder to obtain compressed scene encoding. For example, the scene encoder can be a variational autoencoder (VAE).

[0144] As still shown in FIG. 3, in the process of occlusion modeling for the plurality of decomposition layers obtained by spatial hierarchical decomposition, the occlusion features of the front occlusion layer are input into a pre-trained variational autoencoder (VAE) to obtain compressed occlusion encoding.

[0145] As still shown in FIG. 3, the compressed scene encoding and the occlusion encoding are connected in feature dimension using an adder to obtain scene and occlusion encoding.

[0146] Through the steps S2251 to S2253, the embodiment of the present disclosure enables the model to automatically generate the front occlusion content corresponding to the replaced object based on the front occlusion information through occlusion modeling, produces the interaction effect between the replaced object and the front occlusion, and improves the quality of the generated target video.

[0147] In an optional embodiment, in step S2252, the target scene feature is feature-encoded by using a variational autoencoder to obtain scene encoding, including the following method steps:

[0148] In step S2254, time sequence information of the target scene feature is obtained.

[0149] In step S2255, the missing content associated with the target scene feature is completed based on the time sequence information to obtain a completion result.

[0150] In step S2256, the completion result is encoded by using a variational autoencoder to obtain scene encoding.

[0151] In the above optional embodiment, in the process of scene modeling on the plurality of decomposition layers obtained by spatial hierarchical decomposition, a video inpainting algorithm based on optical flow can also be used to integrate the time sequence information of the target scene feature, automatically complete the missing content in the scene video corresponding to the target scene feature, and obtain a completed scene video; further, the completed scene video is input into a pre-trained scene encoder to obtain compressed scene encoding.

[0152] Through the steps S2254 to S2256, the embodiment of the present disclosure completes the time sequence missing content by using the target scene feature, ensures the integrity of the scene encoding, and further ensures the coherence and fluency of the generated target video.

[0153] In an optional embodiment, in step S23, the encoding result is combined to generate a target video, including the following method steps:

[0154] In step S231, the action encoding, the appearance encoding, and the scene and occlusion encoding are combined by using a diffusion generator to generate a target video.

[0155] In the optional embodiment described above, a combined video generation method is provided, which simultaneously inputs the action encoding, appearance encoding, and scene and occlusion encoding as control conditions of the diffusion generator, obtains the reconstructed video sequence output by the diffusion generator, and obtains the target video based on the video sequence. The diffusion generator can adopt a diffusion structure based on a three-dimensional U-shaped network, and the diffusion generator is trained by taking the mean square error of the prediction noise and the actual noise as the loss function for gradient descent, thereby optimizing the diffusion generator, the appearance encoder, and the action encoder in the video generation framework. That is, the video generation framework provided by the embodiment of the present disclosure realizes content reconstruction between the original video (corresponding to the original video frame) and the target video (corresponding to the synthesized video frame) through reverse denoising.

[0156] Through the above step S231, the embodiment of the present disclosure combines the encoding of multiple dimensions after spatial hierarchical decomposition, normalized decoupling of appearance and action in object modeling, and scene and occlusion encoding, thereby enhancing the generation quality of the target video.

[0157] In addition, in the above video processing scheme provided by the embodiment of the present disclosure, after the network training of the video generation framework is completed, the target person image can be used to replace the appearance feature video, so as to change the person model displayed in the finally generated target video; the target scene video can be used to replace the scene feature video to realize scene editing and replacement; similarly, by inputting a new action sequence parameter, the original action of the person model in the original video can be replaced by a new action.

[0158] It should be noted that the video generation framework shown in FIG. 3 only shows a scheme for object modeling based on video content to be replaced, which is suitable for application scenarios in which the video content to be replaced contains target objects to be replaced, and can replace the original object in the original video frame with the target object to obtain a synthesized video frame. In addition to this, the above-mentioned video generation framework can also be applied to application scenarios in which the video content to be replaced contains target scenes to be replaced, and the video content to be replaced is input into scene modeling to realize replacement of the original scene in the original video frame with the target scene to obtain a synthesized video frame. Alternatively, the above-mentioned video generation framework can also be applied to application scenarios in which the video content to be replaced contains target front occluders to be replaced, and the video content to be replaced is input into occlusion modeling to realize replacement of the original front occluder in the original video frame with the target front occluder to obtain a synthesized video frame.

[0159] Compared with the three-dimensional video generation scheme provided in the related art, the end-to-end controllable video editing scheme is adopted in the embodiment of the disclosure, the real video data is modeled by spatial hierarchical decomposition, a more realistic and natural video rendering result is achieved, the retention degree of user input information is higher, and the appearance and motion of the character model are more natural. Moreover, the end-to-end model inference and deployment in the embodiment of the disclosure are relatively simple, and the cost is lower.

[0160] Compared with the previous two-dimensional video generation scheme provided in the related art, the embodiment of the disclosure can effectively process the interaction and occlusion relationship between objects and scenes by modeling the scene, object and front occlusion in the original video in a spatial hierarchical manner, and embody the object and scene interaction performance. In addition, the embodiment of the disclosure can effectively realize the motion generation of the object in the video under complex space by structuring the coding of the motion features of the object and modeling the standardized appearance of the object, so that the model has stronger expression ability and generalization for complex spatial motion.

[0161] As can be seen from the above, the video processing method provided by the embodiment of the disclosure provides a video spatial hierarchical modeling idea, constructs a decomposition modeling scheme of a display object in a video, and decomposes and models the scene, object appearance, object motion and front occlusion of the video content. Based on this, the embodiment of the disclosure can provide a user with video editing functions such as appearance control, motion control, background control and foreground control of a character model in a picture frame.

[0162] The embodiment of the disclosure decomposes and models the scene, character appearance and motion representation, combines and controls generation to realize various control modes such as character role replacement, scene editing and motion control in the video; the spatial hierarchical modeling is performed through depth information, so that the model can learn the automatic generation of the post-subject and front content, and realize the occlusion interaction between people and scene objects; the motion representation of the structured coding and the standardized ID modeling method are adopted to effectively decouple the character appearance and motion information, and improve the complex motion generation capability. In terms of specific scene application, the above-mentioned video processing method provided by the embodiment of the disclosure can be widely applied to video role replacement, scene fusion, dynamic character generation and the like.

[0163] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions, and provide corresponding operation portal for user to select authorization or refusal.

[0164] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all described as a combination of a series of actions, but those skilled in the art should know that the disclosure is not limited by the order of the actions described, because according to the disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the disclosure.

[0165] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, and of course it can also be realized by hardware. Based on such understanding, the technical solutions of the disclosure can be embodied in the form of a software product in essence or in the form of a part of the prior art that makes a contribution. The computer software product is stored in a storage medium (such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk), and includes a plurality of instructions for causing an end device (which can be a mobile phone, a computer, a server, or a network device) to execute the method of each embodiment of the disclosure.

[0166] Under the foregoing operating environment, the disclosure further provides a video processing method as shown in FIG. 4. FIG. 4 is a flowchart of a video processing method according to an embodiment of the disclosure, as shown in FIG. 4, the video processing method comprises:

[0167] Step S41, obtaining an original role video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace an original character role displayed by the original role video with a target character role;

[0168] Step S42, performing spatial feature decomposition coding on the original role video based on the video content to be replaced, to obtain a coding result;

[0169] Step S43, combining the coding result to generate a target role video.

[0170] The original role video can include a plurality of original role video frames. At least an original task role is displayed in the original role video.

[0171] Based on the video content to be replaced, spatial feature decomposition coding can be performed on the original role video on a large amount of video data to obtain a coding result. The coding result can correspond to a plurality of coding dimensions, such as appearance coding, action coding, scene coding, model coding, etc.

[0172] Further, in the encoding process, the relevant encoding in the original role video can be replaced according to the editing requirement of replacing the original character role with the target character role corresponding to the original character video. Based on this, the encoding result is subjected to combined video generation processing to obtain the target role video containing the target character role.

[0173] The video processing method can be applied to provide video processing services for preset application scenarios, especially character role replacement services in video editing, to obtain a target role video containing a target character role. The preset application scenarios can include, but are not limited to, scenarios involving video processing, video playing, video editing, etc. in the fields of e-commerce, education, medical treatment, conferences, social networks, financial products, logistics, navigation, etc.

[0174] In the embodiments of the present disclosure, an original role video to be edited and video content to be replaced are obtained, wherein the video content to be replaced is used to replace an original character role displayed by the original role video with a target character role; the original role video is subjected to spatial representation decomposition encoding based on the video content to be replaced to obtain an encoding result; and the encoding result is combined to generate a target role video. Thus, based on the original role video and the video content to be replaced, the present disclosure adopts the technical concept of spatial representation decomposition encoding combined with combined video generation to perform video data processing and editing generation in the encoding dimension, enhances the adaptability between the video content to be replaced and the non-replaced part of the original role video, and achieves the purpose of generating a target role video that is more accurate and has stronger realism, thereby realizing the technical effects of improving the scene adaptability, rendering realism and accuracy of video processing, and further solving the technical problems of poor scene adaptability, poor rendering realism and accuracy of the video generated by the video processing scheme in the prior art for the content to be replaced.

[0175] It should be noted that the preferred embodiments of steps S41 to S44 can refer to the related descriptions described above, and will not be described here.

[0176] In the foregoing operating environment, the present disclosure provides a video processing method as shown in FIG. 5. FIG. 5 is a flowchart of a video processing method according to an embodiment of the present disclosure. As shown in FIG. 5, the video processing method includes:

[0177] In step S51, a video processing request is obtained through a first application programming interface, wherein the request data carried in the video processing request includes: an original video to be edited and video content to be replaced, the video content to be replaced being used to replace a foreground target and / or a background target displayed by the original video;

[0178] Step S52, returning a video processing response through the second application programming interface, wherein the response data carried in the video processing response includes a target video, the target video being generated by combining the encoding result, the encoding result being obtained by performing spatial feature decomposition coding on the original video based on the video content to be replaced.

[0179] In the embodiments of the present disclosure, the video processing request is obtained through the first application programming interface, wherein the request data carried in the video processing request includes an original video to be edited and video content to be replaced, the video content to be replaced being used to replace foreground objects and / or background objects displayed in the original video; and the video processing response is returned through the second application programming interface, wherein the response data carried in the video processing response includes a target video, the target video being generated by combining the encoding result, the encoding result being obtained by performing spatial feature decomposition coding on the original video based on the video content to be replaced. Based on the original video and the video content to be replaced, the technical concept of spatial feature decomposition coding combined with combined video generation is adopted in the embodiments of the present disclosure, the video data processing and editing generation are performed in the coding dimension, the adaptability between the video content to be replaced and the non-replaced part of the original video is enhanced, the purpose of generating a target video with higher accuracy and stronger reality is achieved, thereby realizing the technical effects of improving the scene adaptability, rendering reality and accuracy of video processing, and further solving the technical problems of poor scene adaptability, poor rendering reality and accuracy of the video generated by the video processing scheme in the prior art.

[0180] It should be noted that the preferred embodiments of steps S51 to S54 can refer to the related descriptions in the foregoing embodiments, which will not be described here again.

[0181] In the foregoing operating environment, the present disclosure further provides a video processing method as shown in FIG. 6. FIG. 6 is a flowchart of a video processing method according to an embodiment of the present disclosure, a graphical user interface is provided through a terminal device, as shown in FIG. 6, the video processing method includes:

[0182] Step S61, selecting an original video to be edited in response to a first touch operation performed on the graphical user interface;

[0183] Step S62, uploading video content to be replaced in response to a second touch operation performed on the graphical user interface, the video content to be replaced being used to replace foreground objects and / or background objects displayed in the original video;

[0184] Step S63, generating a target video in response to a third touch operation performed on the graphical user interface, the target video being generated by combining the encoding result, the encoding result being obtained by performing spatial feature decomposition coding on the original video based on the video content to be replaced;

[0185] Step S64, displaying the target video in the graphical user interface.

[0186] The graphical user interface can be a graphical user interface of a client corresponding to the video processing method. The graphical user interface can include a first touch component configured to select an original video to be edited in response to a first touch operation, that is, the user performs a first touch operation on the first touch component to specify the original video to be edited.

[0187] The graphical user interface can further include a second touch component configured to obtain user-uploaded video content to be replaced in response to a second touch operation, that is, the user performs a second touch operation on the second touch component to specify the video content to be replaced, thereby describing the requirement for replacing the foreground object and / or the background object displayed in the original video in the video processing.

[0188] The graphical user interface can further include a third touch component configured to trigger the video processing method provided by the foregoing embodiments of the present disclosure to generate a target video in response to a third touch operation. Further, after the target video is generated, the target video is displayed in the graphical user interface.

[0189] It should be noted that the first touch operation, the second touch operation, and the third touch operation can all be operations of the user touching the display screen of the terminal device with a finger and touching the terminal device. The touch operation can include single-point touch and multi-point touch, wherein the touch operation of each touch point can include clicking, long pressing, double-clicking, swiping, etc. The input operation and the sending operation can also be touch operations implemented through a mouse, a keyboard, or other input devices.

[0190] Through the foregoing optional embodiments, the present disclosure provides a visual implementation of the video processing method, which can be implemented on a client to support the user to specify the original video to be edited, the video content to be replaced, and trigger the start of the target video generation process. This way is user-friendly, convenient, and good user experience.

[0191] It should be noted that the preferred embodiments of steps S61 to S64 can refer to the foregoing related description, which will not be repeated here.

[0192] In the foregoing operating environment, the present disclosure further provides a model training method as shown in FIG. 7. FIG. 7 is a flowchart of a model training method according to an embodiment of the present disclosure. The model training method includes the following steps:

[0193] Step S71, obtaining a sample video;

[0194] Step S72, training the initial content reconstruction model using the sample video to obtain a target content reconstruction model;

[0195] The initial content reconstruction model is used to obtain appearance encoding corresponding to the sample video, action encoding corresponding to the sample video, scene and occlusion encoding corresponding to the sample video, and to combine the action encoding, the appearance encoding, and the scene and occlusion encoding to obtain the prediction noise.

[0196] The sample video is a video material used for model training. These video materials contain instances of various scenes, actions, and appearances, which can help the content reconstruction model learn how to handle different video features, such as character actions, scene details, and occlusion conditions.

[0197] The initial content reconstruction model is the model before training. The initial content reconstruction model also includes an appearance encoder, an action encoder, a scene encoder, and a diffusion generator, but the model parameters of the initial content reconstruction model are initial parameters that have not been optimized through training. The target content reconstruction model is a model that has been trained and parameter-adjusted. The target content reconstruction model can more accurately reconstruct the content and generate high-quality target videos.

[0198] In the process of training the initial content reconstruction model using the sample video to obtain the target content reconstruction model, the sample video is used as input to let the model learn how to generate and combine appearance, action, and scene and occlusion encoding to generate predicted video content or features.

[0199] Optionally, the initial content reconstruction model includes an appearance encoder, an action encoder, a scene encoder, and a diffusion generator. The appearance encoder is used to generate appearance encoding corresponding to the sample video. The action encoder is used to generate action encoding corresponding to the sample video. The scene encoder is used to generate scene and occlusion encoding corresponding to the sample video. The diffusion generator is used to combine the action encoding, the appearance encoding, and the scene and occlusion encoding.

[0200] In an optional embodiment, in step S72, the initial content reconstruction model is trained using the sample video to obtain a target content reconstruction model, including the following method steps:

[0201] Step S721, training the initial content reconstruction model using the sample video to obtain prediction noise;

[0202] Step S722, calculating a content reconstruction loss based on the prediction noise and actual noise corresponding to the sample video;

[0203] Step S723, adjusting parameters of the appearance encoder, parameters of the action encoder, and parameters of the diffusion generator according to the content reconstruction loss to obtain the target content reconstruction model.

[0204] The prediction noise is a difference or error generated by the model during the training process that does not completely match the sample video features. The difference or error can be regarded as "noise" between the model prediction and the actual sample. The model is trained using the sample video, and the model generates action encoding, appearance encoding, and scene and occlusion encoding according to the input video content, and combines the encodings through the diffusion generator. The difference between the predicted video content and the original video content is the prediction noise.

[0205] The content reconstruction loss can be a difference between the predicted content generated by the content reconstruction model and the actual content of the sample video, and is an important indicator in the training process of the model, used to guide the adjustment of the model parameters. The loss function used to calculate the content reconstruction loss can be a mean square error function, a cross-entropy loss function, or the like.

[0206] Further, by using an optimization algorithm (such as gradient descent), the parameters of the appearance encoder, the action encoder, and the diffusion generator are updated according to the calculation result of the content reconstruction loss, so that the model can more accurately reconstruct the video content during the next prediction, and the prediction noise is reduced. The final target content reconstruction model has better performance and accuracy.

[0207] In the embodiments of the present disclosure, the training step in the model training method can further include other specific implementation details. For details, refer to the description of the video generation framework shown in FIG. 3 in the foregoing embodiments, which will not be repeated here.

[0208] According to the embodiments of the present disclosure, a device embodiment for implementing the video processing method is also provided. FIG. 8 is a structural schematic diagram of a video processing device according to an embodiment of the present disclosure. As shown in FIG. 8, the device includes:

[0209] The acquisition module 801 is configured to acquire an original video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace a foreground target and / or a background target displayed in the original video;

[0210] The encoding module 802 is configured to perform spatial feature decomposition coding on the original video based on the video content to be replaced, to obtain a coding result.

[0211] The generation module 803 is configured to combine the coding result to generate a target video.

[0212] Optionally, the encoding module 802 is further configured to perform spatial hierarchical decomposition on the original video according to depth information of the original video, to obtain a plurality of decomposition layers, wherein the plurality of decomposition layers include a scene layer, an object layer, and a front occlusion layer; and perform feature coding on the plurality of decomposition layers based on the video content to be replaced, to obtain the coding result.

[0213] Optionally, the encoding module 802 is further configured to: decompose the original video into an object layer by using an instance segmentation model; extract depth values of the original video by using a depth detection model to obtain a front occlusion layer; and determine the object layer and layers other than the front occlusion layer as a scene layer.

[0214] Optionally, the encoding module 802 is further configured to: perform instance segmentation on a target object in the original video by using an instance segmentation model to obtain a segmentation result; and extract an object mask from the segmentation result according to an object detection bounding box to obtain an object layer.

[0215] Optionally, the encoding module 802 is further configured to: extract depth values of the original video by using a depth detection model to obtain a depth threshold, wherein the depth threshold is a frame-by-frame average depth value of the object layer; and find a to-be-fused layer based on the depth threshold, wherein the depth of the to-be-fused layer is lower than the depth threshold; and fuse the to-be-fused layer to obtain a front occlusion layer.

[0216] Optionally, the encoding module 802 is further configured to: encode motion features of the object layer based on the to-be-replaced video content to obtain motion encoding; encode appearance features of the object layer based on the to-be-replaced video content to obtain appearance encoding; and encode scene features of the scene layer and occlusion features of the front occlusion layer based on the to-be-replaced video content to obtain scene and occlusion encoding.

[0217] Optionally, the encoding module 802 is further configured to: perform torso pose estimation and hand pose estimation on the object layer to obtain pose parameters and camera parameters corresponding to the target object; deform a model mesh corresponding to the target object to obtain a deformation result, wherein the deformation result is used to determine a spatial position of a model surface layer code corresponding to the target object under the pose parameters, and the model surface layer code is a surface layer code pre-anchored on a model surface layer vertex corresponding to the target object; and generate the motion encoding based on the camera parameters and the deformation result.

[0218] Optionally, the encoding module 802 is further configured to: map the deformation result to a two-dimensional plane based on the camera parameters to obtain a mapping result; perform texture interpolation calculation on the mapping result to obtain an initial motion representation corresponding to the target object; replace the initial motion representation with a target motion representation according to the to-be-replaced video content; merge the target motion representation in a time sequence dimension to obtain a merged result; and encode the merged result by using a motion encoder to obtain the motion encoding.

[0219] Optionally, the encoding module 802 is further configured to: obtain an original appearance representation of the object layer; replace the original appearance representation with a target appearance representation according to the video content to be replaced; perform pose transformation on the target appearance representation by using a pose transformation model to obtain a transformation result, wherein the pose transformation model is used to map any pose shown in the target frame to a target pose space; and encode the transformation result by using an appearance encoder to obtain the appearance encoding.

[0220] Optionally, the encoding module 802 is further configured to: replace original scene features of the scene layer with target scene features according to the video content to be replaced; encode the target scene features by using a variational autoencoder to obtain scene encoding, and encode the occlusion features of the front occlusion layer by using the variational autoencoder to obtain occlusion encoding; and connect the scene encoding and the occlusion encoding in feature dimension to obtain scene and occlusion encoding.

[0221] Optionally, the encoding module 802 is further configured to: obtain time sequence information of the target scene features; complete missing content associated with the target scene features based on the time sequence information to obtain a completion result; and encode the completion result by using a variational autoencoder to obtain the scene encoding.

[0222] Optionally, the generation module 803 is further configured to: combine the action encoding, the appearance encoding, and the scene and occlusion encoding by using a diffusion generator to generate the target video.

[0223] It should be noted that the obtaining module 801, the encoding module 802, and the generation module 803 correspond to steps S21 to S23 in the embodiments, and the three modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in the foregoing embodiments.

[0224] According to the embodiments of the present disclosure, a device embodiment for implementing the video processing method in the foregoing embodiments is also provided. FIG. 9 is a structural schematic diagram of another video processing device according to an embodiment of the present disclosure. As shown in FIG. 9, the device includes:

[0225] The obtaining module 901 is configured to obtain an original character video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace an original character shown in the original character video with a target character.

[0226] The encoding module 902 is configured to perform spatial representation decomposition encoding on the original character video based on the video content to be replaced to obtain an encoding result.

[0227] The generation module 903 is configured to combine the encoding result to generate a target character video.

[0228] It should be noted that the above obtaining module 901, encoding module 902 and generating module 903 correspond to the foregoing steps S41 to S43, and the three modules have the same instances and application scenarios as the corresponding steps, but are not limited to the contents disclosed in the above embodiments.

[0229] According to the embodiments of the present disclosure, a device embodiment for implementing the video processing method in the above embodiments is also provided. FIG. 10 is a structural schematic diagram of another video processing device according to an embodiment of the present disclosure. As shown in FIG. 10, the device comprises:

[0230] The request module 1001 is configured to obtain a video processing request through a first application programming interface, wherein the request data carried in the video processing request comprises: an original video to be edited and video content to be replaced, the video content to be replaced being used to replace foreground objects and / or background objects shown in the original video;

[0231] The response module 1002 is configured to return a video processing response through a second application programming interface, wherein the response data carried in the video processing response comprises: a target video, the target video being generated by combining the encoding result, the encoding result being obtained by performing spatial feature decomposition coding on the original video based on the video content to be replaced.

[0232] It should be noted that the above request module 1001 and response module 1002 correspond to the foregoing steps S51 to S52, and the two modules have the same instances and application scenarios as the corresponding steps, but are not limited to the contents disclosed in the above embodiments.

[0233] According to the embodiments of the present disclosure, a device embodiment for implementing the video processing method in the above embodiments is also provided. FIG. 11 is a structural schematic diagram of another video processing device according to an embodiment of the present disclosure. The above video processing device provides a graphical user interface through a terminal device, as shown in FIG. 11, the device comprises:

[0234] The selection module 1101 is configured to select an original video to be edited in response to a first touch operation performed on the graphical user interface.

[0235] The replacement module 1102 is configured to upload video content to be replaced in response to a second touch operation performed on the graphical user interface, the video content to be replaced being used to replace foreground objects and / or background objects shown in the original video.

[0236] The generation module 1103 is configured to generate a target video in response to a third touch operation performed on the graphical user interface, the target video being generated by combining the encoding result, the encoding result being obtained by performing spatial feature decomposition coding on the original video based on the video content to be replaced.

[0237] The display module 1104 is configured to display the target video in the graphical user interface.

[0238] It should be noted that the selection module 1101, the replacement module 1102, the generation module 1103 and the display module 1104 correspond to the aforementioned steps S61 to S64, and the four modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments.

[0239] According to the embodiments of the present disclosure, a device embodiment for implementing the model training method in the above embodiments is also provided. FIG. 12 is a structural schematic diagram of a model training device according to an embodiment of the present disclosure. As shown in FIG. 12, the device comprises:

[0240] The acquisition module 1201 is configured to acquire sample videos.

[0241] The training module 1202 is configured to train the initial content reconstruction model by using the sample videos to obtain a target content reconstruction model; wherein the initial content reconstruction model is used to acquire appearance encoding corresponding to the sample videos, action encoding corresponding to the sample videos, scene and occlusion encoding corresponding to the sample videos, and to combine the action encoding, the appearance encoding and the scene and occlusion encoding to acquire predicted noise.

[0242] Optionally, the initial content reconstruction model comprises an appearance encoder, an action encoder, a scene encoder and a diffusion generator, the appearance encoder is used to generate appearance encoding corresponding to the sample videos, the action encoder is used to generate action encoding corresponding to the sample videos, the scene encoder is used to generate scene and occlusion encoding corresponding to the sample videos, and the diffusion generator is used to combine the action encoding, the appearance encoding and the scene and occlusion encoding.

[0243] Optionally, the training module 1202 is further configured to train the initial content reconstruction model by using the sample videos to obtain predicted noise, calculate a content reconstruction loss based on the predicted noise and actual noise corresponding to the sample videos, and adjust parameters of the appearance encoder, parameters of the action encoder and parameters of the diffusion generator according to the content reconstruction loss to obtain the target content reconstruction model.

[0244] It should be noted that the acquisition module 1201 and the training module 1202 correspond to the aforementioned steps S71 to S72, and the two modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments.

[0245] It should be noted that the above modules or units can be hardware components or software components stored in the memory and processed by one or more processors, and the above modules can also be run in a computer terminal as part of the device.

[0246] It should be noted that the preferred embodiments of the present embodiment can refer to the related description in the foregoing embodiments, which will not be repeated here.

[0247] According to the embodiments of the present disclosure, an electronic device is also provided, which can be any one of the terminal devices in the computer terminal group. Alternatively, in the present embodiment, the electronic device can also be replaced by a terminal device such as a mobile terminal.

[0248] Alternatively, in the present embodiment, the electronic device can be located in at least one of the network devices in the computer network.

[0249] In the present embodiment, the electronic device can execute program codes of the following steps in the video processing method: obtaining an original video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace foreground objects and / or background objects displayed in the original video; performing spatial feature decomposition coding on the original video based on the video content to be replaced to obtain a coding result; and combining the coding result to generate a target video.

[0250] Alternatively, FIG. 13 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 13, the electronic device 130 can include one or more (only one is shown in the figure) processors 1302, a memory 1304, a storage controller 1306, and a peripheral interface 1308, wherein the peripheral interface 1308 is connected with a radio frequency module, an audio module, and a display.

[0251] The memory 1304 can be used to store software programs and modules, such as program instructions / modules corresponding to the video processing method and device in the embodiments of the present disclosure. The processor executes various function applications and data processing by running the software programs and modules stored in the memory, that is, the video processing method described above is implemented. The memory 1304 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 1304 can further include a memory remotely arranged with respect to the processor, which can be connected to the electronic device 130 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0252] The processor 1302 can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining an original video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace foreground objects and / or background objects displayed in the original video; performing spatial feature decomposition coding on the original video based on the video content to be replaced to obtain a coding result; and combining the coding result to generate a target video.

[0253] According to the embodiments of the present disclosure, a scheme of an electronic device for implementing the video processing method is provided. The original video to be edited and the video content to be replaced are obtained, wherein the video content to be replaced is used to replace foreground objects and / or background objects displayed in the original video; spatial feature decomposition coding is performed on the original video based on the video content to be replaced to obtain a coding result; and the coding result is combined to generate a target video. Thus, based on the original video and the video content to be replaced, the technical concept of spatial feature decomposition coding combined with combined video generation is adopted, video data processing and editing generation are performed in the coding dimension, the adaptability between the video content to be replaced and the non-replaced part of the original video is enhanced, the purpose of generating a target video with higher accuracy and stronger reality is achieved, the technical effects of improving the scene adaptability, rendering reality and accuracy of video processing are achieved, and the technical problems of the video processing scheme in the prior art that the video generated for the content to be replaced has poor scene adaptability, poor rendering reality and poor accuracy are solved.

[0254] Those skilled in the art can understand that the structure shown in FIG. 13 is only schematic, and the electronic device can also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), etc. FIG. 13 does not limit the structure of the above-mentioned electronic device. For example, the electronic device 130 can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG. 13, or have a different configuration from that shown in FIG. 13.

[0255] Those skilled in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by programs instructing the relevant hardware of the terminal device, and the programs can be stored in a computer readable storage medium, which can include a flash disk, a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0256] According to the embodiments of the present disclosure, a computer readable storage medium is also provided. Optionally, in the present embodiment, the above-mentioned storage medium can be used to save the program code executed by the video processing method provided by the above-mentioned embodiments.

[0257] Optionally, in the embodiment, the storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0258] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: obtaining an original video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace foreground objects and / or background objects shown in the original video; performing spatial feature decomposition coding on the original video based on the video content to be replaced to obtain a coding result; and combining the coding result to generate a target video.

[0259] The embodiment of the present disclosure further provides a computer program product. Optionally, in the embodiment, the computer program product can include a computer program, and the computer program is used to implement the method provided by the above embodiment when executed by a processor.

[0260] The embodiment of the present disclosure further provides a computer program product. Optionally, the computer program product can include a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium can be used to store a computer program, and the computer program is used to implement the method provided by the above embodiment when executed by a processor.

[0261] The embodiment of the present disclosure further provides a computer program. Optionally, in the embodiment, the computer program is used to implement the method provided by the above embodiment when executed by a processor.

[0262] In the above embodiments of the present disclosure, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0263] In several embodiments provided by the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic, for example, the division of units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0264] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0265] In addition, each functional unit in various embodiments of the present disclosure can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0266] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present disclosure. The foregoing storage medium includes: U disk, ROM, RAM, mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0267] The above is only the preferred embodiment of the present disclosure, and it should be pointed out that for those skilled in the art, without departing from the principles of the present disclosure, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present disclosure.

Claims

1. A method of video processing, comprising: obtaining an original video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace foreground objects and / or background objects shown in the original video; performing spatial representation decomposition coding on the original video based on the video content to be replaced to obtain a coding result; combining the coding result to generate a target video.

2. The video processing method of claim 1, wherein, performing spatial representation decomposition coding on the original video based on the video content to be replaced to obtain the coding result comprises: performing spatial hierarchical decomposition on the original video according to depth information of the original video to obtain a plurality of decomposition layers, wherein the plurality of decomposition layers comprise a scene layer, an object layer, and a front occlusion layer; performing feature coding on the plurality of decomposition layers based on the video content to be replaced to obtain the coding result.

3. The video processing method of claim 2, wherein, performing spatial hierarchical decomposition on the original video according to the depth information of the original video to obtain the plurality of decomposition layers comprises: decomposing the object layer from the original video using an instance segmentation model; extracting depth values of the original video using a depth detection model to obtain the front occlusion layer; determining the scene layer as layers other than the object layer and the front occlusion layer.

4. The video processing method of claim 3, wherein, decomposing the object layer from the original video using the instance segmentation model comprises: performing instance segmentation on target objects in the original video using the instance segmentation model to obtain a segmentation result; performing object mask extraction on the segmentation result according to an object detection bounding box to obtain the object layer.

5. The video processing method of claim 3, wherein, extracting depth values of the original video using the depth detection model to obtain the front occlusion layer comprises: extracting depth values of the original video using the depth detection model to obtain a depth threshold, wherein the depth threshold is a frame-by-frame average depth value of the object layer; finding a to-be-fused layer based on the depth threshold, wherein the to-be-fused layer has a depth lower than the depth threshold; fusing the to-be-fused layer to obtain the front occlusion layer.

6. The video processing method of claim 2, wherein, performing feature coding on the plurality of decomposition layers based on the video content to be replaced to obtain the coding result comprises at least one of: performing feature coding on motion features of the object layer based on the video content to be replaced to obtain motion coding; performing feature coding on appearance features of the object layer based on the video content to be replaced to obtain appearance coding; performing feature coding on scene features of the scene layer and occlusion features of the front occlusion layer based on the video content to be replaced to obtain scene and occlusion coding.

7. The video processing method of claim 6, wherein, performing feature coding on motion features of the object layer based on the video content to be replaced to obtain the motion coding comprises: performing trunk pose estimation and hand pose estimation on the object layer to obtain pose parameters and camera parameters corresponding to target objects; deforming a model mesh corresponding to the target object to obtain a deformation result, wherein the deformation result is used to determine a spatial position of a model surface layer corresponding to the target object under the pose parameter, and the model surface layer is a surface layer code pre-anchored on a model surface layer vertex corresponding to the target object; generating the motion code based on the camera parameter and the deformation result.

8. The video processing method of claim 7, wherein, The generating of the motion code based on the camera parameter and the deformation result comprises: mapping the deformation result to a two-dimensional plane based on the camera parameter to obtain a mapping result; performing texture interpolation calculation on the mapping result to obtain an initial motion representation corresponding to the target object; replacing the initial motion representation with a target motion representation according to the video content to be replaced; merging the target motion representation in the time dimension to obtain a merging result; encoding the merging result by using a motion encoder to obtain the motion code.

9. The video processing method of claim 6, wherein, The feature encoding of the appearance feature of the object layer based on the video content to be replaced to obtain the appearance code comprises: obtaining an original appearance representation of the object layer; replacing the original appearance representation with a target appearance representation according to the video content to be replaced; performing pose transformation on the target appearance representation by using a pose transformation model to obtain a transformation result, wherein the pose transformation model is used to map any pose displayed in a target frame to a target pose space; encoding the transformation result by using an appearance encoder to obtain the appearance code.

10. The video processing method of claim 6, wherein, The feature encoding of the scene feature of the scene layer and the occlusion feature of the front occlusion layer based on the video content to be replaced to obtain the scene and occlusion code comprises: replacing the original scene feature of the scene layer with a target scene feature based on the video content to be replaced; encoding the target scene feature by using a variational autoencoder to obtain a scene code, and encoding the occlusion feature of the front occlusion layer by using the variational autoencoder to obtain an occlusion code; connecting the scene code and the occlusion code in the feature dimension to obtain the scene and occlusion code.

11. The video processing method of claim 10, wherein, The feature encoding of the target scene feature by using the variational autoencoder to obtain the scene code comprises: obtaining time sequence information of the target scene feature; completing missing content associated with the target scene feature based on the time sequence information to obtain a completion result; encoding the completion result by using the variational autoencoder to obtain the scene code.

12. The video processing method of claim 6, wherein, The combination of the encoding result to generate the target video comprises: combining the motion code, the appearance code and the scene and occlusion code by using a diffusion generator to generate the target video.

13. A video processing method, comprising: obtaining an original role video to be edited and video content to be replaced, wherein the video content to be replaced is used to replace an original character role displayed by the original role video with a target character role; performing spatial representation decomposition coding on the original role video based on the video content to be replaced to obtain an encoding result; combining the encoding results to generate a target role video.

14. A video processing method, comprising: obtaining a video processing request through a first application programming interface, wherein request data carried in the video processing request comprises an original video to be edited and video content to be replaced, the video content to be replaced being used to replace foreground objects and / or background objects displayed by the original video; returning a video processing response through a second application programming interface, wherein response data carried in the video processing response comprises a target video, the target video being generated by combining encoding results obtained by spatially characterizing and decomposing the original video based on the video content to be replaced.

15. A video processing method, providing a graphical user interface through a terminal device, the video processing method comprising: selecting an original video to be edited in response to a first touch operation performed on the graphical user interface; uploading video content to be replaced in response to a second touch operation performed on the graphical user interface, the video content to be replaced being used to replace foreground objects and / or background objects displayed by the original video; generating a target video in response to a third touch operation performed on the graphical user interface, the target video being generated by combining encoding results obtained by spatially characterizing and decomposing the original video based on the video content to be replaced; displaying the target video in the graphical user interface.

16. A model training method, comprising: obtaining a sample video; training an initial content reconstruction model using the sample video to obtain a target content reconstruction model; wherein the initial content reconstruction model is used to obtain appearance encoding corresponding to the sample video, action encoding corresponding to the sample video, scene and occlusion encoding corresponding to the sample video, and to combine the action encoding, the appearance encoding, and the scene and occlusion encoding to obtain predicted noise.

17. The model training method of claim 16, wherein, The initial content reconstruction model comprises an appearance encoder, an action encoder, a scene encoder, and a diffusion generator, the appearance encoder being used to generate appearance encoding corresponding to the sample video, the action encoder being used to generate action encoding corresponding to the sample video, the scene encoder being used to generate scene and occlusion encoding corresponding to the sample video, and the diffusion generator being used to combine the action encoding, the appearance encoding, and the scene and occlusion encoding. 18.The model training method of claim 17, wherein, Training the initial content reconstruction model using the sample video to obtain the target content reconstruction model comprises: training the initial content reconstruction model using the sample video to obtain the predicted noise; calculating a content reconstruction loss based on the predicted noise and actual noise corresponding to the sample video; adjusting parameters of the appearance encoder, parameters of the action encoder, and parameters of the diffusion generator according to the content reconstruction loss to obtain the target content reconstruction model.

19. An electronic device, comprising: a memory storing an executable program; A processor configured to execute the program, wherein the program performs the video processing method of any one of claims 1-15 or the model training method of any one of claims 16-18 when the program is executed.

20. A computer readable storage medium comprising a stored executable program, wherein, A computer readable storage medium storing an executable program, wherein the program performs the video processing method of any one of claims 1-15 or the model training method of any one of claims 16-18 when the program is executed.

21. A computer program product comprising a computer program which, when executed by a processor, implements the video processing method of any one of claims 1-15 or the model training method of any one of claims 16-18.

Images