Entity object replacement method and device, storage medium and electronic equipment

Abstract

The invention relates to the technical field of computer vision, and particularly provides an entity object replacement method and device, a storage medium and electronic equipment, and the method can comprise the steps: extracting an object parameter of a target object in a video frame; wherein the object parameters comprise attitude parameters and camera parameters; rendering and generating a replacement image based on the object parameters and a pre-reconstructed three-dimensional object model; and fusing and repairing the replacement image and the background image of the video frame to obtain a composite image for replacing the target object in the video frame. According to the embodiment of the invention, the entity object can be accurately replaced, and the video quality is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method, apparatus, storage medium, and electronic device for entity object replacement. Background Technology

[0002] With the rapid development of multimedia technology, the demand for video content creation is increasing day by day. As a key technology, the technology of replacing physical objects in videos has received widespread attention.

[0003] Currently, the traditional method for replacing physical objects involves processing each frame independently using 2D techniques. However, this method easily leads to flickering, jittering, and instability in the replaced character during video playback. For example, in some short videos that use 2D methods for character replacement, viewers can clearly see that the replaced character's body movements appear to be jumping, and the transitions between adjacent frames are unnatural, severely disrupting the video's visual continuity and watchability, and reducing video quality.

[0004] Therefore, how to provide a technical solution for a more effective method of entity object replacement has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of some embodiments of this application is to provide a method, apparatus, storage medium, and electronic device for entity object replacement. The technical solutions of the embodiments of this application can improve the realism of entity object replacement and ensure video quality.

[0006] In a first aspect, some embodiments of this application provide a method for entity object replacement, comprising: extracting object parameters of a target object in a video frame; wherein the object parameters include pose parameters and camera parameters; rendering a replacement image based on the object parameters and a pre-reconstructed three-dimensional object model; and fusing and repairing the replacement image with the background image of the video frame to obtain a composite image in the video frame that replaces the target object.

[0007] Some embodiments of this application can render and generate replacement images based on object parameters of the target object and a reconstructed 3D object model. The replacement image is then fused and repaired with the background image to obtain a replaced composite image. Some embodiments of this application can drive the generation of composite images based on a 3D object model, improving the realism of video frame replacement, resulting in highly stable and smooth video timing after replacement, and ensuring video quality.

[0008] In some embodiments, extracting object parameters of a target object from a video frame includes: decomposing the target video to obtain an image frame sequence; wherein the video frame is any image frame in the image frame sequence; and using a three-dimensional human pose estimation tool to extract parameters from the video frame to obtain the object parameters.

[0009] Some embodiments of this application obtain object parameters by decomposing the target video and then using a three-dimensional human pose estimation tool to extract parameters from any video frame, thereby achieving accurate capture of the target object's motion information.

[0010] In some embodiments, the three-dimensional object model is obtained by: predicting the object morphology parameters of the source image using a three-dimensional human pose estimation tool; constructing an initial three-dimensional model that matches the object morphology parameters and a preset standard pose; analyzing the source image using a target neural network model to obtain a model deviation value; and superimposing the model deviation value and the initial value of the initial three-dimensional model to obtain the three-dimensional object model.

[0011] Some embodiments of this application construct an initial 3D model from source images, then use a target neural network model to obtain model deviation values, and perform comprehensive processing with the initial 3D model to obtain a high-fidelity, drivable 3D object model, thereby realizing the effective construction of 3D digital models.

[0012] In some embodiments, the step of rendering a replacement image based on the object parameters and a pre-reconstructed 3D object model includes: applying the pose parameters to the 3D object model to obtain a driving 3D model; and inputting the driving 3D model and the camera parameters into a Gaussian splash renderer to render and generate the replacement image.

[0013] Some embodiments of this application achieve accurate and effective replacement of the target object by setting pose parameters for the 3D object model and using a Gaussian splash renderer in combination with camera parameters to generate a replacement image.

[0014] In some embodiments, fusing and repairing the replacement image with the background image of the video frame to obtain a composite image replacing the target object in the video frame includes: obtaining a processed image by modifying the alpha channel of the replacement image; superimposing the processed image onto the background image to obtain an initial composite image; and performing transition region repair on the initial composite image to obtain the composite image.

[0015] Some embodiments of this application obtain an initial composite image by modifying and replacing the alpha channel of the image and then overlaying it with a background image. After that, the composite image with better effect is obtained by repairing it. Subsequently, the rendered object can be seamlessly embedded into the target video.

[0016] In some embodiments, the step of performing transition region repair on the initial synthesized image to obtain the synthesized image includes: determining the transition region based on the object mask of the target object extracted from the video frame; inputting the initial synthesized image, the transition region, the background image, and the processed image into an image inpainting model, and outputting the synthesized image.

[0017] Some embodiments of this application use an image inpainting model to repair the initial synthetic image after determining the transition region, thereby obtaining a high-quality, realistic and natural synthetic image.

[0018] In some embodiments, obtaining the processed image by modifying the alpha channel of the replacement image includes: obtaining the scene depth map of the video frame and the object depth map of the target object; and setting the pixel value in the alpha channel by comparing the scene depth map and the object depth map to obtain the processed image.

[0019] Some embodiments of this application modify the alpha channel by comparing scene depth maps and object depth maps, providing data support for obtaining high-quality, realistic and natural synthetic images.

[0020] Secondly, some embodiments of this application provide an apparatus for replacing an entity object, comprising: an extraction module for extracting object parameters of a target object in a video frame; wherein the object parameters include pose parameters and camera parameters; a rendering module for rendering a replacement image based on the object parameters and a pre-reconstructed three-dimensional object model; and a compositing module for fusing and repairing the replacement image with the background image of the video frame to obtain a composite image of the target object in the video frame.

[0021] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0022] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0023] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 System diagrams for entity object replacement provided for some embodiments of this application; Figure 2 One of the flowcharts for entity object replacement provided in some embodiments of this application; Figure 3 Flowchart 2 of the method for entity object replacement provided for some embodiments of this application; Figure 4 A flowchart illustrating entity object replacement provided for some embodiments of this application; Figure 5 Block diagrams of apparatus for entity object replacement provided for some embodiments of this application; Figure 6 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0026] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] In related technologies, 2D methods are typically used for frame-by-frame independent processing, lacking effective perception of 3D geometry and failing to accurately handle complex occlusion and collision relationships between entities (such as people and animals) and the scene. For example, in a scene where a person passes through a railing, the person replaced using traditional methods often passes directly through the railing instead of being naturally occluded by it, or, when occluded by an object, exhibits an unnatural display effect, resulting in severe distortion of the replacement effect. This geometric inconsistency greatly weakens the realism and credibility of the video content, allowing viewers to easily perceive the "fakeness" of the video, thus affecting the effectiveness of the video in conveying information.

[0029] As can be seen from the above-mentioned related technologies, the existing methods for replacing entity objects in videos are ineffective, reduce video quality, and are likely to bring a bad viewing experience to the audience.

[0030] In view of this, some embodiments of this application provide a method for entity object replacement. To address the temporal inconsistency problem caused by 2D frame-by-frame processing, this method employs a scheme combining a parametric human model and 3D Gaussian reconstruction. By extracting the human pose (i.e., pose parameters) and camera parameters (i.e., camera parameters) of each video frame from the target video, accurate capture of human motion information in the video is achieved. Then, innovatively, a 3D Gaussian point cloud is initialized using the SMPLX model, and Gaussian parameters (e.g., color, transparency, offset, etc.) are predicted from a single source image to quickly reconstruct a high-fidelity, actuated 3D digital human (i.e., a 3D object model). The entire target video processing is driven by a unified 3D digital human asset, rather than processing each frame independently, thus fundamentally solving the problem of inter-frame jitter and resulting in a highly stable and smooth temporal sequence for the replaced video.

[0031] To address the geometric glitches present in existing technologies, traditional methods, lacking 3D geometric perception, cannot correctly handle the complex occlusion and collision relationships between characters and scenes. This application fully utilizes the geometric information provided by 3D Gaussian in the reconstruction of the 3D digital human. In the pose-driven and rendering stage, the pose parameters of the target video drive the movement of the 3D digital human, and depth-sensing fusion technology is used to generate an initial synthetic image based on 3D geometric information. This initial synthetic image can accurately identify the positional relationships between the character and objects in the scene. Furthermore, by combining this with an image inpainting model to handle boundaries and occlusions, the replaced character can correctly interact with the scene in 3D, avoiding glitches such as passing through railings or being unnaturally occluded by objects, greatly improving the realism of the video replacement effect.

[0032] The following is in conjunction with the appendix Figure 1 The overall structure of the entity object replacement system provided by some embodiments of this application is illustrated by way of example.

[0033] like Figure 1 As shown, some embodiments of this application provide a system for entity object replacement. This entity object replacement system may include a terminal 100 and a server 200. The terminal 100 can send the target video to be replaced to the server 200. The server 200 can decompose the target video, extract object parameters from each video frame, and then render a replacement image using the object parameters and a reconstructed 3D digital human. Finally, the replacement image and the background image are fused and repaired to obtain a composite image for replacement.

[0034] In some embodiments of this application, the terminal 100 can be a mobile terminal or a non-portable computer terminal, and the embodiments of this application are not specifically limited here.

[0035] The following is in conjunction with the appendix Figure 2 The present application provides an exemplary implementation process for entity object replacement performed by server 200, based on some embodiments of this application.

[0036] Please see the appendix Figure 2 , Figure 2 A flowchart of a method for entity object replacement is provided for some embodiments of this application. The entity object replacement method may include: S210, extract the object parameters of the target object in the video frame; wherein, the object parameters include pose parameters and camera parameters.

[0037] For example, in a specific embodiment of this application, the target object is a person, and the precise capture of the person's movement information can be achieved by extracting the person's parameters (as a specific example of object parameters).

[0038] In some embodiments of this application, S210 may include: decomposing the target video to obtain an image frame sequence; wherein the video frame is any image frame in the image frame sequence; and using a three-dimensional human pose estimation tool to extract parameters from the video frame to obtain the object parameters.

[0039] For example, in a specific embodiment of this application, the SMPLX (Skinned Multi-Person Linear Model eXtended, a parametric 3D human body model) model is used to extract the human pose from each video frame of the target video, the pose parameters and shape parameters of the human body are obtained through the 3D human body pose estimation model, and the camera parameters corresponding to each video frame are obtained using camera technology.

[0040] Specifically, after server 200 acquires the target video A, it decomposes it into a continuous sequence of image frames {I_t}, where t is the frame number. To facilitate the explanation of the implementation process of this application, any video frame in {I_t} is used as an example. Using an existing 3D human pose estimation tool based on SMSLX, each frame I_t is input into the tool, and a set of SMSLX parameters Θ_t = (β, θ, φ) is obtained through regression. Here, β is the shape parameter, which remains constant throughout the target video and is the average of the estimated values ​​of all video frames. θ is the pose parameter, i.e., the global rotation and joint pose parameters of the human body, which are different for each video frame. φ is the facial expression parameter, which is different for each video frame. It should be understood that these parameters can be selected according to the actual replacement scenario; for example, if facial expression parameters are needed, they can be extracted for subsequent operations; otherwise, they can be left unextracted. This embodiment of the application does not impose specific limitations here. In addition, using the same 3D human pose estimation tool, the weak perspective camera parameters cam_t = (s, tx, ty) corresponding to each video frame are extracted for subsequent rendering.

[0041] Since the processing of the target video in this application is driven by a unified 3D digital human, the 3D digital human needs to be constructed before performing the following operations.

[0042] In some embodiments of this application, the three-dimensional object model (i.e., 3D digital human) is obtained by the following method: predicting the object morphology parameters of the source image using a three-dimensional human pose estimation tool; constructing an initial three-dimensional model that matches the object morphology parameters and a preset standard pose; analyzing the source image using a target neural network model to obtain model deviation values; and superimposing the model deviation values ​​and the initial values ​​of the initial three-dimensional model to obtain the three-dimensional object model.

[0043] For example, in a specific embodiment of this application, a 3D Gaussian point cloud is initialized based on SMPLX model data, where each Gaussian point is represented by parameters such as center position, scale, color, and transparency. Using a deep learning algorithm, Gaussian parameters are predicted from a single source image, and a human model (i.e., a 3D digital human) is reconstructed.

[0044] Specifically, acquire the source image B and predict its shape β_init using a 3D human pose estimation tool (as a concrete example of object shape parameters). Set an SMPLX model to a standard pose (such as A-Pose, as a concrete example of a preset standard pose), and set its shape parameter to β_init. At this point, the SMPLX model represents the target body shape. Use the position of each vertex of this SMPLX model as the center point (x, y, z) of an initial 3D Gaussian ellipsoid. This represents that the Gaussian point cloud forms a human body shape from the beginning (as a concrete example of an initial 3D model).

[0045] Construct a target neural network model with an encoder-decoder structure, such as based on the Vision Transformer. Input the source image B into this target neural network model, and its output is a Gaussian attribute set, which is the bias value for predicting the attribute of each Gaussian center point. Specifically, the bias values ​​include center point offset (Δx, Δy, Δz): used for fine-tuning the position and depicting details such as clothing and hairstyle; color RGB: the color represented by the Gaussian point; transparency α: controlling whether the Gaussian point is visible; rotation r and scaling s, used to define the anisotropic covariance and control the shape of the Gaussian. Add the predicted bias values ​​to the initial values ​​corresponding to the human body shape to obtain the final unique 3D Gaussian model G (i.e., 3D digital human) used to represent the person.

[0046] S220, based on the object parameters and the pre-reconstructed 3D object model, render and generate a replacement image.

[0047] For example, in a specific embodiment of this application, a replacement image is generated by combining object parameters and the 3D digital human constructed above with related technologies.

[0048] In some embodiments of this application, S220 may include: applying the pose parameters to the three-dimensional object model to obtain a driving three-dimensional model; inputting the driving three-dimensional model and the camera parameters into a Gaussian splash renderer to render and generate the replacement image.

[0049] For example, in a specific embodiment of this application, the pose parameters of the SMPLX of any video frame in the target video are applied to the 3D digital human to drive its movement. Using 3D Gaussian splashing technology, the 3D Gaussian point cloud is projected onto a 2D image plane according to a formula to generate an RGBA image R_t for each frame (as a specific example of a replacement image).

[0050] Specifically, the pose parameter θ_t is applied to the 3D Gaussian model G using the Linear Blend Skinning (LBS) algorithm. Since each Gaussian center point is bound to an SMPLX vertex, they move with the skeleton, thus completing the actuation. The actuated 3D model and camera parameter cam_t are then input into a differentiable Gaussian splash renderer. The renderer efficiently renders an RGBA image R_t (color image + alpha channel) using a tile-based rasterizer and α-blending.

[0051] S230, the replacement image is fused and repaired with the background image of the video frame to obtain a composite image in the video frame that replaces the target object.

[0052] For example, in a specific embodiment of this application, a pre-trained instance segmentation model (such as the SegmentAnything Model, SAM) is used to extract a high-precision binary mask {M_t} of a person from a video frame. {M_t} is used to erase video frame I_t to obtain the background image Bkg_init. R_t and Bkg_init are then combined using depth-aware fusion and inpainting techniques to obtain a synthetic image.

[0053] Specifically, in some embodiments of this application, S230 may include: S231, the processed image is obtained by modifying the Alpha channel of the replacement image.

[0054] For example, in a specific embodiment of this application, a smart mask alpha image is obtained by modifying the Alpha channel in R_t (as a specific example of the processed image).

[0055] In some embodiments of this application, S231 may include: acquiring a scene depth map of the video frame and an object depth map of the target object; and setting the pixel value in the Alpha channel by comparing the scene depth map and the object depth map to obtain the processed image.

[0056] For example, in a specific embodiment of this application, a depth map D_rend_t (as a specific example of an object depth map) is rendered from a 3D Gaussian renderer based on video frames, where each pixel value represents the z-depth of that Gaussian point. Simultaneously, a scene depth map D_scene_t is estimated from the video frames using a monocular depth estimation model. By comparing D_rend_t and D_scene_t, for the portion of the rendered character whose depth value is greater than the scene depth value (i.e., the portion that should be occluded), the alpha value (i.e., the algebraic value) of the rendered image's R_t is set to 0, thereby modifying the alpha channel of R_t.

[0057] S232, The processed image is superimposed on the background image to obtain the initial composite image.

[0058] For example, in a specific embodiment of this application, the processed R_t (processed occlusion) is superimposed on Bkg_init to obtain the preliminary composite image Comp_init (as a specific example of the initial composite image).

[0059] S233, perform transition region repair on the initial synthesized image to obtain the synthesized image.

[0060] For example, in a specific embodiment of this application, the transition region in Comp_init is repaired to obtain the final synthesized image.

[0061] In some embodiments of this application, S233 may include: determining the transition region based on the object mask of the target object extracted from the video frame; inputting the initial synthesized image, the transition region, the background image, and the processed image into an image inpainting model, and outputting the synthesized image.

[0062] For example, in a specific embodiment of this application, a morphological dilation operation is performed on the original mask M_t (as a specific example of an object mask) to obtain a wide boundary region mask M_boundary_t, which defines the transition region that needs to be repaired. Using R_t and Bkg_init as conditions, Comp_init and M_boundary_t are input into an image inpainting model (such as Stable Diffusion Inpainting). This powerful image inpainting model can effectively fuse boundary colors and textures through a minimum loss function, generating reasonable occluded content, unifying global illumination tones, and outputting a final, perfect composite frame I_final_t ​​(as a specific example of a composite image).

[0063] The following is in conjunction with the appendix Figure 3 The specific process of entity object replacement provided by some embodiments of this application is illustrated by way of example.

[0064] Please see the appendix Figure 3 , Figure 3 A flowchart illustrating a method for entity object replacement provided for some embodiments of this application.

[0065] The above process is illustrated below by example.

[0066] S310 uses a 3D human pose estimation tool to extract parameters from video frames to obtain object parameters.

[0067] For example, please see the appendix. Figure 4 , Figure 4 The first image in the image is the video frame that needs to be replaced.

[0068] S320: Extract people from video frames using an instance segmentation model to obtain a person mask, and use the person mask to process the video frames to obtain a background image.

[0069] For example, Figure 4 The second image in the image is the background image.

[0070] S330, based on source images, constructs 3D digital humans.

[0071] For example, Figure 4 The third image in the series is the source image, from which a 3D digital human can be constructed.

[0072] S340 applies the pose parameters to the 3D object model to obtain the driving 3D model; the driving 3D model and camera parameters are input into the Gaussian splash renderer to render and generate a replacement image.

[0073] For example, Figure 4 The fourth image in the series is the driving 3D model. The fifth image is the replacement image after Gaussian rendering.

[0074] S350 merges and repairs the replacement image with the background image of the video frame to obtain a composite image of the target object in the video frame.

[0075] For example, Figure 4 The sixth image in the series is a composite image obtained through fusion and post-processing (i.e., restoration).

[0076] It should be understood that although the above process is illustrated using any video frame in the target video as an example, in actual application scenarios, if the replacement of the person in the entire target video is to be achieved, the above operation can be performed repeatedly, that is, after the replacement of the current video frame is completed, the replacement of the next frame is performed; throughout the entire replacement process, the 3D digital human is used as a fixed asset and does not need to be reconstructed.

[0077] It should be noted that the specific implementation process of S310~S350 can be referred to the method embodiment provided above. To avoid repetition, detailed descriptions are omitted here.

[0078] As can be seen from the above embodiments of this application, the innovative approach of combining parametric human body models with 3D Gaussian reconstruction brings many beneficial effects compared to existing video character replacement technologies.

[0079] From a temporal consistency perspective, this invention extracts relevant parameters of a person's posture based on a parametric human model and uses 3D Gaussian reconstruction technology to create a unified 3D digital human asset. The entire video processing process is driven by this unified asset. This is like building a stable "skeleton" for the replaced person, making their movement in the video coherent and consistent, completely eliminating inter-frame jitter. Whether in fast-paced action scenes or subtle changes in facial expressions, the replaced person can be presented smoothly and naturally, bringing viewers an extremely smooth visual experience and greatly improving the quality and watchability of the video. This highly stable temporal consistency makes this technology extremely valuable in professional film and television production fields such as movies and TV series, meeting the stringent requirements for image stability.

[0080] In terms of geometric interactivity, existing technologies, lacking 3D geometric perception, cannot correctly handle the complex occlusion and collision relationships between characters and scenes, frequently resulting in continuity errors. This invention, leveraging geometric information provided by 3D Gaussian reconstruction, can accurately perceive the spatial positional relationships between characters and objects in the scene during pose-driven rendering. By generating intelligent masks through depth-sensing fusion technology and combining them with image restoration models, it intelligently processes the boundaries and occlusions between characters and scenes. This enables the replaced character to interact with the scene in 3D in accordance with realistic physical laws; for example, a character is naturally occluded behind a railing, and a reasonable collision effect is presented when in contact with objects. This realistic geometric interactivity significantly enhances the authenticity and credibility of video content, allowing viewers to be more immersed in the scene created by the video. It has a significant driving effect on fields such as virtual reality and augmented reality, which have extremely high requirements for scene realism.

[0081] Furthermore, the 3D Gaussian splashing technology upon which this invention relies possesses real-time rendering capabilities, supporting real-time preview and rapid export. During the creation process, creators can view the replacement effect in real time and adjust parameters promptly, significantly improving creative efficiency. Compared to traditional, time-consuming processing methods, this application, for the first time, makes high-quality video replacement technology practically applicable and industrially viable. This means that in industrial production scenarios such as film and television production and video special effects creation, video character replacement work can be completed quickly, reducing production costs and shortening the production cycle, thereby promoting the development of the entire video content creation industry.

[0082] Please refer to Figure 5 , Figure 5 The diagram illustrates the composition of an entity object replacement apparatus according to some embodiments of this application. It should be understood that this entity object replacement apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this entity object replacement apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0083] Figure 5 The entity object replacement device includes at least one software functional module that can be stored in a memory or embedded in the entity object replacement device in the form of software or firmware. The entity object replacement device includes: an extraction module 510, used to extract object parameters of a target object in a video frame; wherein the object parameters include pose parameters and camera parameters; a rendering module 520, used to render and generate a replacement image based on the object parameters and a pre-reconstructed 3D object model; and a compositing module 530, used to fuse and repair the replacement image with the background image of the video frame to obtain a composite image of the target object in the video frame.

[0084] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0085] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0086] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0087] like Figure 6 As shown, some embodiments of this application provide an electronic device 600, which includes a memory 610, a processor 620, and a computer program stored in the memory 610 and executable on the processor 620. When the processor 620 reads the program from the memory 610 via a bus 630 and executes the program, it can implement the methods of any of the above embodiments.

[0088] Processor 620 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 620 can be a microprocessor.

[0089] The memory 610 can be used to store instructions executed by the processor 620 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 620 of this disclosure embodiment can be used to execute the instructions in the memory 610 to implement the methods shown above. The memory 610 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0090] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0091] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for replacing entity objects, characterized in that, include: Extract object parameters of the target object from the video frame; wherein, the object parameters include pose parameters and camera parameters; Based on the object parameters and the pre-reconstructed 3D object model, a replacement image is rendered. The replacement image is fused and repaired with the background image of the video frame to obtain a composite image in the video frame that replaces the target object.

2. The method as described in claim 1, characterized in that, The extraction of object parameters from the target object in the video frame includes: The target video is decomposed to obtain an image frame sequence; wherein, the video frame is any image frame in the image frame sequence; The parameters of the object are obtained by extracting parameters from the video frames using a 3D human pose estimation tool.

3. The method as described in claim 1 or 2, characterized in that, The three-dimensional object model was obtained through the following method: Predict object morphological parameters in source images using 3D human pose estimation tools; Construct an initial 3D model that matches the object's morphological parameters and preset standard pose; The source image is analyzed using a target neural network model to obtain model bias values; The model deviation value and the initial value of the initial 3D model are superimposed to obtain the 3D object model.

4. The method as described in claim 1 or 2, characterized in that, The process of rendering and generating a replacement image based on the object parameters and the pre-reconstructed 3D object model includes: The attitude parameters are applied to the three-dimensional object model to obtain the driving three-dimensional model; The driving 3D model and the camera parameters are input into the Gaussian splash renderer to generate the replacement image.

5. The method as described in claim 1 or 2, characterized in that, The step of fusing and repairing the replacement image with the background image of the video frame to obtain a composite image replacing the target object in the video frame includes: The processed image is obtained by modifying the alpha channel of the replacement image; The processed image is superimposed on the background image to obtain the initial composite image; The initial synthesized image is repaired in transition regions to obtain the synthesized image.

6. The method as described in claim 5, characterized in that, The process of repairing the transition region of the initial synthesized image to obtain the synthesized image includes: The transition region is determined based on the object mask of the target object extracted from the video frame; The initial synthesized image, the transition region, the background image, and the processed image are input into the image restoration model, and the synthesized image is output.

7. The method as described in claim 5, characterized in that, The step of obtaining the processed image by modifying the alpha channel of the replacement image includes: Obtain the scene depth map of the video frame and the object depth map of the target object; The processed image is obtained by comparing the scene depth map and the object depth map and setting the pixel value in the alpha channel.

8. An apparatus for replacing an entity object, characterized in that, include: An extraction module is used to extract object parameters of a target object in a video frame; wherein, the object parameters include pose parameters and camera parameters; A rendering module is used to render and generate a replacement image based on the object parameters and a pre-reconstructed 3D object model; The compositing module is used to fuse and repair the replacement image with the background image of the video frame to obtain a composite image in the video frame that replaces the target object.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as claimed in any one of claims 1-7.

Images