Character model construction method and device, equipment, medium and program product

Abstract

The invention discloses a character model construction method and device, equipment, a medium and a program product, and the method comprises the steps: constructing a static three-dimensional model of a first target character according to the multi-source image data of the first target character; performing human body posture estimation on the target dynamic video to obtain human body posture parameters of the first target person; wherein the target dynamic video is a video containing the motion process of the first target person; according to the human body posture parameters of the first target person, carrying out posture driving on a static three-dimensional model of the first target person; and rendering the driven static three-dimensional model to a view angle consistent with the target dynamic video to obtain a three-dimensional model sequence of the first target person. By means of the method, the figure postures can be efficiently extracted from the complex dynamic video, high-fidelity three-dimensional reconstruction and multi-view rendering are completed, and the figure reconstruction effect is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, medium and program product for constructing a character model. Background Art

[0002] With the continuous development of mobile Internet technology and network communication technology, online live broadcasting, video entertainment, etc. have been rapidly developed and applied in people's daily work and life, and the development of visual processing technology has also received more and more attention.

[0003] Currently, NeRF (Neural Radiance Fields) technology is a deep learning-based 3D reconstruction technique that maps points in 3D space to color and density values, using volume rendering to render scenes. The SMPL (Skinned Multi-Person Linear Model) pose model is a parameterized human body model that generates a 3D human mesh based on pose and shape parameters. Both technologies are widely used in 3D model construction.

[0004] However, the inventors discovered that existing technologies have at least the following problems: NeRF and SMPL technologies have certain limitations in their application to complex dynamic scenes. For example, due to its high computational complexity and limited support for dynamic scenes, NeRF technology is more suitable for 3D reconstruction of static scenes and is difficult to directly apply to dynamic scenes such as football games. SMPL generally relies on multi-view input or laboratory environments, making it difficult to achieve high-precision pose estimation and reconstruction in monocular video. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, equipment, medium and program product for constructing a character model, which can efficiently extract character postures from complex dynamic videos and complete high-fidelity three-dimensional reconstruction and multi-view rendering, effectively improving the character reconstruction effect.

[0006] To achieve the above objectives, an embodiment of the present invention provides a method for constructing a character model, comprising:

[0007] constructing a static three-dimensional model of a first target person based on the multi-source image data of the first target person;

[0008] Performing human posture estimation on the target dynamic video to obtain human posture parameters of the first target person; wherein the target dynamic video is a video containing the movement process of the first target person;

[0009] Performing posture driving on the static three-dimensional model of the first target person according to the human body posture parameters of the first target person;

[0010] The driven static three-dimensional model is rendered to a viewing angle consistent with the target dynamic video to obtain a three-dimensional model sequence of the first target person.

[0011] As an improvement to the above solution, constructing a static three-dimensional model of the first target person based on the multi-source image data of the target person includes:

[0012] Acquiring multi-source image data of the first target person;

[0013] Inputting the multi-source image data into a preset neural radiation field model for calculation to generate an omnidirectional surround video of the first target person; wherein the neural radiation field model introduces the SMPL human body model as a geometric prior to constrain the learning process;

[0014] A static three-dimensional model of the first target person is generated based on the omnidirectional surround video.

[0015] As a preferred embodiment, the method further comprises:

[0016] Tracking and segmenting a second target person in the original video clip to generate background video and human body posture parameters of the second target person;

[0017] generating an omnidirectional surround video and human body posture parameters of the third target person according to target image data containing the third target person to be replaced;

[0018] The model of the third target person is reconstructed according to the body posture parameters of the second target person, the omnidirectional surround video and the body posture parameters of the third target person, and is fused with the background video to generate a replaced target video clip.

[0019] As an improvement to the above solution, the tracking and segmenting of the second target person in the original video clip to generate the background video and the body posture parameters of the second target person includes:

[0020] Using a preset target tracking and segmentation algorithm, tracking and segmenting the second target person in the original video clip, and outputting a person segmentation mask;

[0021] Segmenting the original video clip into a character video and a background video according to the character segmentation mask;

[0022] Performing human posture estimation on the person video to obtain human posture parameters of the second target person.

[0023] As an improvement to the above solution, generating an omnidirectional surround video and human body posture parameters of the third target person based on the target image data containing the third target person to be replaced includes:

[0024] Extracting content from the original video clip to generate text information describing the details of the character's clothing;

[0025] changing the clothes of the third target person in the target image data according to the text information to generate target image data after the clothes are changed;

[0026] performing facial enhancement processing on the third target person in the target image data after the change of clothing to generate enhanced target image data;

[0027] Processing the enhanced target image data using a preset multi-view diffusion model to generate an omnidirectional surround video;

[0028] Human body posture estimation is performed on the omnidirectional surround video to obtain human body posture parameters of the third target person.

[0029] As an improvement to the above solution, the method of reconstructing a model of the third target person based on the human body posture parameters of the second target person, the omnidirectional surround video of the third target person, and the human body posture parameters, and fusing the model with the background video to generate a replaced target video segment includes:

[0030] Constructing a static three-dimensional model of the third target person based on the omnidirectional surround video and human body posture parameters of the third target person;

[0031] Performing posture driving on the static three-dimensional model of the third target person according to the human body posture parameters of the second target person, and rendering the driven static three-dimensional model to obtain a motion video of the third target person;

[0032] The character motion video and the background video are fused to generate a replaced target video clip.

[0033] The embodiment of the present invention further provides a device for constructing a character model, comprising:

[0034] A static model building module, configured to build a static three-dimensional model of a first target person based on multi-source image data of the first target person;

[0035] A human posture estimation module is used to perform human posture estimation on a target dynamic video to obtain human posture parameters of the first target person; wherein the target dynamic video is a video containing the motion process of the first target person;

[0036] a human posture driving module, configured to perform posture driving on the static three-dimensional model of the first target person according to the human posture parameters of the first target person;

[0037] The 3D model sequence generation module is used to render the driven static 3D model to a viewing angle consistent with the target dynamic video to obtain a 3D model sequence of the first target person.

[0038] An embodiment of the present invention also provides a device for constructing a character model, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for constructing a character model as described in any one of the above items is implemented.

[0039] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-described methods for constructing a character model.

[0040] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or the computer instructions are executed by a processor, the method for constructing a character model as described in any one of the above is implemented.

[0041] Compared with the prior art, the character model construction method, device, equipment, medium and program product disclosed in the present invention construct a static three-dimensional model of the first target person based on multi-source image data of the first target person; perform human posture estimation on the target dynamic video to obtain the human posture parameters of the first target person; wherein the target dynamic video is a video containing the movement process of the first target person; according to the human posture parameters of the first target person, the static three-dimensional model of the first target person is posture-driven; the driven static three-dimensional model is rendered to a perspective consistent with the target dynamic video to obtain a three-dimensional model sequence of the first target person. The present invention adopts a two-stage strategy of modeling first and then driving. The pre-reconstruction stage focuses on high-precision static modeling to avoid noise interference in dynamic scenes, and the driving stage focuses on motion restoration to reduce computational complexity. It combines static high-quality character models with real dynamic scenes, fully utilizing the dynamic video information and the prior knowledge of the character three-dimensional model, and achieves high-precision and high-fidelity character reconstruction effects. It not only effectively improves the reconstruction accuracy, but also has good versatility and scalability, and is suitable for the needs of character modeling and virtual reproduction in a variety of complex dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1This is a flow chart of a method for constructing a character model provided by an embodiment of the present invention;

[0043] Figure 2 is a flow chart of another method for constructing a character model in an embodiment of the present invention;

[0044] Figure 3 1 is a schematic structural diagram of a device for constructing a character model provided by an embodiment of the present invention;

[0045] Figure 4 is a structural diagram of another device for constructing a character model according to an embodiment of the present invention;

[0046] Figure 5 It is a structural diagram of a device for constructing a character model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] In the description of this application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0049] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout this application, unless otherwise specified, "plurality" means two or more.

[0050] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0051] See also Figure 1 , is a flow chart of a method for constructing a character model provided by an embodiment of the present invention. The embodiment of the present invention provides a method for constructing a character model, the method comprising steps S11 to S14:

[0052] S11, constructing a static three-dimensional model of a first target person based on multi-source image data of the first target person;

[0053] S12. Estimating a human posture of the target dynamic video to obtain human posture parameters of the first target person; wherein the target dynamic video is a video containing a motion process of the first target person;

[0054] S13, performing posture driving on the static three-dimensional model of the first target person according to the body posture parameters of the first target person;

[0055] S14: Rendering the driven static 3D model to a viewing angle consistent with the target dynamic video to obtain a 3D model sequence of the first target person.

[0056] This embodiment of the present invention is suitable for applications where a 3D character model is reconstructed from videos captured in complex dynamic scenes (e.g., various ball games, ice skating events, and other sporting events). This embodiment proposes a two-stage reconstruction method for the 3D character model. This method leverages prior knowledge of the character model and the target dynamic video, completing the reconstruction of the character's 3D model through a pre-reconstruction phase and a driver rendering phase.

[0057] The pre-reconstruction stage utilizes existing multi-source information to construct a complete, detailed, and controllable 3D model of a person before formal 3D reconstruction. This stage involves acquiring multi-source image data of a first target person. This multi-source image data can come from previous motion videos of the target person, high-definition portrait photos of the target person, publicly available 3D data resources, or even standard character modeling data of the target person. Based on this multi-source image data, a pre-reconstructed initial static 3D model is obtained using a pre-set 3D model construction method.

[0058] The driven rendering stage focuses on restoring and dynamically modeling the character's movements in the actual target dynamic scene, adapting the static model to the dynamic poses in the target dynamic video to generate a time-consistent 3D sequence. This process takes the target dynamic video, such as a real football game video, as input and relies on multi-view synchronized camera images and corresponding calibration parameters to perform human pose estimation. The pose estimation results provide accurate skeletal information (such as SMPL pose parameters) of the player at each moment. Combined with the high-quality static character model obtained in the first stage, it is pose-driven to achieve dynamic reconstruction. Finally, the driven character model is rendered to a perspective consistent with the real video using camera parameters, completing the motion restoration and 3D reproduction of the first target character. In this process, the texture information and geometric structure are maintained consistent across different time frames, achieving a realistic restoration of the first target character's real-life movement, and ultimately outputting a high-fidelity textured 3D character model sequence.

[0059] The technical means of the embodiments of the present invention are adopted, and a two-stage strategy of modeling first and then driving is adopted. The pre-reconstruction stage focuses on high-precision static modeling to avoid noise interference in dynamic scenes. The driving stage focuses on motion restoration to reduce computational complexity. The static high-quality character model is combined with the real dynamic scene, and the dynamic video information and the prior knowledge of the character three-dimensional model are fully utilized to achieve high-precision and high-fidelity character reconstruction effects. It not only effectively improves the reconstruction accuracy, but also has good versatility and scalability, and is suitable for the needs of character modeling and virtual reproduction in a variety of complex dynamic scenes.

[0060] As a preferred embodiment, the embodiment of the present invention is further implemented on the basis of the above embodiment. Step S11, i.e., constructing a static three-dimensional model of the first target person based on the multi-source image data of the target person, includes steps S111 to S113:

[0061] S111, obtaining multi-source image data of the first target person;

[0062] S112: Inputting the multi-source image data into a preset neural radiation field model for calculation to generate an omnidirectional surround video of the first target person; wherein the neural radiation field model introduces the SMPL human body model as a geometric prior to constrain the learning process;

[0063] S113: Generate a static three-dimensional model of the first target person based on the omnidirectional surround video.

[0064] In an embodiment of the present invention, the InstantAvatar method based on Neural Radiance Fields (NeRF) is combined with multi-source image data of the first target person. InstantAvatar uses these images or videos for high-fidelity reconstruction to learn to generate a complete 360-degree surround video describing the player's image. Combined with the human SMPL model prior, it extracts precise geometric and texture information while maintaining structural constraints, thereby achieving high-quality three-dimensional modeling of the target player.

[0065] It should be noted that InstantAvatar is an efficient 3D human modeling and animation generation technology based on neural radiance fields (NeRF) and parametric human models (such as SMPL). It can reconstruct high-fidelity, dynamically drivable digital human bodies from monocular or multi-view videos in a short time.

[0066] Specifically, multi-source image data of the first target person is first acquired to ensure coverage of all parts of the first target person's body. The multi-source image data is then subjected to deblurring, alignment, background segmentation, and other processing to reduce noise interference. Next, the multi-source image data is input into the NeRF model, and an implicit 3D representation of the player, including density and color fields, is learned through volume rendering. The NeRF framework introduces the SMPL human body model as a geometric prior to constrain the NeRF learning process. SMPL provides basic human topology (such as joint proportions and body shape) to prevent geometric distortion caused by occlusion or low resolution. NeRF is optimized within the SMPL parameterized space, preserving details (such as muscle lines and jersey folds) while ensuring anatomical rationality. Finally, the trained NeRF model is used to synthesize virtual images of the first target person from any perspective to obtain a 360-degree surround video. By filling in the angles missing from the real data, a textured static 3D model is output.

[0067] Compared with traditional methods that directly rely on low-quality frames in the video for reconstruction, the technical means of the embodiments of the present invention can significantly improve the detail quality and geometric consistency of the final model by introducing prior knowledge. In addition, the SMPL model provides an anatomically reasonable initial shape, and NeRF is used to refine the details, thereby solving the failure problem of traditional methods under occlusion or low resolution.

[0068] As a preferred embodiment, the driving rendering stage is further explained in detail. Step S12, that is, performing human posture estimation on the target dynamic video to obtain the human posture parameters of the first target person, includes:

[0069] Extracting multi-view synchronized frames of the first target person from the target dynamic video;

[0070] The multi-view synchronized frames are processed using a pre-trained model to estimate SMPL pose parameters.

[0071] Optionally, the SMPL posture parameters include joint rotation parameters, translation parameters, etc. for each frame.

[0072] Furthermore, step S13, i.e., performing posture driving on the static three-dimensional model of the first target person according to the body posture parameters of the first target person, includes:

[0073] The static three-dimensional model of the first target person is bound to the estimated SMPL posture parameters to achieve posture driving of the static three-dimensional model of the first target person.

[0074] Specifically, Linear Blend Skinning (LBS) is used to drive the static 3D model to deform to match the actions (e.g., running, kicking a ball) in the target dynamic video. When the pose changes, the texture information based on NeRF is dynamically adjusted to avoid stretching or penetration at the joints.

[0075] Furthermore, step S14, namely rendering the driven static 3D model to a viewing angle consistent with the target dynamic video to obtain a 3D model sequence of the first target person, includes:

[0076] According to the camera calibration parameters, the driven static 3D model is rendered to the perspective corresponding to the target dynamic video. During the rendering process, a differentiable renderer is used to generate a synthetic image, which is compared with the target dynamic video frame. By optimizing the posture parameters and lighting conditions, the rendering error is minimized. The continuous frames are time-series smoothed to eliminate jitter and output a textured 3D model sequence.

[0077] By adopting the technical means of the embodiments of the present invention, high-precision posture extraction and driving can be achieved, and the accuracy and visual effect of the reconstruction results can be significantly improved.

[0078] It should be noted that the target person's two-stage reconstruction module is designed for the reconstruction of a single person in a dynamic scene, and does not address broader practical applications, such as allowing the image of other target audience members, such as spectators or fans, to be replaced in a dynamic scene. Therefore, the present invention expands upon this and considers the character replacement function.

[0079] Specifically, the pre-reconstruction part of the target person is expanded to the reconstruction of the replaced person. The surround video reconstruction method is still used, but the original video input is expanded to support single image input for the new person, and then surround video generation is achieved through the video generation (SVD) method; the driving rendering stage is expanded to the video person's posture extraction and replacement rendering module, and the segmentation and splicing of the stadium background are added to the previous posture estimation idea, thereby realizing character reconstruction and replacement.

[0080] See also Figure 2 , is a flow chart of another method for constructing a character model according to an embodiment of the present invention. The embodiment of the present invention provides another method for constructing a character model, the method comprising steps S21 to S23:

[0081] S21, tracking and segmenting the second target person in the original video clip to generate background video and human body posture parameters of the second target person;

[0082] S22, generating an omnidirectional surround video and human body posture parameters of the third target person according to the target image data containing the third target person to be replaced;

[0083] S23 , reconstructing a model of the third target person according to the body posture parameters of the second target person, the omnidirectional surround video and the body posture parameters of the third target person, and fusing the model with the background video to generate a replaced target video clip.

[0084] In an embodiment of the present invention, a character replacement framework for original video clips is proposed. For human-centered motion videos, the character and background in the video are first separated, and then the character is reconstructed in three dimensions using a single reference image. The reconstructed human body model is driven according to the character's posture in the video and then merged with the background to achieve character editing.

[0085] The framework of the embodiment of the present invention includes a video posture extraction module, a surround video generation module, and a reconstruction and driving module. Specifically, given a selected video clip, the characters therein are segmented and tracked, and the SMPL posture estimation module optimized by 2D posture is adopted to support more accurate posture driving. At the same time, the video frames are semantically understood through BLIP (Bootstrapped Language-Image Pre-training, multimodal pre-training framework) to guide the image migration of the reference character pictures to generate a character image that meets the style of the video scene. Then, the viewpoint consistency of the human body trajectory video is generated by fine-tuning the latent diffusion model. Finally, the generated video is used to realize implicit neural reconstruction, and the motion posture is used to drive the character to realize the editing of the reference character in the selected video.

[0086] It should be noted that BLIP is a multimodal pre-training framework that aims to achieve efficient alignment and generation of images and text by jointly learning visual and language representations.

[0087] As a preferred embodiment, step S21, that is, tracking and segmenting the second target person in the original video clip to generate background video and human body posture parameters of the second target person, includes steps S211 to S213:

[0088] S211, using a preset target tracking and segmentation algorithm to track and segment the second target person in the original video clip, and output a person segmentation mask;

[0089] S212, segmenting the original video clip into a character video and a background video according to the character segmentation mask;

[0090] S213: Perform human body posture estimation on the person video to obtain human body posture parameters of the second target person.

[0091] In an embodiment of the present invention, the video posture extraction module in the first part realizes the tracking and posture extraction of the characters, and the input is the replaced video. First, through a preset target tracking and segmentation algorithm, the Track-Anything method can be selected to realize the tracking and segmentation of the characters in the original video clip, output the character segmentation mask, combine the original video to obtain the segmented character video and the separated background video, and use image restoration (inpainting) technology to fill the background vacancy. Then, the SMPL parameter extraction is performed on the tracked character information using ROMP technology (SMPL estimator based on 2D key points), and the most accurate character posture parameters are estimated by combining the two-dimensional key point projection fitting of the character. The SMPL parameters are adjusted by reverse optimization to reduce the posture error caused by occlusion. The output of this module is the optimized SMPL character posture parameters and the separated background video, which are input into the reconstruction and driving module in the third part.

[0092] It should be noted that Track-Anything is a general video target tracking and segmentation method that can semi-automatically track and segment any target in the video through simple user interaction (such as clicking or selecting a box) without the need for training.

[0093] The video posture extraction module is also used to understand the content of the original video clip, extract the content of the image / video through the BLIP module, and obtain the corresponding text description T0, such as "a football player running in a red jersey", etc., to provide style guidance for the character's clothing change in the subsequent second part.

[0094] As a preferred embodiment, step S22, i.e., generating an omnidirectional surround video and human body posture parameters of the third target person based on the target image data containing the third target person to be replaced, includes steps S221 to S225:

[0095] S221, extracting content from the original video clip to generate text information for describing the details of the character's clothing;

[0096] S222, changing the clothes of the third target person in the target image data according to the text information, and generating target image data after the clothes are changed;

[0097] S223, performing facial enhancement processing on the third target person in the target image data after the change of clothing to generate enhanced target image data;

[0098] S224, processing the enhanced target image data using a preset multi-view diffusion model to generate an omnidirectional surround video;

[0099] S225 . Perform human posture estimation on the omnidirectional surround video to obtain human posture parameters of the third target person.

[0100] In this embodiment of the present invention, the surround video generation module in the second part generates a three-dimensional orbital (surround) video of a character. The input is a single photo (target image data) of the new character to be replaced in the original video clip, as well as a text description T0 obtained through video understanding in the previous stage. First, the text description T0 is transformed using a large model, replacing it with precise text information T1 describing the character's clothing, such as "red jersey, white shorts, black sneakers." Based on this text information, the appearance of the clothing image is transferred to achieve the character's outfit design. A controlled diffusion model (such as Stable Diffusion) is used to transform the target image data to ensure that the scene style is consistent with the original video clip.

[0101] GFPGAN (Generative Facial Prior GAN) is used to repair the blurred facial areas of the target image data after the costume change. Specifically, this costume change result is input into the face enhancement operation module of GFPGAN to avoid the problem caused by insufficient facial clarity.

[0102] The enhanced target image data is used to generate orbital video. Using a VAE / CLIP / Diffusion network, an existing multi-view diffusion (SV3D) model is fine-tuned on a specific person dataset. This network is then trained using a VAE (Variational Autoencoder) and CLIP (Contrastive Language–Image Pretraining) text guidance to generate a trained network. This network is then used to generate a 360-degree surround video of the person. SMPL pose estimation is then performed on this generated surround video to obtain the person's pose parameters. This module outputs a 360-degree surround video of the third target person and the corresponding SMPL pose parameters, which are then fed into the reconstruction and driving modules in the third component.

[0103] As a preferred embodiment, step S23, i.e., reconstructing the model of the third target person according to the human body posture parameters of the second target person, the omnidirectional surround video of the third target person, and the human body posture parameters, and fusing it with the background video to generate a replaced target video clip, includes steps S231 to S233:

[0104] S231, constructing a static three-dimensional model of the third target person based on the omnidirectional surround video and human body posture parameters of the third target person;

[0105] S232: performing posture driving on the static three-dimensional model of the third target person according to the body posture parameters of the second target person, and rendering the driven static three-dimensional model to obtain a motion video of the third target person;

[0106] S233: Fusing the character motion video and the background video to generate a replaced target video clip.

[0107] In an embodiment of the present invention, the reconstruction and driving module of the third part uses the separated background, SMPL posture, and surround video of the replaced character input by the first two parts to realize the reconstruction and replacement of the new character, completing the overall interactive and intelligent viewing.

[0108] First, the model of the new character being replaced is reconstructed using the omnidirectional surround video of the third target character input from the second part and the corresponding SMPL human pose parameters. The surround video is input into the NeRF framework to learn the implicit 3D representation (geometry + texture) of the reference character. The SMPL pose of the surround video is bound to the character image, and the character image is mapped from the observation space of the input video to a three-dimensional canonical space to achieve a complete representation of the character information. The SMPL pose transmitted from the second part is used for character fitting, and the character geometry and texture information are learned using the neural radiation field training method. The output of this step is the character image in a standard canonical space (for example, a "large" character image). The blank space jump method is used here to achieve faster character modeling.

[0109] Next, based on the SMPL human pose parameters of the second target person transmitted from the first part, linear blend skinning (LBS) is used to drive the character's pose, mapping the character image in the standard specification space to the pose space corresponding to the new pose. This is then rendered from the camera's perspective to produce a fully driven motion video of the character in the new pose. Finally, the motion video of the character in the new pose is output and combined with the background video from the first part to produce the final, replaced target video clip.

[0110] The technical approach of the present invention employs semantically driven style transfer, extracting text through BLIP to achieve textual descriptions of clothing, ensuring that the replaced character matches the style of the scene. A fine-tuned diffusion model is used to generate multi-view videos, addressing the perspective loss problem associated with single-image reconstruction. By separating standardized spatial reconstruction from pose-driven processing, the high computational cost of direct dynamic reconstruction is effectively avoided. This invention supports intelligent interactive character replacement, enabling high-precision character replacement and editing in dynamic videos, significantly improving reconstruction accuracy and visual quality, and enhancing the audience's interactive experience.

[0111] The embodiments of the present invention have multiple application scenarios:

[0112] Sports Analysis and Entertainment Applications: This solution efficiently extracts player poses from game videos and achieves high-fidelity 3D reconstruction, providing technical support for sports analysis, tactical research, and athlete training. Furthermore, intelligent character replacement capabilities can be used in the entertainment sector, for example, by integrating celebrities or virtual characters into game scenes to enhance audience interaction.

[0113] Virtual Reality (VR) and Augmented Reality (AR): This solution's high-quality 3D reconstruction and multi-view rendering capabilities can be widely used in VR / AR content production, such as live broadcasts of virtual sports events and immersive viewing experiences, meeting users' demand for high-quality virtual content.

[0114] Film, television, and advertising production: This solution's intelligent character replacement function provides efficient tools for film, television, and advertising production, enabling seamless integration of specific characters into video scenes, reducing post-production costs and improving creative efficiency.

[0115] Education and Training: This solution can be used for sports teaching and training. Through 3D reconstruction and posture-driven technology, it provides students or athletes with intuitive movement analysis and guidance, thereby improving teaching effectiveness.

[0116] Game and virtual character development: The high-fidelity reconstruction and posture-driven capabilities of this solution can be applied to game development and virtual character creation, providing game characters with more realistic movement performance and interactive experience.

[0117] See also Figure 3 , is a structural diagram of a device for constructing a character model provided by an embodiment of the present invention. The embodiment of the present invention provides a device 10 for constructing a character model, comprising:

[0118] A static model building module 11 is configured to build a static three-dimensional model of a first target person based on multi-source image data of the first target person;

[0119] A human posture estimation module 12 is configured to perform human posture estimation on a target dynamic video to obtain human posture parameters of the first target person; wherein the target dynamic video is a video containing the motion process of the first target person;

[0120] A human posture driving module 13 is configured to perform posture driving on the static three-dimensional model of the first target person according to the human posture parameters of the first target person;

[0121] The 3D model sequence generation module 14 is configured to render the driven static 3D model to a viewing angle consistent with the target dynamic video to obtain a 3D model sequence of the first target person.

[0122] As a preferred embodiment, the static model building module 11 is specifically used to:

[0123] Acquiring multi-source image data of the first target person;

[0124] Inputting the multi-source image data into a preset neural radiation field model for calculation to generate an omnidirectional surround video of the first target person; wherein the neural radiation field model introduces the SMPL human body model as a geometric prior to constrain the learning process;

[0125] A static three-dimensional model of the first target person is generated based on the omnidirectional surround video.

[0126] The technical means of the embodiments of the present invention are adopted, and a two-stage strategy of modeling first and then driving is adopted. The pre-reconstruction stage focuses on high-precision static modeling to avoid noise interference in dynamic scenes. The driving stage focuses on motion restoration to reduce computational complexity. The static high-quality character model is combined with the real dynamic scene, and the dynamic video information and the prior knowledge of the character three-dimensional model are fully utilized to achieve high-precision and high-fidelity character reconstruction effects. It not only effectively improves the reconstruction accuracy, but also has good versatility and scalability, and is suitable for the needs of character modeling and virtual reproduction in a variety of complex dynamic scenes.

[0127] See also Figure 4 , is a schematic diagram of the structure of another device for constructing a character model according to an embodiment of the present invention. The embodiment of the present invention further provides another device 20 for constructing a character model, the device 20 comprising:

[0128] A video posture extraction module 21 is used to track and segment the second target person in the original video clip and generate background video and human posture parameters of the second target person;

[0129] A surround video generation module 22 is configured to generate an omnidirectional surround video and human body posture parameters of the third target person based on target image data containing the third target person to be replaced;

[0130] The reconstruction and driving module 23 is used to reconstruct the model of the third target person according to the human posture parameters of the second target person, the omnidirectional surround video and the human posture parameters of the third target person, and fuse it with the background video to generate a replaced target video clip.

[0131] As a preferred embodiment, the video posture extraction module 21 is specifically used to:

[0132] Using a preset target tracking and segmentation algorithm, tracking and segmenting the second target person in the original video clip, and outputting a person segmentation mask;

[0133] Segmenting the original video clip into a character video and a background video according to the character segmentation mask;

[0134] Performing human posture estimation on the person video to obtain human posture parameters of the second target person.

[0135] As a preferred embodiment, the surround video generation module 22 is specifically configured to:

[0136] Extracting content from the original video clip to generate text information describing the details of the character's clothing;

[0137] changing the clothes of the third target person in the target image data according to the text information to generate target image data after the clothes are changed;

[0138] performing facial enhancement processing on the third target person in the target image data after the change of clothing to generate enhanced target image data;

[0139] Processing the enhanced target image data using a preset multi-view diffusion model to generate an omnidirectional surround video;

[0140] Human body posture estimation is performed on the omnidirectional surround video to obtain human body posture parameters of the third target person.

[0141] As a preferred embodiment, the reconstruction and driving module 23 is specifically configured to:

[0142] Constructing a static three-dimensional model of the third target person based on the omnidirectional surround video and human body posture parameters of the third target person;

[0143] Performing posture driving on the static three-dimensional model of the third target person according to the human body posture parameters of the second target person, and rendering the driven static three-dimensional model to obtain a motion video of the third target person;

[0144] The character motion video and the background video are fused to generate a replaced target video clip.

[0145] The technical means of the embodiments of the present invention are adopted to support intelligent interactive replacement of characters, which can achieve high-precision character replacement and editing in dynamic videos, significantly improve reconstruction accuracy and visual effects, and enhance the audience's interactive experience.

[0146] It should be noted that the device for constructing a character model provided in an embodiment of the present invention is used to execute all the process steps of the method for constructing a character model in the above embodiment. The working principles and beneficial effects of the two correspond one to one, and thus will not be described in detail.

[0147] See also Figure 5 , is a structural diagram of a character model construction device provided by an embodiment of the present invention. The embodiment of the present invention also provides a character model construction device 30, including a processor 31, a memory 32, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the character model construction method described in any of the above embodiments.

[0148] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for constructing a character model as described in any of the above embodiments.

[0149] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the method for constructing a character model as described in any one of the above embodiments is implemented.

[0150] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0151] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for constructing a character model, characterized in that: include: constructing a static three-dimensional model of a first target person based on the multi-source image data of the first target person; Performing human posture estimation on the target dynamic video to obtain human posture parameters of the first target person; wherein the target dynamic video is a video containing the movement process of the first target person; Performing posture driving on the static three-dimensional model of the first target person according to the human body posture parameters of the first target person; The driven static three-dimensional model is rendered to a viewing angle consistent with the target dynamic video to obtain a three-dimensional model sequence of the first target person.

2. The method for constructing a character model according to claim 1, wherein: The step of constructing a static three-dimensional model of the first target person based on the multi-source image data of the target person includes: Acquiring multi-source image data of the first target person; Inputting the multi-source image data into a preset neural radiation field model for calculation to generate an omnidirectional surround video of the first target person; wherein the neural radiation field model introduces the SMPL human body model as a geometric prior to constrain the learning process; A static three-dimensional model of the first target person is generated based on the omnidirectional surround video.

3. The method for constructing a character model according to claim 1 or 2, wherein: The method further comprises: Tracking and segmenting a second target person in the original video clip to generate background video and human body posture parameters of the second target person; generating an omnidirectional surround video and human body posture parameters of the third target person according to target image data containing the third target person to be replaced; The model of the third target person is reconstructed according to the body posture parameters of the second target person, the omnidirectional surround video and the body posture parameters of the third target person, and is fused with the background video to generate a replaced target video clip.

4. The method for constructing a character model according to claim 3, wherein: Tracking and segmenting the second target person in the original video clip to generate background video and human body posture parameters of the second target person includes: Using a preset target tracking and segmentation algorithm, tracking and segmenting the second target person in the original video clip, and outputting a person segmentation mask; Segmenting the original video clip into a character video and a background video according to the character segmentation mask; Performing human posture estimation on the person video to obtain human posture parameters of the second target person.

5. The method for constructing a character model according to claim 3, wherein: The step of generating an omnidirectional surround video and human body posture parameters of the third target person based on the target image data containing the third target person to be replaced includes: Extracting content from the original video clip to generate text information describing the details of the character's clothing; changing the clothes of the third target person in the target image data according to the text information to generate target image data after the clothes are changed; performing facial enhancement processing on the third target person in the target image data after the change of clothing to generate enhanced target image data; Processing the enhanced target image data using a preset multi-view diffusion model to generate an omnidirectional surround video; Human body posture estimation is performed on the omnidirectional surround video to obtain human body posture parameters of the third target person.

6. The method for constructing a character model according to claim 3, wherein: The step of reconstructing a model of the third target person based on the body posture parameters of the second target person, the omnidirectional surround video of the third target person, and the body posture parameters, and fusing the model with the background video to generate a replaced target video segment includes: Constructing a static three-dimensional model of the third target person based on the omnidirectional surround video and human body posture parameters of the third target person; Performing posture driving on the static three-dimensional model of the third target person according to the human body posture parameters of the second target person, and rendering the driven static three-dimensional model to obtain a motion video of the third target person; The character motion video and the background video are fused to generate a replaced target video clip.

7. A device for constructing a character model, characterized in that: include: A static model building module, configured to build a static three-dimensional model of a first target person based on multi-source image data of the first target person; A human posture estimation module is used to perform human posture estimation on a target dynamic video to obtain human posture parameters of the first target person; wherein the target dynamic video is a video containing the motion process of the first target person; a human posture driving module, configured to perform posture driving on the static three-dimensional model of the first target person according to the human posture parameters of the first target person; The 3D model sequence generation module is used to render the driven static 3D model to a viewing angle consistent with the target dynamic video to obtain a 3D model sequence of the first target person.

8. A device for constructing a character model, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for constructing a character model according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for constructing a character model according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the method for constructing a character model according to any one of claims 1 to 6 is implemented.

Images