Virtual character replacement method based on 3D key point information

Abstract

The invention discloses a virtual character replacement method based on 3D key point information. The method comprises the following steps: S1, preprocessing video data; s2, tracking a target object in the video to obtain detection frame information of the person; s3, 3D key point detection is carried out based on the detected target person, and smoothing processing is carried out; s4, generating a virtual role by using a CG model; s5, segmenting a target person in the video, removing the target person from the background, and filling the blank area; s6, pasting the generated virtual character CG picture into the video subjected to target elimination processing by the system, and performing illumination and shadow processing on the video; and S7, outputting the replaced image, and storing the image. By preprocessing the video data, detecting the target person, obtaining 3D key point detection, generating the virtual character by using the CG model, pasting the CG picture of the virtual character into the video subjected to target elimination processing and outputting, manual operation is reduced, and the efficiency of video replacement is greatly improved.

Description

Technical Field

[0001] The invention relates to the technical field of animation capture, and in particular to a virtual character replacement method based on 3D key point information. Background Art

[0002] In the production of contemporary film, television, animation and games, computer-generated images (CGI) are widely used to replace physical photography or create virtual scenes. CGI simulates real-world objects and scenes through 3D modeling and rendering, and plays an important role in the field of film, television and animation production. However, the traditional CGI production process is very complicated and involves multiple steps, such as character capture, scene rendering, character animation, etc. Each link takes a lot of time and resources, especially when the production involves replacing actors, objects or scenes, manual operation is extremely inefficient and costly. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a virtual character replacement method based on 3D key point information.

[0004] The object of the present invention is achieved by the following technical solution: A virtual character replacement method based on 3D key point information comprises the following steps: S1: Input video data and detection frame, and preprocess the video data; S2: Track the target object in the video according to the person detection frame in the specified frame, and obtain the detection frame information of the person in each frame; S3: 3D key point detection is performed based on the detected target person, and the key point information is smoothed; S4: Generate a virtual character using a CG model based on the 3D key point information of the target person; S5: Segment the target person in the video according to the tracked detection frame, remove the target person from the background, and fill the blank area after the person is removed; S6: Based on the key point detection results, the system pastes the generated virtual character CG image into the video after the target elimination process, and performs lighting and shadow processing on it; S7: Output the replaced image and store it.

[0005] Preferably, in step S1, the preprocessing includes denoising, super-resolution enhancement, blur restoration, color balancing and edge enhancement.

[0006] Preferably, in step S2, for a single-person video, a single target tracking algorithm is used to track the target object in the video; for a multi-person video, a multi-target tracking algorithm is used to distinguish different people for tracking; for a video with scene segmentation or shot changes, a target re-identification algorithm is used to relocate the detection target.

[0007] Preferably, in step S3, the key points include body key points, face key points and hand key points.

[0008] Preferably, in step S3, the key points are smoothed by a filtering algorithm and an interpolation algorithm.

[0009] Preferably, in step S5, the target person is segmented by an image segmentation algorithm, and the background is filled by a background filling network.

[0010] The present invention has the following advantages: the present invention pre-processes the video data, detects the target person, obtains 3D key point detection, uses the CG model to generate a virtual character, pastes the virtual character CG picture into the video after the target elimination processing and outputs it, thereby reducing manual operations and greatly improving the efficiency of video replacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A structural diagram of the virtual character replacement method process; Figure 2 This is a schematic diagram of the target tracking process; Figure 3 A structural diagram of the key points. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0013] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0015] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0016] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use, or the positions or positional relationships commonly understood by those skilled in the art, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0017] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0018] In this embodiment, if Figure 1 As shown, a virtual character replacement method based on 3D key point information includes the following steps: S1: Input video data and detection frame, and pre-process the video data; specifically, the video data is in common formats such as MP4 and AVI, and the detection frame is used to specify the target object to be replaced. The main function of pre-processing the video data is to improve the visual quality of the video and provide high-quality input for subsequent processing.

[0019] S2: Track the target object in the video according to the person detection frame in the specified frame, and obtain the detection frame information of the person in each frame; S3: 3D key point detection is performed based on the detected target person, and the key point information is smoothed; S4: Generate a virtual character using a CG model according to the 3D key point information of the target character; specifically, the CG model is a preset character model, and the generated CG image has high resolution and includes a transparent channel to facilitate subsequent replacement.

[0020] S5: Segment the target person in the video according to the tracked detection frame, remove the target person from the background, and fill the blank area after the person is removed; S6: Based on the key point detection result, the system pastes the generated virtual character CG image into the video after the target elimination processing, and performs lighting and shadow processing on it; specifically, the generated virtual character CG image is pasted into the video after the target elimination processing, and the position and posture of the CG character are adjusted according to the key points to make it consistent with the target motion trajectory in the original video, and the lighting and shadow of the replaced CG character are adjusted, that is, the light source conditions in the video are simulated to make the light and shadow effects of the CG character consistent with the background, and the color and tone of the character are automatically adjusted to ensure the color matching degree between the character and the surrounding environment.

[0021] S7: Output the replaced image and store it. By preprocessing the video data, detecting the target person, obtaining 3D key point detection, and using the CG model to generate a virtual character, the virtual character CG image is pasted into the video after the target elimination process and output, thereby reducing manual operations and greatly improving the efficiency of video replacement.

[0022] Furthermore, in step S1, the preprocessing includes denoising, super-resolution enhancement, blur repair, color balancing and edge enhancement. Specifically, denoising is used to eliminate video noise to improve the quality of video images; super-resolution enhancement is used to perform super-resolution processing on the video to improve detail clarity; blur repair is used to repair blurred areas in the video caused by motion or inaccurate focus; color balancing is used to adjust the contrast and brightness of the video color and enhance color saturation; edge enhancement is used to enhance the edges of the target object in the video to improve the detection accuracy of subsequent processing. In this embodiment, the preprocessing process is implemented using existing methods, which are not improved here and will not be described in detail.

[0023] Further, if Figure 2 As shown, in step S2, for a single-person video, a single target tracking algorithm is used to track the target object in the video; for a multi-person video, a multi-target tracking algorithm is used to distinguish different people for tracking; for a video with scene segmentation or lens changes, a target re-identification algorithm is used to relocate the detection target, so that the target ID is maintained consistent throughout the tracking process.

[0024] In this embodiment, if Figure 3As shown, in step S3, the key points include body key points, face key points and hand key points. Specifically, the key points are used to capture the dynamic posture and motion trajectory of the target, wherein the body key points include the position key points of the shoulders, elbows, knees, ankles and feet, the face key points include the positions of the eyes, nose and mouth, and the hand key points are the joint positions of the hands, so as to ensure the accuracy of the hand movements.

[0025] Furthermore, in step S3, the key points are smoothed by filtering and interpolation algorithms, which ensures the continuity of the key points in the video and the naturalness of the movement, and avoids jitter or incoherence.

[0026] Furthermore, in step S5, the target person is segmented by an image segmentation algorithm, and the background is filled by a background filling network, thereby ensuring the consistency of the video background.

[0027] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual character replacement method based on 3D key point information, characterized in that: The following steps are involved: S1: Input video data and detection frame, and preprocess the video data; S2: Track the target object in the video according to the person detection frame in the specified frame, and obtain the detection frame information of the person in each frame; S3: 3D key point detection is performed based on the detected target person, and the key point information is smoothed; S4: Generate a virtual character using a CG model based on the 3D key point information of the target person; S5: Segment the target person in the video according to the tracked detection frame, remove the target person from the background, and fill the blank area after the person is removed; S6: Based on the key point detection results, the system pastes the generated virtual character CG image into the video after the target elimination process, and performs lighting and shadow processing on it; S7: Output the replaced image and store it.

2. The virtual character replacement method based on 3D key point information according to claim 1, characterized in that: In step S1, the preprocessing includes denoising, super-resolution enhancement, blur restoration, color balancing and edge enhancement.

3. The virtual character replacement method based on 3D key point information according to claim 2, characterized in that: In step S2, for a single-person video, a single target tracking algorithm is used to track the target object in the video; for a multi-person video, a multi-target tracking algorithm is used to distinguish different people for tracking; for a video with scene segmentation or lens changes, a target re-identification algorithm is used to relocate the detection target.

4. The virtual character replacement method based on 3D key point information according to claim 3, characterized in that: In step S3, the key points include body key points, face key points and hand key points.

5. The virtual character replacement method based on 3D key point information according to claim 4, characterized in that: In step S3, the key points are smoothed by using a filtering algorithm and an interpolation algorithm.

6. The virtual character replacement method based on 3D key point information according to claim 5, characterized in that: In step S5, the target person is segmented by an image segmentation algorithm, and the background is filled by a background filling network.

Images