Virtual human video generation method and device, electronic equipment and storage medium

CN119364141APending Publication Date: 2025-01-24IFLYTEK CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411462352.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-18
Publication Date
2025-01-24

AI Technical Summary

Technical Problem

The existing 2D virtual human video generation technology cannot fully utilize the information in the reference picture, resulting in a large gap between the generated virtual humans and the real humans, and the generation effect is not good.

Method used

By obtaining the video to be processed and driving voice containing the face area, extract the face image and voice features in the video frame, cover the lip area of ​​the face image, and then input the face image, masked image, and voice features into the target image to generate The model uses the self-attention mechanism and the cross-attention mechanism to fuse image features and speech features to generate face repair images.

Benefits of technology

The fidelity of the virtual human video is improved, so that the virtual human's mouth shape is consistent with the driving voice and the posture is consistent, which enhances the user's viewing experience and reduces the difference between the virtual human and the real person.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119364141A_ABST
    Figure CN119364141A_ABST
Patent Text Reader

Abstract

The invention provides a virtual human video generation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. In the method, a face image, a covering image and a voice feature in each video frame are input into a target image generation model together for processing; compared with the video to be processed, only the mouth shape of the virtual human changes and is consistent with the driving voice, and the posture is still kept consistent, so that the virtual human image is more vivid, and the watching experience of the user is improved. Moreover, according to the method, the face image and the covering image are respectively coded, so that the information of the face image can be fully applied, and image features and voice features are fused by combining a self-attention mechanism and a cross attention mechanism, so that the detail retention degree of the face image is higher, the difference between a virtual person and a real person is reduced, and the user experience is improved. And the vivid effect of the virtual human video is improved.
Need to check novelty before this filing date? Find Prior Art

Citation Information

Patent Citations

  • Intelligent cloth paving machine control method and device and electronic equipment

    CN115795278A

  • Digital human image generation method and device based on voice driving and storage medium

    CN116758189A

  • Video generation method and device, equipment, storage medium and product

    CN116994307A

  • Motion image and video generation method and device and storage medium

    CN117593473A

  • Automatic alignment method and system for digital population types

    CN118612490A