Video generation method, device, computer equipment, storage medium and product

By generating background videos and combining digital human videos, synchronous explanations between digital human and background videos are realized, and the problem of monotonous digital human video generation methods in the existing technology is solved, and it is adapted to diversified application scenarios.

CN114363529BActive Publication Date: 2024-11-26SHANGHAI PUDONG DEVELOPMENT BANK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111491351.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-08
Publication Date
2024-11-26
Estimated Expiration
2041-12-08

AI Technical Summary

Technical Problem

In the prior art, the digital human video generation method is relatively monotonous, unable to adapt to diverse application scenarios, and cannot realize the synchronous explanation of background videos and digital human videos.

Method used

By obtaining materials to generate background videos, extracting video audio files, generating digital human videos based on video audio files, and synthesizing digital human videos with background videos, so that digital humans can synchronously explain each video picture in the background video through dynamic images.

Benefits of technology

The synchronization of background video and digital human video is realized. The video picture of background video is relatively independent from digital human video, which is convenient for post-editing and adapt to diverse application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114363529B_ABST
    Figure CN114363529B_ABST
Patent Text Reader

Abstract

This application relates to a video generation method, apparatus, computer device, storage medium, and computer program product. The method includes: acquiring materials for generating a video; generating a background video based on the materials; the background video including video frames and audio aligned with each video frame; extracting the audio aligned with each video frame from the background video to obtain a video-audio file; generating a digital human video based on the video-audio file; the digital human video being a video containing a dynamic image of a digital human; and combining the digital human video and the background video to form a content video, wherein the content video is a video in which the digital human provides synchronized narration of each video frame in the background video through its dynamic image. This method enables synchronization of the background video, audio content, digital human video, and digital human lip movements, and facilitates subsequent editing of the background video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium and product for generating digital human content video. Background Art

[0002] With the development of virtual reality technology, the application of digital humans is becoming more and more extensive. At present, it has been rapidly applied to digital customer service, anchors, games, live broadcasts, advertising, education and other fields. In order to apply digital human technology to various fields and scenarios, it is necessary to generate content videos based on digital humans based on different application scenarios.

[0003] In the prior art, a static background and a dynamic digital human video are generally synthesized into a content video, and the content format is relatively monotonous and cannot cope with an increasing number of application scenarios. Summary of the invention

[0004] Based on this, it is necessary to provide a digital human content video method, device, computer equipment, computer readable storage medium and computer product to address the above technical problems.

[0005] In a first aspect, the present application provides a video generation method. The method comprises:

[0006] Obtaining material for generating a video, and generating a background video based on the material; the background video includes video images and audio aligned with each video image;

[0007] Extracting the audio in the background video that is aligned with each video screen to obtain a video and audio file, and generating a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human;

[0008] The digital human video and the background video are combined into a content video, which is a video in which the digital human explains each video frame in the background video synchronously through a dynamic image.

[0009] In one embodiment, the material includes a plurality of image materials and texts corresponding to each image material, and generating a background video based on the material includes:

[0010] Based on the text corresponding to each image material, obtain the audio corresponding to each image material;

[0011] Generate a corresponding video screen based on each image material, and align the video screen with the audio corresponding to the image material;

[0012] A background video is generated based on a plurality of video frames and audio aligned with the video frames.

[0013] In one embodiment, obtaining the audio corresponding to each image material based on the text corresponding to each image material includes:

[0014] Typesetting the text corresponding to each image material, wherein a text paragraph obtained by typeset corresponds to one image material;

[0015] Based on each text paragraph, audio corresponding to each image material is generated.

[0016] In one embodiment, the material is a presentation, the image material is a manuscript page in the presentation or an animation page contained in the manuscript page, and the text corresponding to the image material is a note text corresponding to the manuscript page or the animation page; generating a corresponding video screen based on each image material, and aligning the video screen with the audio corresponding to the image material, includes:

[0017] Based on the document page in the presentation, a video screen corresponding to the document page is generated; for a document page containing an animation page, the document page corresponds to multiple video screens, and for a document page not containing an animation page, the document page corresponds to one video screen;

[0018] For each video screen, a text audio is obtained based on the note text corresponding to the manuscript page or animation page that generates the video screen, and the video screen is aligned with the text audio.

[0019] In one embodiment, obtaining a text audio based on the note text corresponding to the manuscript page or animation page that generates the video screen, and aligning the video screen with the text audio includes:

[0020] Acquire the text audio of the note text corresponding to the multiple page animations or document pages, and align the multiple page animations or document pages with each text audio respectively;

[0021] Get the text audio of the note text corresponding to the first document page in the presentation as the opening audio, and get the duration of the opening audio;

[0022] Set each text audio before the corresponding page animation or document page;

[0023] Based on the duration of the opening audio, multiple page animations or document pages and their corresponding text audio are delayed and played for the same duration.

[0024] In one embodiment, generating a digital human video based on the video and audio files includes:

[0025] Based on the video and audio files, the digital human audio is generated by the voice-driven algorithm, and the video screen containing the dynamic image of the digital human is generated by the digital human image generation program;

[0026] A digital human video is generated based on the digital human audio and the video image containing the dynamic image of the digital human.

[0027] In one embodiment, synthesizing a digital human video and a background video into a content video includes:

[0028] Aligning the digital human audio corresponding to the digital human video with the video and audio of the background video to obtain the digital human video and background video after audio alignment;

[0029] The digital human video after audio alignment is synthesized with the background video to obtain the content video.

[0030] In one embodiment, the digital human video after audio alignment is synthesized with the background video to obtain the content video, including:

[0031] According to the different forms of overlap between the digital human video and the background video in time sequence, the digital human video is cut or supplemented;

[0032] The cropped or supplemented digital human video is synthesized with the background video through the picture-in-picture technology to obtain the content video.

[0033] In a second aspect, the present application also provides a video generation device. The device comprises:

[0034] A background video generation module, used to obtain materials for generating a video, and generate a background video based on the materials; the background video includes video images and audio aligned with each video image;

[0035] The digital human video generation module is used to extract the audio in the background video that is aligned with each video screen, obtain the video and audio files, and generate the digital human video based on the video and audio files; the digital human video is a video containing the dynamic image of the digital human;

[0036] The content video synthesis module is used to synthesize the digital human video and the background video into a content video, which is a video in which the digital human uses a dynamic image to synchronously explain each video frame in the background video.

[0037] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the above embodiments are implemented.

[0038] In a fourth aspect, the present application further provides a computer device readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0039] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0040] The above-mentioned video generation method, device, computer equipment, storage medium and computer program product first obtain the material for generating the video, and generate the background video based on the material; the background video includes the video screen and the audio aligned with each video screen. Further, the audio aligned with each video screen in the background video is extracted to obtain the video and audio file, and the digital human video is generated accordingly based on the video and audio file; the digital human video is a video containing the dynamic image of the digital human. Further, the digital human video and the background video are synthesized into a content video, and the content video is a video in which the digital human uses the dynamic image to synchronously explain each video screen in the background video, which can achieve the synchronization of the background video, voice content and digital human video. Among them, the video screen of the background video is relatively independent of the video screen of the digital human video, which is convenient for the later editing of the background video. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a video generation method in one embodiment;

[0042] Figure 2 A schematic diagram of an interface for typeset text in an embodiment;

[0043] Figure 3 A schematic diagram of a method for aligning a note text with a document page including an animation page in one embodiment;

[0044] Figure 4 A schematic diagram of a digital human video generated in one embodiment;

[0045] Figure 5 is a schematic diagram of a synthesized content video in one embodiment;

[0046] Figure 6 is a structural block diagram of a video generating device in one embodiment;

[0047] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The video generation method provided in the embodiment of the present application can be applied to a single-side implementation of a server or a terminal, or can be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0050] In one embodiment, Figure 1 As shown, a video generation method is provided, which is described by taking the method applied to a single-side implementation of a server as an example, and includes the following steps:

[0051] Step 202, obtaining material for generating a video, and generating a background video based on the material; the background video includes video frames and audio aligned with each video frame.

[0052] In this embodiment, the materials used to generate the video include a plurality of image materials and texts corresponding to each image material.

[0053] In this embodiment, the server can obtain the material for generating the video through a presentation tool (PPT, PowerPoint). It is understandable that in other embodiments, the server can also obtain the material for generating the video through other presentation tools (such as WPS presentation in Kingsoft Office software suite, Google Docs, etc.) or video production tools.

[0054] In this embodiment, the server can convert the presentation into a video output through the PPT / PPTX format conversion function in PPT to obtain a background video.

[0055] Step 204, extract the audio in the background video that is aligned with each video screen to obtain a video and audio file, and generate a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human.

[0056] In this embodiment, the video and audio of the background video are generated by aligning the video screen with the audio corresponding to the image material. After the server extracts the video and audio, it resamples it and inputs the sampled video and audio into the digital human voice driver and the digital human image generation program, which can generate the digital human audio and the video screen of the digital human's dynamic image respectively.

[0057] Step 206: The digital human video and the background video are combined into a content video, where the content video is a video in which the digital human explains each video frame in the background video synchronously through a dynamic image.

[0058] In this embodiment, since the digital human video is generated based on the video and audio file, the audio in the digital human video and the background video can be aligned, and the digital human video with the aligned audio is synthesized with the background video to obtain the content video.

[0059] In this embodiment, the server may use the background video as the backbone and align the digital human video with the background video through an algorithm.

[0060] In another example, the server may also use the digital human video as the backbone and align the background video with the digital human video through an algorithm.

[0061] In the above-mentioned video generation method, firstly, the material for generating the video is obtained, and the background video is generated based on the material; the background video includes video screens and audio aligned with each video screen. Further, the audio aligned with each video screen in the background video is extracted to obtain a video and audio file, and a digital human video is generated based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human. Further, the digital human video and the background video are synthesized into a content video, and the content video is a video in which the digital human uses a dynamic image to synchronously explain each video screen in the background video, which can achieve synchronization of the background video, voice content and digital human video. Among them, the video screen of the background video is relatively independent of the video screen of the digital human video, which is convenient for the later editing of the background video.

[0062] In one embodiment, the material includes multiple image materials and text corresponding to each image material, and a background video is generated based on the material, including: obtaining the audio corresponding to each image material based on the text corresponding to each image material; generating a corresponding video screen based on each image material, and aligning the video screen with the audio corresponding to the image material; generating a background video based on multiple video screens and the audio aligned with the video screens.

[0063] In this embodiment, one image material may correspond to multiple texts.

[0064] In this embodiment, the server can complete the matching of text and image materials through the remarks column of the presentation or an additional editor. For example, the server can complete the matching of text and image materials through the remarks column of each manuscript page of the PPT.

[0065] In another example, when the presentation tool or video production tool used by the server does not have its own note column or text editor, the server can also complete the matching of text and image materials through other editors.

[0066] In this embodiment, the server can complete the alignment of the image material and the text through the Office Automation program of Microsoft Office software.

[0067] In another example, due to the versatility of Office automation, Office Automation programs are not limited to traditional Visual Basic macro language (VBA, Visual Basic for Applications) scripts. Other languages ​​can also operate Office automation functions through application programming interfaces (API, Application Programming Interface) to complete the alignment of image materials and text.

[0068] In this embodiment, the server generates audio corresponding to each text, and based on the correspondence between each image material and each text, the audio is used as the audio corresponding to each image material. Based on the alignment of the image material and the text, the video screen generated by the multiple image materials is also aligned with the audio corresponding to each image material.

[0069] In one embodiment, if Figure 2 As shown, based on the text corresponding to each image material, the audio corresponding to each image material is obtained, including: typeset the text corresponding to each image material, and a text paragraph obtained by typeset corresponds to one image material; based on each text paragraph, the audio corresponding to each image material is generated.

[0070] In this embodiment, one image material may correspond to multiple texts. The server typeset the multiple texts corresponding to one image material through an editor or a remarks column provided in the presentation to obtain text paragraphs corresponding to the image material.

[0071] In this embodiment, the server obtains the audio corresponding to each image material based on each text paragraph, that is, based on the order of the text after typeset. For example, when an image material corresponds to three texts (such as text 1, text 2, and text 3), the server generates the audio corresponding to text 1, text 2, and text 3 respectively. Further, when the order of the text after typeset is text 2, text 1, and text 3, the server arranges the above-generated audio in the same order based on the above order to obtain the audio of the text paragraph corresponding to the image material, that is, the audio corresponding to the image material.

[0072] In one of the embodiments, the material is a presentation, the image material is a document page in the presentation or an animation page contained in the document page, and the text corresponding to the image material is the note text corresponding to the document page or the animation page; a corresponding video screen is generated based on each image material, and the video screen is aligned with the audio corresponding to the image material, and also includes: based on the document page in the presentation, a video screen corresponding to the document page is generated; for a document page containing an animation page, the document page corresponds to multiple video screens, and for a document page not containing an animation page, the document page corresponds to one video screen; for each video screen, a text audio obtained based on the note text corresponding to the document page or animation page that generates the video screen is obtained, and the video screen is aligned with the text audio.

[0073] In this embodiment, for a document page that does not include an animation page, the document page corresponds to a video screen, and the video screen can correspond to multiple note texts. The server typesets multiple texts corresponding to an image material through an editor or a note column provided in the presentation to obtain a text paragraph corresponding to the image material. Further, the server obtains the audio corresponding to each image material based on each text paragraph, that is, based on the order of the typeset text.

[0074] In this embodiment, for a document page containing an animation page, the document page corresponds to multiple note texts. The server can typeset the multiple note texts corresponding to the document page through the editor or the note column provided by the presentation to obtain the text paragraphs corresponding to each animation page of the document page. For example, Figure 3 As shown, a document page including two animation pages, assuming that the document page corresponds to three remark texts, when the first remark text is the opening statement of the document page, the second remark text corresponds to the first animation page, and the third remark text corresponds to the second animation page, the first remark text is not aligned with any animation page of the document page, and is set at the initial position of the document page. Further, the server aligns the second remark text with the first animation page, and the third remark text with the second animation page.

[0075] In this embodiment, for a document page containing an animation page, when the animation page corresponding to the document page is n, and the document page has an opening statement, and the number of note texts exceeds n+1, the note texts exceeding n+1 can be merged into the n+1th note text. For example, for a document page containing two animation pages, assuming that the document page corresponds to four note texts, when the first note text is the opening statement of the document page, the second note text corresponds to the first animation page, and the third note text corresponds to the second animation page, the first note text is not aligned with any animation page of the document page, and is set at the initial position of the document page. Further, the server merges the fourth note text with the third note text to form a new third note text. Further, the server aligns the second note text with the first animation page, and aligns the new third note text with the second animation page.

[0076] In another example, for a document page containing an animation page, when the animation page corresponding to the document page is n, and the document page does not have an opening statement, and the number of note texts exceeds n, the note texts exceeding n can be merged into the nth note text. For example, for a document page containing three animation pages, assuming that the document page corresponds to four note texts, and the document page does not have a note text corresponding to the opening statement, the first note text corresponds to the first animation page, the second note text corresponds to the second animation page, and the third note text corresponds to the third animation page, the server merges the fourth note text with the third note text to form a new third note text. Further, the server aligns the first note text with the first animation page, the second note text with the second animation page, and the new third note text with the third animation page.

[0077] In one of the embodiments, text audio obtained based on the note text corresponding to the document page or animation page that generates the video screen is obtained, and the video screen is aligned with the text audio, including: obtaining text audio of the note text corresponding to multiple page animations or document pages, and aligning the multiple page animations or document pages with each text audio respectively; obtaining the text audio of the note text corresponding to the first document page in the presentation as the opening audio, and obtaining the duration of the opening audio; setting each text audio before the corresponding page animation or document page; based on the duration of the opening audio, delaying the playback of multiple page animations or document pages and each corresponding text audio by the same duration.

[0078] In this embodiment, the server typesets multiple texts corresponding to an image material through the editor or the notes column of the presentation to obtain the text paragraphs corresponding to the image material. Further, the server obtains the text audio of the notes text corresponding to multiple page animations or manuscript pages based on each text paragraph, that is, based on the text order after typeset.

[0079] In this embodiment, the server can complete the alignment of multiple page animations or document pages with the note text through the office automation (OfficeAutomation) program of Microsoft's office software (Office), thereby realizing the alignment of multiple page animations or document pages with each text audio.

[0080] In this embodiment, when a document page includes multiple animation pages, the playing of the multiple animation pages can be triggered by a click.

[0081] In one of the embodiments, a digital human video is generated based on a video and audio file, including: based on the video and audio file, generating digital human audio through a voice-driven algorithm, and generating a video screen containing a dynamic image of the digital human through a digital human image generation program; generating a digital human video based on the digital human audio and the video screen containing the dynamic image of the digital human.

[0082] In this embodiment, based on the video and audio files, the digital human audio is generated correspondingly through a voice-driven algorithm, which can ensure that the digital human video, the background video, the video and audio files, and the digital human's lip movements remain synchronized during video playback.

[0083] In this embodiment, based on different application scenarios, the server can generate digital human audio that meets the needs through a voice-driven algorithm. For example, according to different application scenarios and user needs, the digital human audio can use different voices (such as children's voices, adult male / female voices, elderly voices, etc.), can be based on different languages ​​(such as Chinese, English, Arabic, etc.), and can also use regional dialects (such as Tianjin dialect, Sichuan dialect, etc.).

[0084] In this embodiment, if Figure 4 As shown, based on different application scenarios, the server can generate different dynamic images of digital humans through the digital human image generation program. For example, according to different application scenarios and user needs, the dynamic image of the digital human can be a virtual human image, or an animated character, animal, plant, robot, etc.

[0085] In one of the embodiments, the digital human video and the background video are synthesized into the content video, including: aligning the digital human audio corresponding to the digital human video with the video audio of the background video to obtain the digital human video and the background video after the audio is aligned; synthesizing the digital human video and the background video after the audio is aligned to obtain the content video.

[0086] In this embodiment, the server can use the background video as the backbone, so that the video images in the background video are not missing. The server aligns the video and audio of the digital human video with the background video. It is understandable that based on business needs, the server can also use the digital human video as the backbone and then align the video and audio of the background video with the digital human video.

[0087] In this embodiment, the server may use a fixed parameter method (eg, assuming a fixed Δt′ here) to align the audio of the digital human with the video audio of the background video.

[0088] In another embodiment, the server may also use feature-based comparison or further utilize methods such as dynamic time warping (DTW) to align the audio of the digital human with the video audio of the background video.

[0089] In one embodiment, if Figure 5 As shown, the digital human video after audio alignment is synthesized with the background video to obtain the content video, including: cropping or supplementing the digital human video according to the different forms of temporal overlap between the digital human video and the background video; and synthesizing the cropped or supplemented digital human video with the background video through the picture-in-picture technology to obtain the content video.

[0090] In this embodiment, there may be a time error between the digital human video and the background video. Therefore, after the video and audio of the digital human video and the background video are aligned, the video lengths of the digital human video and the background video may not be aligned.

[0091] In this embodiment, when the server uses the background video as the main body, the server can cut or supplement the digital human video according to the different forms of overlap between the digital human video and the background video in time sequence.

[0092] In this embodiment, when the server uses the background video as the main body and needs to supplement the digital human video, a blank still picture or a digital human still picture can be supplemented at the end of the digital human video. For example, when the time length of the background video is 1 minute and the length of the digital human video is 59.8 seconds, after the server uses the background video as the main body and the video and audio of the digital human video and the background video are aligned, the service can supplement a 0.2 second blank still picture or a digital human still picture at the end of the digital human video based on the different forms of the digital human video and the background video overlapping in time sequence, so that the time length of the background video and the digital human video is consistent.

[0093] In this embodiment, when the server adds a still image of the digital human to the digital human video, the server can select a frame in the digital human video as the still image (the digital human is motionless at this time); when the server adds a blank still image to the digital human video, the blank still image is a blank page or a transparent blank page that does not display the digital human.

[0094] In this embodiment, when the server uses the background video as the main body and needs to crop the digital human video, after the digital human video and the audio are aligned, the part of the digital human video that is not aligned with the background video can be cropped based on the different forms of overlap between the digital human video and the background video in time sequence. For example, when the time length of the background video is 1 minute and 31.6 seconds, and the length of the digital human video is 1 minute and 32 seconds, that is, the length of the digital human video exceeds the background video by 0.4 seconds, when the server uses the background video as the main body, after the server aligns the video and audio of the digital human video with the background video, if the digital human video is not aligned with the background video only at the end of the video, the server will crop the last 0.4 seconds of the digital human video; if the beginning and the end of the digital human video are not aligned with the background video by 0.1 seconds and 0.3 seconds respectively, the server will crop the first 0.1 seconds and the last 0.3 seconds of the digital human video respectively, so that the time length of the background video and the digital human video are consistent.

[0095] In another embodiment, the server can also use the digital human video as the backbone, and cut or supplement the background video according to the different forms of overlap between the digital human video and the background video in time sequence. The method for the server to cut or supplement the background video is roughly the same as the above, and will not be repeated here.

[0096] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0097] Based on the same inventive concept, the embodiment of the present application also provides a video generation device for implementing the video generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more video generation device embodiments provided below can refer to the limitations of the video generation method above.

[0098] In one embodiment, Figure 6 As shown, a video generation device is provided, comprising: a background video generation module 100, a digital human video generation module 200 and a content video synthesis module 300, wherein:

[0099] The background video generation module 100 is used to obtain materials for generating a video and generate a background video based on the materials; the background video includes video frames and audio aligned with each video frame.

[0100] The digital human video generation module 200 is used to extract the audio in the background video that is aligned with each video screen, obtain a video and audio file, and generate a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of a digital human.

[0101] The content video synthesis module 300 is used to synthesize the digital human video and the background video into a content video, wherein the content video is a video in which the digital human uses a dynamic image to synchronously explain each video frame in the background video.

[0102] In one embodiment, the material includes multiple image materials and text corresponding to each image material, and the background video generation module 100 may include:

[0103] The material audio acquisition submodule is used to obtain the audio corresponding to each image material based on the text corresponding to each image material.

[0104] The audio and video alignment submodule is used to generate corresponding video images based on each image material, and align the video images with the audio corresponding to the image material.

[0105] The background video generation submodule is used to generate a background video based on a plurality of video frames and audio aligned with the video frames.

[0106] In one embodiment, the material audio acquisition submodule may include:

[0107] The text typesetting unit is used to typeset the text corresponding to each image material, and a text paragraph obtained by typeset corresponds to one image material.

[0108] The material audio generation unit is used to generate audio corresponding to each image material based on each text paragraph.

[0109] In one embodiment, the material is a presentation, the image material is a manuscript page in the presentation or an animation page contained in the manuscript page, and the text corresponding to the image material is a note text corresponding to the manuscript page or the animation page; the audio and video alignment submodule may include:

[0110] The audio-visual corresponding unit is used to generate a video screen corresponding to the document page based on the document page in the presentation; for a document page containing an animation page, the document page corresponds to multiple video screens, and for a document page not containing an animation page, the document page corresponds to one video screen.

[0111] The audio-video alignment unit is used to obtain, for each video screen, a text audio based on the note text corresponding to the manuscript page or animation page that generates the video screen, and align the video screen with the text audio.

[0112] In one embodiment, the audio and video alignment unit may include:

[0113] The audio-visual alignment subunit is used to obtain the text audio of the note text corresponding to multiple page animations or document pages, and align the multiple page animations or document pages with each text audio respectively.

[0114] The opening audio duration acquisition subunit is used to obtain the text audio of the note text corresponding to the first document page in the presentation as the opening audio, and obtain the duration of the opening audio.

[0115] The audio and video setting subunit is used to set the audio of each text before the corresponding page animation or manuscript page.

[0116] The audio and video delay adjustment subunit is used to delay the playback of multiple page animations or document pages and their corresponding text audio by the same length based on the length of the opening audio.

[0117] In one embodiment, the digital human video generation module 200 may include:

[0118] The digital human audio generation submodule is used to generate digital human audio based on the video and audio files through a voice-driven algorithm.

[0119] The digital human dynamic video generation submodule is used to generate a video screen containing a dynamic image of a digital human based on a video and audio file through a digital human image generation program.

[0120] The digital human video generation submodule is used to generate a digital human video based on the digital human audio and the video screen containing the dynamic image of the digital human.

[0121] In one embodiment, the content video synthesis module 300 may include:

[0122] The video alignment submodule is used to align the digital human audio corresponding to the digital human video with the video and audio of the background video to obtain the digital human video and background video after audio alignment.

[0123] The video synthesis submodule is used to synthesize the digital human video after audio alignment with the background video to obtain the content video.

[0124] In one embodiment, the video synthesis submodule may include:

[0125] The video cropping unit is used to crop or supplement the digital human video according to the different forms of overlap between the digital human video and the background video in time sequence.

[0126] The post-synthesis unit is used to synthesize the cropped or supplemented digital human video with the background video through the picture-in-picture technology to obtain the content video.

[0127] Each module in the above video generation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0128] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image materials, texts corresponding to each image material, audio corresponding to each image material and other data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video generation method is implemented.

[0129] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining material for generating a video, and generating a background video based on the material; the background video includes video frames and audio aligned with each video frame; extracting the audio aligned with each video frame in the background video to obtain a video and audio file, and generating a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human; the digital human video and the background video are synthesized into a content video, and the content video is a video in which the digital human synchronously explains each video frame in the background video through a dynamic image.

[0131] In one embodiment, the material includes a plurality of image materials and text corresponding to each image material.

[0132] In this embodiment, when the processor executes the computer program, it also realizes the generation of background video based on the material, which may include: based on the text corresponding to each image material, obtaining the audio corresponding to each image material; based on each image material, generating a corresponding video screen, and aligning the video screen with the audio corresponding to the image material; based on multiple video screens and the audio aligned with the video screens, generating a background video.

[0133] In one of the embodiments, when the processor executes the computer program, it also obtains the audio corresponding to each image material based on the text corresponding to each image material, which can include: typesetting the text corresponding to each image material, and a text paragraph obtained by typesetting corresponds to one image material; based on each text paragraph, generating the audio corresponding to each image material.

[0134] In one embodiment, the material is a presentation, the image material is a document page in the presentation or an animation page contained in the document page, and the text corresponding to the image material is the note text corresponding to the document page or the animation page.

[0135] In this embodiment, when the processor executes the computer program, it also realizes generating corresponding video pictures based on each image material, and aligning the video pictures with the audio corresponding to the image material, which may include: based on the document page in the presentation, generating the video picture corresponding to the document page; for the document page containing the animation page, the document page corresponds to multiple video pictures, and for the document page not containing the animation page, the document page corresponds to one video picture; for each video picture, obtaining the text audio obtained based on the note text corresponding to the document page or animation page that generates the video picture, and aligning the video picture with the text audio.

[0136] In one of the embodiments, when the processor executes the computer program, it also implements obtaining text audio based on the note text corresponding to the document page or animation page that generates the video screen, and aligning the video screen with the text audio, which may include: obtaining text audio of the note text corresponding to multiple page animations or document pages, and aligning the multiple page animations or document pages with each text audio respectively; obtaining the text audio of the note text corresponding to the first document page in the presentation as the opening audio, and obtaining the duration of the opening audio; setting each text audio before the corresponding page animation or document page; based on the duration of the opening audio, delaying the playback of multiple page animations or document pages and each corresponding text audio by the same duration.

[0137] In one of the embodiments, when the processor executes the computer program, it also realizes the generation of digital human video based on the video and audio files, which can include: based on the video and audio files, generating digital human audio through a voice-driven algorithm, and generating a video screen containing a dynamic image of the digital human through a digital human image generation program; generating digital human video based on the digital human audio and the video screen containing the dynamic image of the digital human.

[0138] In one of the embodiments, when the processor executes the computer program, it also realizes synthesizing the digital human video and the background video into the content video, which may include: aligning the digital human audio corresponding to the digital human video with the video audio of the background video to obtain the digital human video and the background video after the audio is aligned; synthesizing the digital human video and the background video after the audio is aligned to obtain the content video.

[0139] In one of the embodiments, when the processor executes the computer program, it also realizes synthesizing the digital human video after audio alignment with the background video to obtain the content video, which may include: cropping or supplementing the digital human video according to the different forms of temporal overlap between the digital human video and the background video; synthesizing the cropped or supplemented digital human video with the background video through picture-in-picture technology to obtain the content video.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining material for generating a video, and generating a background video based on the material; the background video includes video frames and audio aligned with each video frame; extracting the audio aligned with each video frame in the background video to obtain a video and audio file, and generating a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human; the digital human video and the background video are combined into a content video, and the content video is a video in which the digital human synchronously explains each video frame in the background video through a dynamic image.

[0141] In one embodiment, the material includes a plurality of image materials and text corresponding to each image material.

[0142] In this embodiment, when the computer program is executed by the processor, it also realizes the generation of background video based on the material, which may include: based on the text corresponding to each image material, obtaining the audio corresponding to each image material; based on each image material, generating a corresponding video screen, and aligning the video screen with the audio corresponding to the image material; based on multiple video screens and the audio aligned with the video screens, generating a background video.

[0143] In one of the embodiments, when the computer program is executed by the processor, it also realizes obtaining audio corresponding to each image material based on the text corresponding to each image material, which may include: typeset the text corresponding to each image material, and a text paragraph obtained by typeset corresponds to one image material; based on each text paragraph, generating audio corresponding to each image material.

[0144] In one embodiment, the material is a presentation, the image material is a document page in the presentation or an animation page contained in the document page, and the text corresponding to the image material is the note text corresponding to the document page or the animation page.

[0145] In this embodiment, when the computer program is executed by the processor, it also realizes generating corresponding video screens based on each image material, and aligning the video screens with the audio corresponding to the image material, which may include: based on the document page in the presentation, generating the video screen corresponding to the document page; for the document page containing the animation page, the document page corresponds to multiple video screens, and for the document page not containing the animation page, the document page corresponds to one video screen; for each video screen, obtaining the text audio obtained based on the note text corresponding to the document page or animation page that generates the video screen, and aligning the video screen with the text audio.

[0146] In one of the embodiments, when the computer program is executed by the processor, it also implements obtaining text audio based on the note text corresponding to the document page or animation page that generates the video screen, and aligning the video screen with the text audio, which may include: obtaining text audio of the note text corresponding to multiple page animations or document pages, and aligning the multiple page animations or document pages with each text audio respectively; obtaining the text audio of the note text corresponding to the first document page in the presentation as the opening audio, and obtaining the duration of the opening audio; setting each text audio before the corresponding page animation or document page; based on the duration of the opening audio, delaying the playback of multiple page animations or document pages and each corresponding text audio by the same duration.

[0147] In one embodiment, when the computer program is executed by the processor, it also realizes the generation of digital human video based on the video and audio files, which may include: based on the video and audio files, generating digital human audio through a voice-driven algorithm, and generating a video screen containing a dynamic image of the digital human through a digital human image generation program; generating digital human video based on the digital human audio and the video screen containing the dynamic image of the digital human.

[0148] In one of the embodiments, when the computer program is executed by the processor, it also realizes synthesizing the digital human video and the background video into the content video, which may include: aligning the digital human audio corresponding to the digital human video with the video audio of the background video to obtain the digital human video and the background video after audio alignment; synthesizing the digital human video and the background video after audio alignment to obtain the content video.

[0149] In one of the embodiments, when the computer program is executed by the processor, it also realizes synthesizing the digital human video after audio alignment with the background video to obtain the content video, which may include: cropping or supplementing the digital human video according to the different forms of temporal overlap between the digital human video and the background video; synthesizing the cropped or supplemented digital human video with the background video through picture-in-picture technology to obtain the content video.

[0150] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining material for generating a video, generating a background video based on the material; the background video includes video frames and audio aligned with each video frame; extracting the audio aligned with each video frame in the background video to obtain a video and audio file, and generating a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human; the digital human video and the background video are combined into a content video, and the content video is a video in which the digital human synchronously explains each video frame in the background video through a dynamic image.

[0151] In one embodiment, the material includes a plurality of image materials and text corresponding to each image material.

[0152] In this embodiment, when the computer program is executed by the processor, it also realizes the generation of background video based on the material, which may include: based on the text corresponding to each image material, obtaining the audio corresponding to each image material; based on each image material, generating a corresponding video screen, and aligning the video screen with the audio corresponding to the image material; based on multiple video screens and the audio aligned with the video screens, generating a background video.

[0153] In one embodiment, when the computer program is executed by the processor, it also implements obtaining audio corresponding to each image material based on the text corresponding to each image material, which may include: typeset the text corresponding to each image material, and a text paragraph obtained by typeset corresponds to one image material; based on each text paragraph, generating audio corresponding to each image material.

[0154] In one embodiment, the material is a presentation, the image material is a document page in the presentation or an animation page contained in the document page, and the text corresponding to the image material is the note text corresponding to the document page or the animation page.

[0155] In this embodiment, when the computer program is executed by the processor, it also realizes generating corresponding video screens based on each image material, and aligning the video screens with the audio corresponding to the image material, which may include: based on the document page in the presentation, generating the video screen corresponding to the document page; for the document page containing the animation page, the document page corresponds to multiple video screens, and for the document page not containing the animation page, the document page corresponds to one video screen; for each video screen, obtaining the text audio obtained based on the note text corresponding to the document page or animation page that generates the video screen, and aligning the video screen with the text audio.

[0156] In one embodiment, when the computer program is executed by the processor, it also implements obtaining text audio based on the note text corresponding to the document page or animation page that generates the video screen, and aligning the video screen with the text audio, which may include: obtaining text audio of the note text corresponding to multiple page animations or document pages, and aligning the multiple page animations or document pages with each text audio respectively; obtaining the text audio of the note text corresponding to the first document page in the presentation as the opening audio, and obtaining the duration of the opening audio; setting each text audio before the corresponding page animation or document page; based on the duration of the opening audio, delaying the playback of multiple page animations or document pages and each corresponding text audio by the same duration.

[0157] In one embodiment, when the computer program is executed by the processor, it also realizes the generation of digital human video based on the video and audio files, including: based on the video and audio files, the digital human audio is generated by a voice-driven algorithm, and a video screen containing the dynamic image of the digital human is generated by a digital human image generation program; based on the digital human audio and the video screen containing the dynamic image of the digital human, the digital human video is generated.

[0158] In one embodiment, when the computer program is executed by the processor, it also realizes synthesizing the digital human video and the background video into the content video, which may include: aligning the digital human audio corresponding to the digital human video with the video audio of the background video to obtain the digital human video and the background video after audio alignment; synthesizing the digital human video and the background video after audio alignment to obtain the content video.

[0159] In one embodiment, when the computer program is executed by the processor, it also realizes synthesizing the digital human video after audio alignment with the background video to obtain the content video, which may include: cropping or supplementing the digital human video according to the different forms of temporal overlap between the digital human video and the background video; synthesizing the cropped or supplemented digital human video with the background video through picture-in-picture technology to obtain the content video.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire material for generating a video, and generate a background video based on the material; the background video includes video frames and audio aligned with each video frame; Extracting the audio in the background video that is aligned with each video screen to obtain a video and audio file, and generating a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human; The digital human video and the background video are combined into a content video, wherein the content video is a video in which the digital human synchronously explains each video frame in the background video through a dynamic image; The material includes a plurality of image materials and texts corresponding to each image material, and the generating of the background video based on the material includes: Based on the text corresponding to each image material, obtain the audio corresponding to each image material; Generate a corresponding video screen based on each image material, and align the video screen with the audio corresponding to the image material; Generate the background video based on multiple video frames and audio aligned with the video frames; The material is a presentation, the image material is a document page in the presentation or an animation page contained in the document page, and the text corresponding to the image material is the note text corresponding to the document page or the animation page; When the manuscript page with the opening remarks contains n animation pages and the number of note texts exceeds n+1, the note texts exceeding n+1 are merged into the n+1th note text; When a document page without an opening statement includes n animation pages and the number of note texts exceeds n, the note texts exceeding n are merged into the nth note text.

2. The method according to claim 1, characterized in that: The step of obtaining the audio corresponding to each image material based on the text corresponding to each image material includes: Typesetting the text corresponding to each image material, wherein a text paragraph obtained by typeset corresponds to one of the image materials; Based on each text paragraph, audio corresponding to each image material is generated.

3. The method according to claim 2, characterized in that The generating a corresponding video screen based on each image material and aligning the video screen with the audio corresponding to the image material includes: Based on the document page in the presentation, a video screen corresponding to the document page is generated; for a document page containing an animation page, the document page corresponds to multiple video screens, and for a document page not containing an animation page, the document page corresponds to one video screen; For each video screen, a text audio is obtained based on the note text corresponding to the manuscript page or animation page that generates the video screen, and the video screen is aligned with the text audio.

4. The method according to claim 3, characterized in that The step of obtaining a text audio based on the note text corresponding to the manuscript page or animation page that generates the video screen, and aligning the video screen with the text audio includes: Acquire the text audio of the note text corresponding to the plurality of page animations or manuscript pages, and align the plurality of page animations or manuscript pages with each of the text audio respectively; Obtain the text audio of the note text corresponding to the first document page in the presentation as the opening audio, and obtain the duration of the opening audio; Setting each of the text audios before the corresponding page animations or manuscript pages; Based on the duration of the opening audio, the plurality of page animations or document pages and the corresponding text audio are delayed for the same duration for playback.

5. The method according to claim 1, characterized in that The step of generating a digital human video based on the video and audio file includes: Based on the video and audio files, a digital human audio is generated by a voice-driven algorithm, and a video screen containing a dynamic image of the digital human is generated by a digital human image generation program; A digital human video is generated based on the digital human audio and the video image containing the dynamic image of the digital human.

6. The method according to claim 1, characterized in that The step of synthesizing the digital human video and the background video into a content video includes: Aligning the digital human audio corresponding to the digital human video with the video audio of the background video to obtain the digital human video and the background video after audio alignment; The digital human video after audio alignment is synthesized with the background video to obtain a content video.

7. The method according to claim 6, characterized in that The step of synthesizing the digital human video after audio alignment with the background video to obtain a content video includes: According to different forms of overlap between the digital human video and the background video in time sequence, cutting or supplementing the digital human video; The cut or supplemented digital human video is synthesized with the background video through picture-in-picture technology to obtain a content video.

8. A video generating device, characterized in that: The device comprises: A background video generation module, used to obtain materials for generating a video, and generate a background video based on the materials; the background video includes video images and audio aligned with each video image; A digital human video generation module is used to extract the audio in the background video that is aligned with each video screen to obtain a video and audio file, and generate a digital human video based on the video and audio file; the digital human video is a video containing a dynamic image of the digital human; A content video synthesis module, used to synthesize the digital human video and the background video into a content video, wherein the content video is a video in which the digital human synchronously explains each video frame in the background video through a dynamic image; The material includes a plurality of image materials and text corresponding to each image material, and the background video generation module is further configured to: Based on the text corresponding to each image material, obtain the audio corresponding to each image material; Generate a corresponding video screen based on each image material, and align the video screen with the audio corresponding to the image material; Generate the background video based on multiple video frames and audio aligned with the video frames; The material is a presentation, the image material is a document page in the presentation or an animation page contained in the document page, and the text corresponding to the image material is the note text corresponding to the document page or the animation page; The background video generation module is also configured to: When the manuscript page with the opening remarks contains n animation pages and the number of note texts exceeds n+1, the note texts exceeding n+1 are merged into the n+1th note text; When a document page without an opening statement includes n animation pages and the number of note texts exceeds n, the note texts exceeding n are merged into the nth note text.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Method and terminal for demonstrating PPTX document in video mode

    CN111246288A

  • Method and system for automatically generating demonstration video, equipment and storage medium

    CN111538851A

  • Virtual human teaching video generation method, system and device and storage medium

    CN113194348A

  • Automatic slide explanation method and device

    CN113626013A