Video and slide generation method, electronic equipment and storage medium

Abstract

The invention relates to a video and slide generation method, electronic equipment and a storage medium, and the method comprises the steps: responding to a received file, extracting picture information and / or table information in the file, obtaining a file type corresponding to the file through detection, processing the file based on the file type, and converting the file into a preset form; in response to the received story background and text cue word information, taking the text information subjected to form conversion as input of a large language model, and taking the story background and the text cue word information as guidance to generate a lightweight language text; and converting the obtained lightweight language text to obtain a voice file, inserting the picture information and / or table information into the obtained lightweight language text, converting the lightweight language text into a slide file, and synthesizing the voice file and the slide file to obtain a video file, thereby realizing controllable video generation. Compared with the prior art, the method has the advantages of high correlation degree of the generated video and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video and slide generation method, electronic equipment, and storage medium. Background Art

[0002] In recent years, the rapid development of deep learning technology has made breakthrough progress in video generation technology. In particular, in the field of generative artificial intelligence, the emergence of video generation models, such as diffusion models, has shown the world the huge application potential of generative artificial intelligence. This method can automatically generate realistic video content under certain input guidance. This type of method has indeed brought great convenience to video creation, but due to its characteristics based on autoregressive models, the generated results have great uncertainty in stability and cannot be applied in actual scenarios.

[0003] In addition to deep learning methods, traditional video generation and editing technologies rely on manual operation and editing (Premiere Pro), and users need to master certain video editing skills to complete video production. Although some automated tools can generate videos, they usually only process short text inputs, and the generated results are relatively simple and cannot meet the needs of complex video production.

[0004] In addition, existing generation systems are often inefficient when processing long text inputs and the generated videos are of poor quality. Current technology still lacks an automated system that can efficiently process long text inputs and generate high-quality videos. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a video and slide generation method, electronic device, and storage medium to solve or partially solve the problems of not being able to input long texts and the high randomness and low consistency of text content in the videos generated by the existing deep learning methods.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides a method for generating a video and a slide, comprising the following steps:

[0008] Step S1, in response to receiving a file, extracting image information and / or table information in the file, obtaining a file type corresponding to the file by detection, processing the file based on the file type, and converting the file into a preset format;

[0009] Step S2, in response to receiving the story background and text prompt word information, using the text information after the form conversion as the input of the large language model, and using the story background and text prompt word information as a guide to generate a lightweight language text through the understanding reasoning of the large language model;

[0010] Step S3, for the lightweight language text obtained in step S2, obtain a voice file corresponding to the emotion in the text by conversion, insert the picture information and / or table information into the lightweight language text obtained in step S2, and convert it into a slide file, synthesize the voice file and the slide file to obtain a video file, and realize the generation of video and slide.

[0011] As a preferred technical solution, the process of processing files based on file types and converting files into a preset format includes the following steps:

[0012] For the text information, when the file type is multimodal text, the structured text is converted into extensible markup language and invalid symbols and content are removed through data cleaning. When the file type is plain text, invalid symbols and content are removed through data cleaning.

[0013] As a preferred technical solution, the file types of the multimodal text include PDF, HTML, DOCX, etc., and the file types of the plain text include txt, md, etc.

[0014] As a preferred technical solution, the process of obtaining the story background and text prompt word information includes the following steps:

[0015] Obtain the specified predefined story background and text prompt word information from the preset template library, or obtain the user-defined story background and text prompt word information through interaction.

[0016] As a preferred technical solution, the picture information includes a caption and a position, and the table information includes a table title and a position.

[0017] As a preferred technical solution, the following steps are also included:

[0018] Step S4, in response to receiving the modification instruction, acquiring the modified lightweight language text as the lightweight language text obtained in step S2 through interaction, and executing step S3.

[0019] As a preferred technical solution, in step S3, the similarity between the lightweight language text and the image information and / or table information is calculated to match the text with the image and / or table, thereby realizing the insertion of the image information and / or table information.

[0020] As a preferred technical solution, in step S3, the conversion between lightweight language text and voice file is achieved through speech synthesis.

[0021] As a preferred technical solution, the story background and prompt words designed based on the feedback from professionals are optimized multiple times using a large language model to obtain the story background and text prompt word information.

[0022] Another aspect of the present invention provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned video and slide generation method.

[0023] Another aspect of the present invention provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned video and slideshow generation method.

[0024] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0025] (1) The generated video has a high degree of relevance: The present invention automatically extracts pictures and tables from structured text, and inserts these elements into the generated text and slides in combination with similarity calculation, thereby realizing the automatic extraction and integration of pictures and tables, greatly enriching the content of videos and slides. In addition, the method can generate slides, which can display more detailed content than videos, helping users understand more details of the file.

[0026] (2) Strong flexibility: The present invention is based on an interactive video generation method and adopts a flexible story background and text prompt word design. Basic content creators can choose predefined story background and text prompt words, and it also supports user customization through interactive generation with a large language model interface. This flexibility enables the system to generate more personalized and diversified video content.

[0027] (3) High quality of generated video and simple process: The present invention integrates speech synthesis and video generation, converts text content into high-quality speech through speech synthesis method, and then automatically generates high-quality video content by combining the generated slides and pictures. This integrated processing method simplifies the video generation process and improves efficiency and effect.

[0028] (4) Full process automation: The present invention automates the entire process from text processing, image and table extraction, speech synthesis to video and slide generation. Users only need to make a small amount of input and selection to generate complete multimedia content. This greatly reduces the user's operating difficulty and time cost.

[0029] (5) Customizable: For different users, you can generate multiple videos of different lengths for the same file based on their needs, according to custom or pre-set prompt words and story background libraries.

[0030] (6) Convenient for users to modify: If the user needs to modify the generated video, the video content can be modified by modifying the text output in the previous round. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the process of generating video and slides in the embodiment. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] Example 1

[0034] In view of the problems existing in the above-mentioned prior art, this embodiment provides a method for generating a video and a slide show. Figure 1 , the method mainly includes the following steps:

[0035] Step S1, information processing of structured text.

[0036] When a user uploads a file, the file type is detected and processed differently according to the file type:

[0037] If the file type is detected as a multimodal text (such as PDF, HTML, and DOCX), it is converted into a form that can be directly processed by the large language model, specifically by converting the structured text into Extensible Markup Language (XML), and then cleaning its content to remove invalid symbols and content;

[0038] If the file type is detected as plain text (such as txt, md, etc.), the long text content is directly processed to remove invalid symbols and content, and analyzed and processed through a large language model.

[0039] Step S2, story background and text prompt word design.

[0040] Users can choose predefined story backgrounds and text prompts.

[0041] Users can also customize story backgrounds and text prompts by modifying predefined story backgrounds and text prompts and interacting with the large language model interface provided by the method.

[0042] Use predefined story context and text prompts to guide the generation process of the large language model;

[0043] Parse and process the input long text to generate corresponding data and content based on the text prompt words.

[0044] Specifically, in this embodiment, the story background and prompt words are designed based on the feedback of professionals, and are optimized multiple times using a large language model, ultimately generating three versions of prompt words and story background. They are used to generate 1-minute and 3-minute videos and 15-20-page slides, respectively. The optimization goal here is to optimize the expression of prompt words and the content required for videos and slides of different lengths, so that the subsequent large model can better extract the required content based on the prompt words and story background.

[0045] Step S3, extracting pictures and tables of structured text, generating slides, and generating videos.

[0046] Step S301, extracting pictures and tables from structured text for processing.

[0047] Extract relevant images and tables from structured text, and use existing tools (such as Grobid, PDFFigures2) to extract all images, tables and related information (including but not limited to captions, table titles, locations, etc.) to enrich video content.

[0048] Step S302: Slide generation.

[0049] Process the text generated by the large language model and convert it into lightweight markup language text (Markdown) through the understanding and reasoning of the large language model;

[0050] According to the image or table information contained in the generated text, matching images and tables are found through similarity calculation and inserted into the lightweight markup language text;

[0051] Convert lightweight markup language text into slide files (pptx) through existing tools (python-pptx).

[0052] Specifically, during the similarity calculation process, the understanding and reasoning capabilities of the large language model are used to search for and output the pictures and table titles that are most relevant to the lightweight tagged text from the input text; the pictures and tables extracted in the previous step contain their locations and names; to address the problem of hallucinations in the large language model, the extracted titles and all the names in the previous step are vectorized, and then the cosine similarity of the two is calculated one by one. When the similarity exceeds the threshold, the match is successful.

[0053] Step S303: video generation.

[0054] Based on the text generated by the large language model, it is converted into audio text corresponding to the emotions in the text;

[0055] Convert the audio text content into speech according to the speech synthesis method as the narration of the video;

[0056] Generate video content based on the generated slides, pictures and voice.

[0057] In the speech synthesis process, the character sequence is directly mapped to the acoustic feature sequence by constructing an encoder-decoder framework. Specifically, first, the input text is preprocessed, including text normalization and word segmentation. Next, the encoder converts the preprocessed text representation (discrete text tokens (such as words or subwords)) into a continuous latent space representation containing contextual information, which can be implemented using neural networks such as RNN, LSTM, GRU or Transformer. The decoder then converts these latent representations into audio features (such as Mel spectrograms), which are then converted into actual audio waveforms through a vocoder (such as WaveNet, WaveGlow or MelGAN). The final audio undergoes post-processing steps such as denoising and volume adjustment to improve the sound quality.

[0058] Optionally, the method further includes step S4 of regenerating the video.

[0059] If the user needs to modify the generated video, the video content can be modified by modifying the text output in the previous round.

[0060] This method has the following beneficial effects:

[0061] (1) Flexible story background and text prompt word design: This method allows users to select predefined story background and text prompt words, and also supports user customization through interactive generation with the large language model interface. This flexibility enables the system to generate more personalized and diverse video content.

[0062] (2) Automatic extraction and integration of images and tables: Use tools (such as Grobid and PDFFigures2) to automatically extract images and tables from structured text, and combine similarity calculations to intelligently insert these elements into the generated text and slides, greatly enriching the video and slide content.

[0063] (3) Integrated speech synthesis and video generation: The text content is converted into high-quality speech through existing speech synthesis methods (such as ChatTTS, edge-tts, xtts, etc.), and then combined with the generated slides and pictures to automatically generate high-quality video content. This integrated processing method simplifies the video generation process and improves efficiency and effectiveness.

[0064] (4) Full process automation: This method automates the entire process from text processing, image and table extraction, speech synthesis to video and slide generation. Users only need to make a small amount of input and selection to generate complete multimedia content. This greatly reduces the user's operating difficulty and time cost.

[0065] (5) Convenient for users to modify: If the user needs to modify the generated video, the video content can be modified by modifying the text output in the previous round.

[0066] Example 2

[0067] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the video and slide generation method as described in Example 1.

[0068] Example 3

[0069] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the video and slideshow generation method as described in Embodiment 1.

[0070] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for generating a video and a slide, characterized in that: The steps include: Step S1, in response to receiving a file, extracting image information and / or table information in the file, obtaining a file type corresponding to the file by detection, processing the file based on the file type, and converting the file into a preset format; Step S2, in response to receiving the story background and text prompt word information, using the text information after the form conversion as the input of the large language model, and using the story background and text prompt word information as a guide to generate a lightweight language text through the understanding reasoning of the large language model; Step S3, for the lightweight language text obtained in step S2, obtain a voice file corresponding to the emotion in the text by conversion, insert the picture information and / or table information into the lightweight language text obtained in step S2, and convert it into a slide file, synthesize the voice file and the slide file to obtain a video file, and realize the generation of video and slide.

2. A video and slideshow generation method according to claim 1, characterized in that: The process of processing the file based on the file type and converting the file into a preset format includes the following steps: For the text information, when the file type is multimodal text, the structured text is converted into extensible markup language and invalid symbols and content are removed through data cleaning. When the file type is plain text, invalid symbols and content are removed through data cleaning.

3. A video and slideshow generation method according to claim 2, characterized in that: The file types of the multimodal text include PDF, HTML and DOCX, and the file types of the plain text include txt and md.

4. A video and slideshow generation method according to claim 1, characterized in that: The process of obtaining the story background and text prompt word information includes the following steps: Obtain the specified predefined story background and text prompt word information from the preset template library, or obtain the user-defined story background and text prompt word information through interaction.

5. A video and slideshow generation method according to claim 1, characterized in that: The picture information includes a caption and a position, and the table information includes a table and a position.

6. A video and slideshow generation method according to claim 1, characterized in that: The following steps are also included: Step S4, in response to receiving the modification instruction, acquiring the modified lightweight language text as the lightweight language text obtained in step S2 through interaction, and executing step S3.

7. A video and slideshow generation method according to claim 1, characterized in that: In step S3, the similarity between the lightweight language text and the picture information and / or table information is calculated to match the text with the picture and / or table, so as to insert the picture information and / or table information, and the conversion between the lightweight language text and the voice file is realized through speech synthesis.

8. A video and slideshow generation method according to claim 1, characterized in that: In the step S3, the story background and prompt words designed according to the feedback from professionals are optimized multiple times using a large language model to obtain the story background and text prompt word information.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the video and slide generation method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: It comprises one or more programs executed by one or more processors of an electronic device, wherein the one or more programs comprise instructions for executing the video and slideshow generating method as claimed in any one of claims 1 to 8.

Images