Multi-format document intelligent audio content generation system and method

Abstract

The invention discloses a multi-format document intelligent audio content generation system and method, and relates to the technical field of artificial intelligence and audio generation. Comprising the steps of 1, analyzing various formats of documents uploaded by a user to generate a structured plain text, 2, performing semantic analysis according to the content of the structured plain text, identifying different emotional colors and mood characteristics in the plain text, converting the plain text into a multi-role dialogue script according to the emotional colors and the mood characteristics, and storing the multi-role dialogue script in a database. 3, calling a text-to-audio model, and generating an audio clip of a corresponding role for each line segment of the dialogue script, and 4, splicing and synthesizing the audio clips in the audio stream list into a complete audio file according to a line sequence; automatic conversion of multi-format document content and multi-role audio synthesis are realized, dependence on cloud service is not needed, data privacy is guaranteed, and processing efficiency is improved.

Description

Technical Field

[0001] The invention discloses a system and method for intelligently generating audio content of multi-format documents, and relates to the technical field of artificial intelligence and audio generation. Background Art

[0002] With the popularization of digital content, various document formats are increasing, such as PDF, Word, Markdown, etc. These documents usually contain rich text and image content, which takes a long time to read manually and cannot be directly converted into multimedia content for rapid dissemination. However, the demand for automatically converting static documents into audio content is growing, especially in the fields of education, corporate training, cultural communication, etc. However, existing technologies still have significant limitations in processing documents in complex formats, the content and quality of dialogue generation, and role diversity. In addition, most existing document-to-audio technologies rely on cloud processing, involving more privacy issues and lower response speeds. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a system and method for generating intelligent audio content for multi-format documents. Through localized AI technology, automatic conversion of multi-format document content and multi-role audio synthesis are achieved without relying on cloud services, thereby ensuring data privacy and improving processing efficiency.

[0004] The specific scheme proposed by the present invention is: The present invention provides a method for generating intelligent audio content of multi-format documents, comprising: Step 1: Parse the documents in various formats uploaded by users to generate structured plain text. Step 2: Perform semantic analysis based on the content of structured plain text, identify different emotional colors and tone features in the plain text, and convert the plain text into a multi-role dialogue script based on the emotional colors and tone features: load the llama_cpp library, call the text generation model, receive the content of the plain text, specify prompt words, and guide the text generation model to generate a multi-role dialogue script in JSON format according to the content of the plain text and the emotional colors and tone features, and specify the lines of each role in chronological order. Step 3: Call the text-to-audio model to generate audio clips of the corresponding characters for each line of the dialogue script: Load the Transformer library, call the large text-to-audio model, and pre-match different audio packages according to the character identifier. Receive the dialogue script, traverse each line in the dialogue script, mark the line with the role ID, send the role ID in the line to the text-to-audio model, generate an audio clip in numpy format, adjust the voice parameters of the role in the audio clip, the voice parameters include timbre, speaking speed and tone, and store all the audio clips generated according to the dialogue script in the audio stream list in the order of the lines. Step 4: Combine the audio clips in the audio stream list into a complete audio file in the order of the lines and output it.

[0005] Furthermore, step 1 of the method for generating intelligent audio content for multi-format documents includes: identifying the file type based on the file suffixes of the documents in various formats uploaded by the user, calling the parsing library of each file type to extract the text of the document, and generating structured plain text after removing the uniform resource locator URL, special characters and irrelevant tags based on regular expressions.

[0006] Furthermore, step 4 of the method for generating intelligent audio content for multi-format documents includes: when splicing the audio clips in the audio stream list in the order of lines, adding silent audio of a preset length at the end of the audio of each character as a buffer between characters to form a complete audio stream, and then converting the audio stream into an audio file in WAV format and providing it for download.

[0007] Furthermore, the method for generating intelligent audio content of multi-format documents also includes step 5, in which end-to-end process control and parameter adjustment are performed through a visual interface, wherein document uploading, real-time dialogue script preview, voice parameter configuration and dialogue script modification are performed through the visual interface.

[0008] The present invention also includes a multi-format document intelligent audio content generation system, including a document format parsing module, a text understanding and distribution module, a multi-role synthesis module, and an audio generation and output module. The document format parsing module parses documents in various formats uploaded by users and generates structured plain text. The text understanding and distribution module performs semantic analysis based on the content of structured plain text, identifies different emotional colors and tone features in the plain text, and converts the plain text into a multi-role dialogue script based on the emotional colors and tone features: loads the llama_cpp library, calls the text generation model, receives the content of the plain text, specifies prompt words, and guides the text generation model to generate a multi-role dialogue script in JSON format according to the content of the plain text and the emotional colors and tone features, and specifies the lines of each role in chronological order. The multi-role synthesis module calls the text-to-audio model to generate audio clips of the corresponding characters for each line of the dialogue script: load the Transformer library, call the large text-to-audio model, and pre-match different audio packages according to the character identifier. Receive the dialogue script, traverse each line in the dialogue script, mark the line with the role ID, send the role ID in the line to the text-to-audio model, generate an audio clip in numpy format, adjust the voice parameters of the role in the audio clip, the voice parameters include timbre, speaking speed and tone, and store all the audio clips generated according to the dialogue script in the audio stream list in the order of the lines. The audio generation and output module combines the audio clips in the audio stream list into a complete audio file according to the order of the lines and outputs it.

[0009] Furthermore, in the multi-format document intelligent audio content generation system, the document format parsing module identifies the file type based on the file suffixes of the documents in various formats uploaded by the user, calls the parsing library of each file type to extract the text of the document, and generates structured plain text after removing the uniform resource locator URL, special characters and irrelevant tags based on regular expressions.

[0010] Furthermore, when the audio generation and output module in the multi-format document intelligent audio content generation system splices the audio clips in the audio stream list according to the order of lines, silent audio of a preset length is added at the end of the audio of each character as a buffer between characters to form a complete audio stream, and then the audio stream is converted into an audio file in WAV format and provided for download.

[0011] Furthermore, the multi-format document intelligent audio content generation system also includes a definition control and user interaction module, which performs end-to-end process control and parameter adjustment through a visual interface, wherein document uploading, real-time dialogue script preview, voice parameter configuration and dialogue script modification are performed through the visual interface.

[0012] The benefits of the present invention are: It can process a variety of document formats, automatically generate high-quality multi-role interactive audio content, and provide functions such as custom role configuration and voice style adjustment to achieve personalized content production. At the same time, all data processing and reasoning processes can be completed locally and can be executed without a network connection, avoiding the risk of data leakage while improving response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the interaction of modules in the system of the present invention. DETAILED DESCRIPTION

[0014] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0015] Embodiment 1:

[0016] The present invention provides a method for generating intelligent audio content of multi-format documents, comprising: Step 1: Parse the documents in various formats uploaded by users to generate structured plain text. According to the file suffixes of the documents in various formats uploaded by users, such as .html, .pdf, .txt, .docx, etc., identify the file type and call the parsing library of each file type to extract the text of the document. For example, the .docx file uses the Document library to extract the text of the document, and the .pdf file uses the PyPDF2 library to extract the text of the document. After removing the Uniform Resource Locator (URL), special characters and irrelevant tags based on regular expressions, structured plain text is generated.

[0017] Step 2: Perform semantic analysis based on the content of structured plain text, identify different emotional colors and tone features in plain text, and convert plain text into multi-role dialogue scripts based on emotional colors and tone features: load llama_cpp library, call text generation model, receive the content of plain text, specify prompt words, guide text generation model to generate multi-role dialogue scripts in JSON format according to the content of plain text, emotional colors and tone features, and specify the lines of each role in chronological order. For example, the text generation model can be deployed locally, and the text generation model can be selected according to actual needs, such as ChatGLM-6B model, LLama 2 model, Mistral-7B model, Baichuan-7B model, etc. After the text generation model is started, it resides in CPU memory. Every time the plain text content is input, it can rely only on CPU without GPU for inference to generate text-based dialogue script results.

[0018] Step 3: Call the text-to-audio model to generate audio clips of the corresponding characters for each line of the dialogue script: Load the Transformer library, call the large text-to-audio model, and pre-match different audio packages according to the character identifier. Receive the dialogue script, traverse each line in the dialogue script, mark the line with the role ID, send the role ID in the line to the text-to-audio model, generate an audio clip in numpy format, adjust the voice parameters of the role in the audio clip, the voice parameters include timbre, speaking speed and tone, and store all the audio clips generated according to the dialogue script in the order of the lines into the audio stream list.

[0019] For example, a large lightweight text-to-speech model can be deployed locally. The large lightweight text-to-speech model can be selected according to actual needs, such as the Parler-TTS model, the Kokoro-TTS model, etc. After the large lightweight text-to-speech model is started, it resides in the CPU memory. It can preset voice packages with different voice styles, input text content, and generate corresponding audio output.

[0020] Step 4: Splice the audio clips in the audio stream list in the order of the lines to form a complete audio file and output it. When splicing the audio clips in the audio stream list in the order of the lines, add a preset length of silent audio at the end of each character's audio as a buffer between characters to form a complete audio stream, and then convert the audio stream into a WAV format audio file and provide it for download.

[0021] Step 5 performs end-to-end process control and parameter adjustment through a visual interface, including document upload, real-time dialogue script preview, voice parameter configuration and dialogue script modification.

[0022] For example, after a user uploads a document, he or she can view the generated conversation script and a short playable audio file in real time in the output window of the visual interface.

[0023] Embodiment 2: The present invention also includes a multi-format document intelligent audio content generation system, including a document format parsing module, a text understanding and distribution module, a multi-role synthesis module, and an audio generation and output module. The document format parsing module parses documents in various formats uploaded by users and generates structured plain text. The text understanding and distribution module performs semantic analysis based on the content of structured plain text, identifies different emotional colors and tone features in the plain text, and converts the plain text into a multi-role dialogue script based on the emotional colors and tone features: loads the llama_cpp library, calls the text generation model, receives the content of the plain text, specifies prompt words, and guides the text generation model to generate a multi-role dialogue script in JSON format according to the content of the plain text and the emotional colors and tone features, and specifies the lines of each role in chronological order. The multi-role synthesis module calls the text-to-audio model to generate audio clips of the corresponding characters for each line of the dialogue script: load the Transformer library, call the large text-to-audio model, and pre-match different audio packages according to the character identifier. Receive the dialogue script, traverse each line in the dialogue script, mark the line with the role ID, send the role ID in the line to the text-to-audio model, generate an audio clip in numpy format, adjust the voice parameters of the role in the audio clip, the voice parameters include timbre, speaking speed and tone, and store all the audio clips generated according to the dialogue script in the audio stream list in the order of the lines. The audio generation and output module combines the audio clips in the audio stream list into a complete audio file according to the order of the lines and outputs it.

[0024] As the information interaction and execution process between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0025] Similarly, the system of the present invention can process a variety of document formats, automatically generate high-quality multi-role interactive audio content, and can provide functions such as custom role configuration and voice style adjustment to achieve personalized content production. At the same time, all data processing and reasoning processes can be completed locally and can be executed without a network connection, avoiding the risk of data leakage while improving response speed.

[0026] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0027] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for generating intelligent audio content of multi-format documents, characterized in that include: Step 1: Parse the documents in various formats uploaded by users to generate structured plain text. Step 2: Perform semantic analysis based on the content of structured plain text, identify different emotional colors and tone features in the plain text, and convert the plain text into a multi-role dialogue script based on the emotional colors and tone features: load the llama_cpp library, call the text generation model, receive the content of the plain text, specify prompt words, and guide the text generation model to generate a multi-role dialogue script in JSON format according to the content of the plain text and the emotional colors and tone features, and specify the lines of each role in chronological order. Step 3: Call the text-to-audio model to generate audio clips of the corresponding characters for each line of the dialogue script: Load the Transformer library, call the large text-to-audio model, and pre-match different audio packages according to the character identifier. Receive the dialogue script, traverse each line in the dialogue script, mark the line with the role ID, send the role ID in the line to the text-to-audio model, generate an audio clip in numpy format, adjust the voice parameters of the role in the audio clip, the voice parameters include timbre, speaking speed and tone, and store all the audio clips generated according to the dialogue script in the audio stream list in the order of the lines. Step 4: Combine the audio clips in the audio stream list into a complete audio file in the order of the lines and output it.

2. The method for generating intelligent audio content of a multi-format document according to claim 1, characterized in that Step 1 includes: identifying the file type according to the file suffixes of the documents in various formats uploaded by the user, calling the parsing library of each file type to extract the text of the document, and generating structured plain text after removing the uniform resource locator URL, special characters and irrelevant tags based on regular expressions.

3. The method for generating intelligent audio content of a multi-format document according to claim 1, characterized in that Step 4 includes: when splicing the audio clips in the audio stream list in the order of the lines, adding silent audio of a preset length at the end of each character's audio as a buffer between characters to form a complete audio stream, and then converting the audio stream into an audio file in WAV format and providing it for download.

4. The method for generating intelligent audio content of a multi-format document according to claim 1, characterized in that It also includes step 5, in which end-to-end process control and parameter adjustment are performed through a visual interface, wherein document uploading, real-time dialogue script preview, voice parameter configuration and dialogue script modification are performed through the visual interface.

5. A multi-format document intelligent audio content generation system, characterized by It includes document format parsing module, text understanding and distribution module, multi-role synthesis module, audio generation and output module. The document format parsing module parses documents in various formats uploaded by users and generates structured plain text. The text understanding and distribution module performs semantic analysis based on the content of structured plain text, identifies different emotional colors and tone features in the plain text, and converts the plain text into a multi-role dialogue script based on the emotional colors and tone features: loads the llama_cpp library, calls the text generation model, receives the content of the plain text, specifies prompt words, and guides the text generation model to generate a multi-role dialogue script in JSON format according to the content of the plain text and the emotional colors and tone features, and specifies the lines of each role in chronological order. The multi-role synthesis module calls the text-to-audio model to generate audio clips of the corresponding characters for each line of the dialogue script: load the Transformer library, call the large text-to-audio model, and pre-match different audio packages according to the character identifier. Receive the dialogue script, traverse each line in the dialogue script, mark the line with the role ID, send the role ID in the line to the text-to-audio model, generate an audio clip in numpy format, adjust the voice parameters of the role in the audio clip, the voice parameters include timbre, speaking speed and tone, and store all the audio clips generated according to the dialogue script in the audio stream list in the order of the lines. The audio generation and output module combines the audio clips in the audio stream list into a complete audio file according to the order of the lines and outputs it.

6. A multi-format document intelligent audio content generation system according to claim 5, characterized in that the document The format parsing module identifies the file type based on the file suffixes of documents in various formats uploaded by users, calls the parsing library of each file type to extract the document text, and generates structured plain text after removing the uniform resource locator URL, special characters and irrelevant tags based on regular expressions.

7. The multi-format document intelligent audio content generation system according to claim 5, characterized in that When the audio generation and output module splices the audio clips in the audio stream list in the order of the lines, it adds silent audio of a preset length at the end of each character's audio as a buffer between characters to form a complete audio stream, and then converts the audio stream into an audio file in WAV format and provides it for download.

8. The multi-format document intelligent audio content generation system according to claim 5, characterized in that It also includes a definition control and user interaction module, which performs end-to-end process control and parameter adjustment through a visual interface, including document upload, real-time dialogue script preview, voice parameter configuration and dialogue script modification.

Images