Method, device and equipment for generating personalized oral broadcast based on AI and medium

Abstract

The invention provides a method, a device and equipment for generating personalized oral broadcast based on AI, and a medium. The method comprises the following steps: inputting oral broadcast data and a set cue word into a multi-modal model; the oral broadcast data comprises an original text and a corresponding scene picture; the multi-modal model outputs an identification text with a set output format, wherein the identification text comprises a set identifier; segmenting the identification text, and generating a corresponding audio file through a voice model; if only one audio file is provided, the generation is completed; if not, the audio is spliced to complete generation, the corresponding audio can be generated according to the user demand, and the time cost of the user is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for generating personalized oral broadcast based on AI. Background Art

[0002] Video oral broadcast refers to the process in which the creator conveys information, expresses opinions, tells stories or introduces products through oral narration in video content creation. It is very common in e-commerce videos and product recommendation videos. Bloggers introduce the various attributes of products in detail through oral broadcast. Product recommendation oral broadcasts not only introduce the product itself, but also involve information such as price, discount activities, and purchase channels, and usually combine their own usage experience to enhance credibility and appeal.

[0003] Requirements for video broadcast:

[0004] 1. Clear and fluent: Make sure you speak clearly and avoid stuttering or slurred expressions. For example, practice your lines in advance and familiarize yourself with the content so that you can express yourself naturally and fluently when recording.

[0005] 2. Speed ​​and tone: Control the speed of speech reasonably and adjust the tone according to the importance and emotional color of the content. You can slow down the speed appropriately when introducing key content, and raise the tone when expressing emotions such as excitement. For example, when recommending a surprising product, you can use a slightly higher tone to highlight its advantages.

[0006] 3. Colloquial language: Try to use easy-to-understand spoken language, avoid overly obscure words and complex sentence structures, so that the audience can easily understand.

[0007] The existing video broadcast is done by the creator recording the audio himself, and then combining it with pictures to display it in the form of video; although there is existing AI technology that can convert text into speech, the generated speech is stereotyped and lifeless, which is not conducive to user use. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for generating personalized oral broadcasts based on AI, which can generate corresponding audio according to user needs and reduce the user's time cost.

[0009] In a first aspect, the present invention provides a method for generating personalized oral broadcast based on AI, comprising the following steps:

[0010] Step 1: input the oral data and the set prompt words into the multimodal model; the oral data includes the original text and the corresponding scene picture;

[0011] Step 2: The multimodal model outputs an identification text of a set output format, wherein the identification text includes a set identification;

[0012] Step 3: Segment the identification text and generate corresponding audio files through a speech model;

[0013] Step 4: If there is only one audio file, the generation is completed; otherwise, they are concatenated to complete the generation.

[0014] In a second aspect, the present invention provides a device for generating personalized oral broadcast based on AI, comprising:

[0015] A setting module inputs the oral data and the set prompt words into the multimodal model; the oral data includes the original text and the corresponding scene picture;

[0016] An output module, wherein the multimodal model outputs a mark text of a set output format, wherein the mark text includes a set mark;

[0017] A segmentation module, which segments the identification text and generates corresponding audio files through a speech model;

[0018] The generation module completes the generation if there is only one audio file; otherwise, the audio files are concatenated to complete the generation.

[0019] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0021] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0022] The present invention combines a multimodal model to mark the text according to the settings and specific scenarios, and then enables the speech model to generate speech with various speaking speeds, volumes, and emotions, greatly improving the speech quality, so that it can be used in different scenarios and users can use it very conveniently; and its generation efficiency is fast, which greatly reduces the user's time cost and improves the user experience.

[0023] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings in conjunction with embodiments.

[0025] Figure 1 It is a flowchart of the method in the first embodiment of the present invention;

[0026] Figure 2 It is a schematic structural diagram of the device in the second embodiment of the present invention. Detailed implementation manners

[0027] By providing a method, device, equipment and medium for generating personalized voiceovers based on AI, the embodiments of the present application solve the technical problems of the existing voiceovers being stereotyped and unable to be used in specific scenarios.

[0028] The general idea of the technical solution in the embodiments of the present application is as follows:

[0029] The text content is: Hey TikTok fam! You won't believe the amazing product I just discovered! It's a total game-changer for anyone who loves online shopping!

[0030] I was absolutely blown away when I found out about this deal! Imagine getting top-quality items at such an incredible price—it's almost too good to be true!

[0031] This product not only offers exceptional value but also stands out for its durability and style. Perfect for enhancing your daily life and making everything just a bit more fabulous. Trust me, you don't want to miss out on this one!

[0032] Translation: Hey, TikTok fam! You won't believe the amazing product I just discovered! It's a total game-changer for anyone who loves online shopping!

[0033] When I found this deal, I was totally blown away! Imagine getting top quality merchandise at such an incredible price – it’s almost too good to be true!

[0034] Not only does this product offer exceptional value, it also stands out for its durability and style. Perfect for elevating your everyday life and making everything a little more exciting. Trust me, you definitely don't want to miss this one!

[0035] The above are the original texts of 3 paragraphs. The original texts need to be converted into spoken texts for playback. The above three paragraphs can correspond to three different scenes or the same scene.

[0036] 1. Generate a spoken script

[0037] The input scene image is understood through the multimodal model; the material in the scene, that is, a frame of the picture or video, is uploaded to the multimodal model, and prompts are configured to allow the model to generate a spoken script;

[0038] The key code for calling the multimodal model (GPT-4O model):

[0039]

[0040] Where <% content%> is:

[0041] Hey TikTok fam! You won't believe the amazing product Ijustdiscovered! It's a total game-changer for anyone who loves online shopping!

[0042] Suppose you are an internet celebrity with a large number of fans, and now you want to introduce a product to everyone. Please design how to give the above speech. You hope that the whole process has different emotions, tones, speaking speeds and pauses, but you must ensure that the overall rhythm is coherent, without any discontinuous feeling.

[0043] Notice:

[0044] 1. You must pause after each paragraph and mark it with: <% stop: 1%>;

[0045] 2. In order to emphasize the tone, a pause is also required after certain words. You can put a mark after the word, and the pause time can be different, marked as: <% stop: x%>, x is in seconds, supporting a range of 0.01-1.00;

[0046] 3. The emotion of each paragraph will change. Put a mark before each paragraph to indicate which emotion should be used in this paragraph: <% emotion: y%>, y can be replaced by happy, sad, angry, fearful, disgusted, surprised, neutral;

[0047] 4. The speaking speed of each sentence can be changed. Put a mark before each sentence, which is <%speed:z%>, where z is the speaking speed, which can be between 0.5-1.3;

[0048] 5. The volume of each sentence can be changed. It is indicated before each sentence as <% vol:v%>, where v is the volume and can be between 0.5-2.

[0049] The output of the multimodal model is:

[0050] <%emotion:happy%><%speed:1.0%><%vol:1.5%>Hey TikTok fam! <%stop:0.5%><%vol:1%>You won't believe<%stop:0.2%>the amazing product<%stop:0.3%><%vol:2%>I just discovered! <%stop:0.5%><%vol:1%>It's a totalgame-changer<%stop:0.2%>for anyone<%stop:0.1%>who loves online shopping! <%stop:1%>;

[0051] 2. Segment text

[0052] The output of the multimodal model includes changes in emotions, pauses, speaking speed, and volume, which personalizes the single sentence dimension and is more appropriate when combined with the understanding of the material:

[0053] The T2A speech model is used. The T2A speech model itself supports pauses, so it can be segmented according to <%emotion%>, <%speed%>, and <%vol%>, and then the T2A speech model API is called respectively;

[0054] The result of segmentation:

[0055]

[0056]

[0057] 3. The sentences of other scenes are processed in sequence according to the above method, and finally spliced ​​with ffmpeg:

[0058] Suppose you have three audio files: audio1.mp3, audio2.mp3, and audio3.mp3. You can concatenate them together using the following command:

[0059] ffmpeg-i audio1.mp3-i audio2.mp3-i audio3.mp3-filter_complex\"[0:a][1:a][2:a]concat=n=3:v=0:a=1[outa]"\

[0060] -map "[outa]" output.mp3

[0061] Command analysis

[0062] -i audio1.mp3-i audio2.mp3-i audio3.mp3: specifies the input audio file.

[0063] concat=n=3:v=0:a=1: connect three audio streams together, n=3 means there are three audio streams, v=0 means there is no video stream, and a=1 means there is one audio stream output.

[0064] -map "[outa]" output.mp3: Map the final audio stream to the output file output.mp3.

[0065] [0:a][1:a][2:a] are identifiers used to reference the input audio streams. Specifically: 0, 1, 2: These numbers represent the index of the input file. FFmpeg starts numbering the input files at 0, so:

[0066] 0 represents the first input file ( audio1.mp3 in this example).

[0067] 1 represents the second input file (audio2.mp3).

[0068] 2 represents the third input file (audio3.mp3).

[0069] :a: indicates that the audio stream is being referenced. FFmpeg supports multiple stream types, including audio (a) and video (v). Among them, :a refers to the audio stream.

[0070] According to the above method, generate corresponding audio for each paragraph, and then splice them. After that, splice the audio of all paragraphs to complete it.

[0071] Embodiment 1

[0072] like Figure 1 As shown, this embodiment provides a method for generating personalized oral broadcast based on AI, including the following steps:

[0073] Step 1: input the oral data and the set prompt words into the multimodal model; the oral data includes the original text and the corresponding scene picture;

[0074] Step 2: The multimodal model outputs an identification text of a set output format, wherein the identification text includes a set identification;

[0075] Step 3: Segment the identification text and generate corresponding audio files through a speech model;

[0076] Step 4: If there is only one audio file, the generation is completed; otherwise, they are concatenated to complete the generation.

[0077] In this embodiment, preferably, the step 1 is specifically:

[0078] The prompt words include: setting output requirements and output formats; the output formats include:

[0079] Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph;

[0080] Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word;

[0081] Set an emotion mark before each paragraph to set the emotion used in the paragraph;

[0082] A speech rate indicator is set before each sentence to set the speech rate used for the sentence;

[0083] A volume mark is set before each sentence to set the volume used for the sentence;

[0084] The spoken data and the set prompt words are input into the multimodal model; the spoken data includes original text and corresponding scene pictures, and the original text includes at least one paragraph, and each paragraph corresponds to a scene picture.

[0085] In this embodiment, preferably, the step 3 is specifically: using regular matching to segment the identification text according to the emotion identification, the speaking rate identification, the volume identification and the second pause identification to obtain segmented text, and generating a corresponding audio file from the segmented text through the T2A speech model.

[0086] In this embodiment, preferably, the step 4 is specifically as follows: if there is only one audio file, the generation is completed; if not, all audio files are concatenated in a set order by FFmpeg to complete the generation.

[0087] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0088] Embodiment 2

[0089] like Figure 2 As shown, in this embodiment, a device for generating personalized oral broadcast based on AI is provided, including:

[0090] A setting module inputs the oral data and the set prompt words into the multimodal model; the oral data includes the original text and the corresponding scene picture;

[0091] An output module, wherein the multimodal model outputs a mark text of a set output format, wherein the mark text includes a set mark;

[0092] A segmentation module, which segments the identification text and generates corresponding audio files through a speech model;

[0093] The generation module completes the generation if there is only one audio file; otherwise, the audio files are concatenated to complete the generation.

[0094] In this embodiment, preferably, the setting module is specifically: the prompt word includes: setting output requirements and output format; the output format includes:

[0095] Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph;

[0096] Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word;

[0097] Set an emotion mark before each paragraph to set the emotion used in the paragraph;

[0098] A speech rate indicator is set before each sentence to set the speech rate used for the sentence;

[0099] A volume mark is set before each sentence to set the volume used for the sentence;

[0100] The spoken data and the set prompt words are input into the multimodal model; the spoken data includes original text and corresponding scene pictures, and the original text includes at least one paragraph, and each paragraph corresponds to a scene picture.

[0101] In this embodiment, preferably, the segmentation module specifically: uses regular matching to segment the identification text according to the emotion identification, the speaking speed identification, the volume identification and the second pause identification to obtain segmented text, and generates a corresponding audio file from the segmented text through the T2A speech model.

[0102] In this embodiment, preferably, the generating module is specifically as follows: if there is only one audio file, the generating module completes the generating; if not, all the audio files are concatenated in a set order by FFmpeg to complete the generating.

[0103] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0104] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0105] Embodiment 3

[0106] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0107] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0108] Based on the same inventive concept, the present application provides a storage medium corresponding to the first embodiment, see the fourth embodiment for details.

[0109] Embodiment 4

[0110] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0111] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0112] This embodiment integrates multimodal technology and can intelligently mark text according to preset scenarios. On this basis, the voice model can generate voice output with different speech speeds, volumes and emotions, significantly improving the quality and applicability of the voice, enabling it to flexibly respond to diverse application scenarios. Users can easily adjust voice parameters as needed for convenient use. In addition, this technology also greatly speeds up the generation efficiency, effectively reduces the user's time cost, and thus greatly enhances the user experience.

[0113] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for generating personalized oral broadcast based on AI, characterized by: The steps include: Step 1: input the oral data and the set prompt words into the multimodal model; the oral data includes the original text and the corresponding scene picture; Step 2: The multimodal model outputs an identification text of a set output format, wherein the identification text includes a set identification; Step 3: Segment the identification text and generate corresponding audio files through a speech model; Step 4: If there is only one audio file, the generation is completed; otherwise, they are concatenated to complete the generation.

2. The method for generating personalized oral broadcast based on AI according to claim 1, characterized in that: The step 1 is specifically as follows: The prompt words include: setting output requirements and output formats; the output formats include: Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph; Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word; Set an emotion mark before each paragraph to set the emotion used in the paragraph; A speech rate indicator is set before each sentence to set the speech rate used for the sentence; A volume mark is set before each sentence to set the volume used for the sentence; The spoken data and the set prompt words are input into the multimodal model; the spoken data includes original text and corresponding scene pictures, and the original text includes at least one paragraph, and each paragraph corresponds to a scene picture.

3. The method for generating personalized oral broadcast based on AI according to claim 2, characterized in that: The step 3 is specifically as follows: using regular matching to segment the identification text according to the emotion identification, the speaking speed identification, the volume identification and the second pause identification to obtain segmented text, and generating corresponding audio files from the segmented text through the T2A speech model.

4. The method for generating personalized oral broadcast based on AI according to claim 1, characterized in that: The step 4 is specifically as follows: if there is only one audio file, the generation is completed; if not, all the audio files are concatenated in a set order by FFmpeg to complete the generation.

5. A device for generating personalized oral broadcast based on AI, characterized in that: include: A setting module inputs the oral data and the set prompt words into the multimodal model; the oral data includes the original text and the corresponding scene picture; An output module, wherein the multimodal model outputs a mark text of a set output format, wherein the mark text includes a set mark; A segmentation module, which segments the identification text and generates corresponding audio files through a speech model; The generation module completes the generation if there is only one audio file; otherwise, the audio files are concatenated to complete the generation.

6. The device for generating personalized oral broadcast based on AI according to claim 5, characterized in that: The setting module is specifically: the prompt words include: setting output requirements and output format; the output format includes: Setting a first pause mark after each paragraph, wherein the first pause mark includes a first pause time, and the first pause mark is used to set the pause time after the paragraph; Setting a second pause mark after the word, wherein the second pause mark includes a second pause time, and the second pause mark is used to set the pause time after the word; Set an emotion mark before each paragraph to set the emotion used in the paragraph; A speech rate indicator is set before each sentence to set the speech rate used for the sentence; A volume mark is set before each sentence to set the volume used for the sentence; The spoken data and the set prompt words are input into the multimodal model; the spoken data includes original text and corresponding scene pictures, and the original text includes at least one paragraph, and each paragraph corresponds to a scene picture.

7. The device for generating personalized oral broadcast based on AI according to claim 5, characterized in that: The segmentation module specifically comprises: using regular matching to segment the identification text according to the emotion identification, the speech rate identification, the volume identification and the second pause identification to obtain segmented text, and generating corresponding audio files from the segmented text through the T2A speech model.

8. The device for generating personalized oral broadcast based on AI according to claim 5, characterized in that: The generation module is specifically as follows: if there is only one audio file, the generation is completed; if not, all audio files are spliced ​​in a set order by FFmpeg to complete the generation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images