Video explanation generation method and device

Abstract

The embodiment of the invention provides a video explanation generation method. The video explanation generation method comprises the following steps: acquiring audio and visual information of a target video; based on the audio and a pre-trained automatic speech recognition model, obtaining a recognition text; inputting the recognition text into a pre-trained language model to obtain a commentary text of the target video through the language model; and inputting the visual information and the explanation text into a pre-trained multi-modal model to obtain an explanation audio of the target video through the multi-modal model. According to the technical scheme, the automatic speech recognition model, the language model and the multi-modal model can be deeply integrated, the full-automatic process from the video to the high-quality explanation audio is achieved, the method is suitable for various complex explanation scenes, the features of different modals such as vision and texts can be effectively combined, and the intellectual property of the system is improved. The commentary audio which is more natural and smoother and highly matched with the video content is synthesized, and the audience experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating a video commentary. Background Art

[0002] Videos can convey information to viewers through images and sounds. In scenarios such as teaching, sports events, and news reports, the plot, background, or details of the video can also be supplemented to help viewers better understand the video content. Therefore, you can write commentary text for the video and convert it into commentary audio to enhance the expression effect of the video.

[0003] However, the current video commentary generation scheme still has the following defects: the commentary text is relatively stiff and stereotyped, requiring human intervention and difficult to adapt to complex commentary scenarios; it is impossible to intelligently synthesize speech based on the commentary text, resulting in poor naturalness of the commentary audio, affecting the audience experience.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a video commentary generation method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a method for generating a video description, the method comprising: Obtain audio and visual information of the target video; Based on the audio and a pre-trained automatic speech recognition model, obtaining a recognized text; Inputting the recognized text into a pre-trained language model to obtain a description text of the target video through the language model; The visual information and the commentary text are input into a pre-trained multimodal model to obtain the commentary audio of the target video through the multimodal model.

[0007] Optionally, obtaining recognized text based on the audio and a pre-trained automatic speech recognition model includes: Preprocessing the audio until the audio quality meets a preset standard; The preprocessed audio is input into a pre-trained automatic speech recognition model to obtain the recognition text through the automatic speech recognition model.

[0008] Optionally, the recognized text includes a plurality of text segments, each of which has a corresponding timestamp; Correspondingly, the language model obtains the commentary text of the target video through the following operations: Based on the timestamps corresponding to the multiple text segments, the recognized text is semantically analyzed to obtain semantic information; Performing topic modeling on the identified text to obtain topic information; Based on the semantic information and the subject information, the recognized text is processed to obtain an enhanced text; Based on the enhanced text and a preset narration style, a narration text of the target video is generated.

[0009] Optionally, based on the semantic information and the subject information, the recognized text is processed to obtain an enhanced text, including: Determine the key part, the non-key part and the deviation of each text segment according to the semantic information and the topic information; For each text segment, the non-critical part is deleted and the deviation is corrected to obtain the enhanced text.

[0010] Optionally, the target video includes a plurality of video frames, each video frame having a corresponding timestamp; Correspondingly, obtain the visual information of the target video, including: Get the picture content of each video frame; Based on the picture content of each video frame, obtaining a visual feature corresponding to each video frame, wherein the visual feature is associated with a timestamp of the corresponding video frame; The visual information of the target video is determined according to the visual features of the multiple video frames.

[0011] Optionally, the multimodal model obtains the narration audio of the target video by the following operations: Generating a target spectrogram according to the visual information and the explanation text; Generate a narration audio of the target video according to the target spectrogram.

[0012] Optionally, the explanation text includes a plurality of explanation segments, each of which has a corresponding timestamp and emotion tag; Correspondingly, generating a target spectrogram according to the visual information and the explanation text includes: Get the timestamp and sentiment label of each narration segment; Match the corresponding video frame according to the timestamp of each commentary segment to obtain the visual features corresponding to each commentary segment; Generating an initial spectrogram according to the multiple commentary segments and the corresponding emotion labels; Based on the visual features corresponding to the multiple commentary segments, the initial spectrogram is adjusted to obtain a target spectrogram.

[0013] Another aspect of an embodiment of the present application provides a video commentary generating device, the device comprising: An acquisition module, used to acquire audio and visual information of a target video; A recognition module, used to obtain recognition text based on the audio and a pre-trained automatic speech recognition model; A first generating module, used for inputting the recognition text into a pre-trained language model to obtain the explanation text of the target video through the language model; The second generating module is used to input the visual information and the commentary text into a pre-trained multimodal model to obtain the commentary audio of the target video through the multimodal model.

[0014] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0015] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0016] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0017] The above technical solution adopted in the embodiment of the present application may have the following advantages: The audio and visual information of the target video is obtained, and the audio is converted into recognized text using a pre-trained automatic language recognition model. The recognized text is deeply analyzed using a pre-trained language model and a narration text is generated. The video content and the narration text are deeply integrated using a pre-trained multimodal model to obtain the narration audio. It can be seen that the embodiment of the present application can realize a fully automated process from video to high-quality narration audio through the efficient integration of an automatic speech recognition model, a language model, and a multimodal model. Specifically, the audio can be accurately converted into recognized text through an automatic language recognition model. The recognized text can be more accurately analyzed and interpreted through a language model to generate high-quality narration text that can adapt to various complex narration scenarios. The multimodal model can effectively combine the features of different modalities (such as visual information and narration text) to synthesize a more natural and smooth narration audio that is highly matched with the video content, thereby enhancing the audience experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 A flowchart of a method for generating a video description according to Embodiment 1 of the present application is schematically shown; Figure 2 Schematically shows Figure 1 Flow chart of sub-steps of step S100; Figure 3 Schematically shows Figure 1 Flow chart of sub-steps of step S104; Figure 4 A flowchart of a method for generating a video description according to Embodiment 1 of the present application is schematically shown; Figure 5 Schematically shows Figure 3 Flow chart of sub-steps of step S304; Figure 6 Schematically shows Figure 1 Flow chart of sub-steps of step S106; Figure 7 Schematically shows Figure 6 Flow chart of sub-steps of step S600; Figure 8 A block diagram schematically shows a video description generating device according to the second embodiment of the present application; and Fig. 9 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0023] First, the following terms are explained: Modality: The form in which information is presented, such as text, visual, audio, etc.

[0024] ASR (Automatic Speech Recognition): Automatic speech recognition is a technology that converts speech into text and is widely used in voice assistants, subtitle generation, voice commands and other fields.

[0025] LLM (Large Language Model): A large language model that is trained with a large amount of data and can understand and generate natural language text. LLM can be used for natural language processing, dialogue systems, content generation, etc.

[0026] TTS (Text-to-Speech): Text to speech synthesis is a technology that converts text into speech output. It can generate high-quality artificial synthesized speech through a speech synthesis engine and is widely used in voice assistants, navigation systems and other fields.

[0027] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: Videos can convey information to viewers through images and sounds. In scenarios such as teaching, sports events, and news reports, the plot, background, or details of the video can also be supplemented to help viewers better understand the video content. Therefore, you can write commentary text for the video and convert it into commentary audio to enhance the expression effect of the video.

[0028] However, the applicant has learned that the related video description generation scheme still has the following defects: (1) Generate commentary text based on rules: For specific types of videos, such as sports events and news reports, commentary text is written based on pre-set rules, templates and keywords. Since the rules and templates are relatively fixed and lack flexibility and adaptability, it is difficult to accurately capture the complex content and emotional changes in videos of different styles and themes, resulting in the generated commentary text being relatively stiff and stereotyped.

[0029] (2) Generating commentary text based on manually written scripts or annotations: This requires a lot of manpower and has high update and maintenance costs. At the same time, due to the subjectivity of manual writing, different people may write commentary texts in different styles, resulting in inconsistent commentary quality and style, which is difficult to meet the needs of large-scale video commentary.

[0030] (3) The commentary audio with specific emotional tendencies synthesized based on the emotional keywords of the commentary text can only express relatively single emotions, and is unable to express the complex emotional changes in videos with rich plots and adapt to the needs of multi-emotional fusion. It lacks an effective fusion and comprehensive utilization mechanism for multimodal information, making it difficult to generate high-quality commentary audio.

[0031] To this end, the embodiment of the present application provides a technical solution for video commentary generation. In this technical solution: (1) an integrated system architecture is designed to organically combine ASR, LLM and TTS modules, and smooth data transmission and processing are achieved through standardized interfaces. The entire commentary audio generation process is automated through editing tools and automated scripts, from data input to voice output, without manual intervention. (2) By integrating ASR, LLM, and multimodal TTS technology, many problems in related video commentary generation solutions are effectively solved, adapting to various complex commentary scenarios, improving the quality and naturalness of commentary audio, and optimizing the viewing experience. Specifically, ASR can convert the audio carried by the video into (recognized) text, and LLM deep analysis is used to recognize the text to generate high-quality commentary text (commentary words). Based on the parameters such as emotion and speech rate obtained by LLM analysis, combined with the visual information of the video, multimodal TTS is guided to synthesize natural, emotional and expressive commentary audio that is highly matched with the video content. See below for details.

[0032] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0033] Embodiment 1 Figure 1 The flowchart of the video description generating method according to the first embodiment of the present application is schematically shown.

[0034] like Figure 1 As shown, the video description generation method may include steps S100 to S106, wherein: Step S100, obtaining audio and visual information of the target video.

[0035] Step S102, obtaining recognition text based on the audio and a pre-trained automatic speech recognition model.

[0036] Step S104: input the recognized text into a pre-trained language model to obtain the explanation text of the target video through the language model.

[0037] Step S106: input the visual information and the commentary text into a pre-trained multimodal model to obtain the commentary audio of the target video through the multimodal model.

[0038] The video commentary generation method provided in this embodiment obtains the audio and visual information of the target video, uses a pre-trained automatic language recognition model to convert the audio into recognized text, and uses a pre-trained language model to deeply analyze the recognized text and generate the commentary text. The pre-trained multimodal model is used to deeply fuse the video content and the commentary text to obtain the commentary audio. It can be seen that the embodiment of the present application can realize a fully automated process from video to high-quality commentary audio through the efficient integration of automatic speech recognition models, language models and multimodal models. Specifically, the audio can be accurately converted into recognized text through the automatic language recognition model. The language model can be used to more accurately analyze and interpret the recognized text, generate high-quality commentary text, and adapt to various complex commentary scenes. The multimodal model can effectively combine the features of different modalities (such as visual information and commentary text) to synthesize a more natural and smooth commentary audio that is highly matched with the video content, thereby enhancing the audience experience.

[0039] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0040] Step S100 , obtain the audio and visual information of the target video.

[0041] The target video can be any type of video, such as teaching videos, sports events, news reports, film and television works, etc. The target video can be a video file or a video stream. Through decapsulation technology, audio can be extracted from the video. Visual information can include visual features corresponding to each video frame of the target video. Visual features can be key information extracted from the corresponding video frame, such as picture atmosphere, color, etc. Visual information can be obtained through a variety of methods, such as image processing algorithms, deep learning networks, etc. An exemplary solution will be provided below.

[0042] In an optional embodiment, the target video includes multiple video frames, each of which has a corresponding timestamp. Figure 2 As shown, step S100 may include, Step S200, obtaining the picture content of each video frame.

[0043] Step S202: based on the picture content of each video frame, obtain the visual features corresponding to each video frame, wherein the visual features are associated with the timestamp of the corresponding video frame.

[0044] Step S204: determining visual information of the target video according to the visual features of the multiple video frames.

[0045] For example, the image content of each video frame can be identified through visual analysis, such as color, light, composition, intense movement, and character smiles. Based on the image content of each video frame, the visual features corresponding to each video frame can be further extracted. Among them, the visual features can be used to reflect the image atmosphere (or emotional tendency) of the corresponding video frame, and the visual features can be associated with the timestamp of the corresponding video frame. Based on the visual features of multiple video frames, the visual information of the target video can be determined.

[0046] In this embodiment, the corresponding visual features are determined based on the picture content of each video frame, so that the emotional atmosphere of the video can be fully and accurately grasped for subsequent multimodal TTS.

[0047] Step S102 , based on the audio and a pre-trained automatic speech recognition model, obtain the recognized text.

[0048] The automatic speech recognition model can be used for speech-to-text conversion. By inputting the audio into the automatic speech recognition model, the recognized text can be obtained. The recognized text corresponds to all the speech content in the target video, including lines, background dialogues, and other related audio. The recognized text may include multiple text fragments, each with a corresponding timestamp. In subsequent text processing, the text fragments can be associated with the corresponding content in the target video through the timestamp. The automatic speech recognition model can be driven by deep learning, such as Whisper, DeepSpeech, etc. Through a complex neural network architecture, high-precision conversion of audio can be achieved to improve the accuracy of recognizing text. In some embodiments, the automatic speech recognition model can also be accelerated by GPU to improve recognition efficiency. In order to further improve recognition efficiency and accuracy, the audio can also be appropriately processed according to actual conditions (such as model performance, etc.) to make the audio suitable for speech recognition. An exemplary solution is provided below.

[0049] In an optional embodiment, step S102 may include: preprocessing the audio until the audio quality meets a preset standard; inputting the preprocessed audio into a pre-trained automatic speech recognition model to obtain the recognized text through the automatic speech recognition model.

[0050] For example, after extracting the audio from the target video, the audio can be preprocessed, for example, noise reduction, volume adjustment, sampling rate conversion and other audio signal processing steps, until the audio quality reaches a preset standard. The preprocessed audio is input into the automatic speech recognition model to obtain the recognized text.

[0051] In this embodiment, preprocessing the audio can ensure that the audio quality is suitable for speech recognition and effectively optimize the speech recognition effect.

[0052] Step S104 , input the recognized text into a pre-trained language model to obtain the commentary text of the target video through the language model.

[0053] The language model can be a large language model LLM or a language model of other scales, which can be used to understand and generate natural language. The recognized text is input into the language model, and the recognized text can be intelligently analyzed and deeply understood through the language model. Combined with multi-level semantic understanding and text generation technology, high-quality commentary text can be generated in the end. The text processing process of the language model will be exemplified by multiple embodiments below.

[0054] In an optional embodiment, if Figure 3 As shown, step S104 may include: Step S300: Based on the timestamps corresponding to the multiple text segments, semantic analysis is performed on the recognized text to obtain semantic information.

[0055] Step S302, performing topic modeling on the recognized text to obtain topic information.

[0056] Step S304: Process the recognized text based on the semantic information and the subject information to obtain enhanced text.

[0057] Step S306: Generate a narration text for the target video based on the enhanced text and a preset narration style.

[0058] For example, LLM can perform semantic analysis on the recognized text to obtain semantic information. To ensure the accuracy of the narration text, semantic analysis can be performed based on the timing information of the recognized text (i.e., the timestamps corresponding to each of the multiple text segments) to ensure the accuracy of the semantic information and facilitate the association of the subsequently generated narration text with a specific time point in the video. Figure 4 As shown, through the semantic analysis of LLM, the context of the recognized text can be deeply understood, and the main content, key time, participating subjects (such as people and objects) and overall emotional tone (such as cheerfulness, tension, and seriousness) in the target video can be identified, so as to obtain comprehensive and comprehensive semantic information. In some embodiments, semantic analysis can also include the contextual relationship (such as causal relationship, time and space order) of the recognized text to further improve the accuracy of semantic information, thereby ensuring that the subsequent generated commentary text is logically clear and practical. LLM can also extract the core themes and sub-themes of the target video from the recognized text through subject modeling technology to obtain theme information. The theme information can be used to guide the generation of the commentary text, so that the commentary text can focus on the information that the audience needs to know most, and improve the quality of the commentary text. Based on semantic information and theme information, the recognized text can be sorted, corrected, de-redundant, etc. to obtain enhanced text. An exemplary processing scheme is provided below.

[0059] In an optional embodiment, if Figure 5 As shown, step S304 may include: Step S500, determining the key part, non-key part and deviation of each text segment according to the semantic information and the topic information.

[0060] Step S502: for each text segment, delete the non-critical part and correct the deviation to obtain the enhanced text.

[0061] Exemplarily, based on semantic information, the deviations of each text segment, such as vocabulary errors, incomplete sentences, etc., can be determined based on the preliminary understanding of the recognized text. Based on the subject information, the key information of each text segment can be determined, thereby determining the key and non-key parts of each text segment. Perform the following operations on each text segment: Delete the non-key parts to filter out redundant and unnecessary details. In this way, the core content that is helpful for the explanation can be retained first, and background noise or redundant dialogue that is not related to the topic can be removed, so that the subsequently generated explanation text is more concise and clear. Perform grammatical correction and logical combing on each text segment to correct grammatical problems caused by ASR recognition errors or incomplete sentences. Specifically, it can include: adjusting subject-verb consistency, correcting typos, and reconstructing incoherent sentences. After the above steps, enhanced text can be obtained.

[0062] In this embodiment, the recognized text is de-redundant and corrected based on semantic information and topic information, so that enhanced text that is actually helpful for explanation can be obtained, which can be used to generate high-quality explanation text later.

[0063] After obtaining the enhanced text, an explanation text can be generated based on a pre-set explanation style. The explanation text can include multiple explanation segments, each of which has a timestamp and an emotional label (such as positive, negative, neutral, cheerful, serious, etc.). Among them, the explanation style can be set according to the type of the target video to flexibly adjust the tone and style of the explanation text. For example, for educational videos, the model can generate formal and academic explanation texts. For entertainment videos, a more relaxed and humorous explanation text can be generated. The explanation style can also be adjusted by fine-tuning the model or prompts input by the user to generate explanation texts with diverse styles. Here are some example prompts: "Please generate a formal and academic explanation for this video about the laws of physics, highlighting the importance of the principles and application scenarios", "Write an explanation for this math teaching video in a serious and professional style, and explain complex concepts clearly and accurately", "Make the explanation of this funny pet video relaxed and humorous, full of fun, and make the audience laugh.

[0064] In this embodiment, the multi-level semantic understanding and text generation technology of the language model is used to perform intelligent analysis and deep understanding of the recognition text output by the ASR, so that an explanation text that conforms to the context and target audience can be generated to adapt to various complex explanation scenarios.

[0065] Step S106 , input the visual information and the commentary text into a pre-trained multimodal model to obtain the commentary audio of the target video through the multimodal model.

[0066] The multimodal model can synthesize speech by combining multiple speech features, such as intonation, speech speed, emotion, etc. For example, by synthesizing speech with emotion, context awareness, etc., the naturalness and expressiveness of the synthesized speech can be improved. By inputting visual information and narration text into the pre-trained multimodal model, the narration text can be converted into narration audio by combining multiple modalities (such as vision and text) to generate natural, fluent and emotional narration audio. The speech synthesis process of the multimodal model will be exemplified through multiple embodiments.

[0067] In an optional embodiment, if Figure 6 As shown, step S106 may include: Step S600, generating a target spectrogram according to the visual information and the explanatory text.

[0068] Step S602: Generate audio commentary of the target video according to the target spectrogram.

[0069] For example, a spectrogram can be generated based on the content of the commentary text, the overall emotional tone, etc., and the spectrogram can be fine-tuned using visual information to obtain a target spectrogram to make the speech more natural and consistent. The target spectrogram is then converted into a final speech waveform (i.e., the commentary audio).

[0070] In this embodiment, based on visual information, commentary text and multimodal models, not only can the commentary text be accurately converted into speech, but the emotional characteristics of the commentary text and the video content can also be deeply combined to generate commentary audio with emotional color and matching the video content, significantly improving the naturalness of the commentary audio and the viewing experience.

[0071] In an optional embodiment, if Figure 7 As shown, step S600 may include: Step S700, obtaining the timestamp and emotion label of each narration segment.

[0072] Step S702 , matching the corresponding video frame according to the timestamp of each commentary segment to obtain the visual features corresponding to each commentary segment.

[0073] Step S704: generating an initial spectrogram according to the multiple commentary segments and their corresponding emotion tags.

[0074] Step S706: adjusting the initial spectrogram based on the visual features corresponding to the multiple commentary segments to obtain a target spectrogram.

[0075] Exemplarily, multiple commentary segments of the commentary text and the emotional label of each commentary segment can be read to obtain the emotional tendency of each commentary segment. According to the content and emotional tendency of each commentary segment, suitable speech synthesis parameters can be determined for each commentary segment, including intonation, speaking speed, volume and emotional expression, etc., to convert each commentary segment into an initial spectrogram. For example: for a commentary segment with an emotional label of "cheerful", the intonation can be increased, the speaking speed can be increased, and more pitch changes can be added to increase the liveliness of the voice. For a commentary segment with an emotional label of "serious", the intonation and speaking speed can be reduced, the pitch fluctuation can be reduced, and a more stable voice can be presented. Further, the initial spectrogram can be adjusted in combination with visual information to achieve multimodal feature fusion. Specifically, the corresponding video frame can be matched according to the timestamp of each commentary segment to obtain the corresponding visual features. In combination with various corresponding visual features of multiple commentary segments, the initial spectrogram is adjusted to obtain a target spectrogram that highly matches the video content.

[0076] In this embodiment, based on the emotional tags of the commentary clips, audio with corresponding emotional characteristics can be generated, and the visual information is deeply combined with the video content to generate a natural, smooth, emotional and highly matched commentary audio. This speech synthesis technology that integrates multimodal information significantly improves the naturalness of the commentary audio and the user experience.

[0077] The embodiment of the present application deeply integrates ASR, LLM and multimodal TTS, and realizes efficient generation and high-quality output of commentary audio through multimodal information fusion and emotion-oriented speech synthesis. It not only significantly improves the production efficiency and quality of commentary audio, but also enhances the appeal of commentary and audience experience through emotional expression. The video commentary generation method provided in the embodiment of the present application can significantly reduce manual intervention, quickly generate natural and expressive commentary audio, greatly improve the efficiency and effect of video content creation, and can be used in education, media, entertainment and other fields.

[0078] In order to make this application easier to understand, the following Figure 4 An exemplary application is provided.

[0079] This application example will build a system that integrates the automatic speech recognition module (ASR), the large language model (LLM) processing module, and the multimodal speech synthesis (TTS) module to automatically extract voice information from the video and generate high-quality commentary audio with sentiment analysis. The following will introduce the generation process of the commentary audio in modules.

[0080] (1) Automatic Speech Recognition Module (ASR): S11: receiving an original video file or a video stream.

[0081] S12: Extract audio from video.

[0082] S13: Preprocess the audio, including noise reduction, volume adjustment, sampling rate conversion and other steps, to ensure that the audio quality is suitable for speech recognition.

[0083] S14: Use a deep learning-driven ASR model to convert the preprocessed audio into recognized text and preserve the timestamp in each text segment of the recognized text.

[0084] S15: Output the recognized text with timestamp and pass it to the large language model processing module.

[0085] (2) Large Language Model (LLM) processing module: S21: Perform semantic analysis and topic modeling on the recognized text to extract semantic information and topic information to fully understand the recognized text, thereby identifying the main content, key events, participating subjects and emotional tone in the video.

[0086] S22: Based on the semantic information and subject information, the recognized text is optimized (corrected and de-redundant) to obtain enhanced text.

[0087] S23: Based on the enhanced text and the explanation style set by the user, generate an explanation text that conforms to the explanation style. The explanation text includes multiple explanation segments, each of which has a timestamp and an emotion label.

[0088] S24: Output the commentary text and pass it to the multimodal language synthesis (TTS) module.

[0089] (3) Multimodal speech synthesis (TTS) module: S31: Generate an initial spectrogram based on the content and emotion label of each narration segment.

[0090] S32: Based on the visual information of the pre-acquired video, the initial spectrogram is adjusted to obtain a target spectrogram.

[0091] S33: Convert the target spectrogram into a final speech waveform, i.e., the commentary audio.

[0092] In this exemplary application: (1) An integrated system architecture is designed to organically combine ASR, LLM and TTS modules to achieve smooth data transmission and processing through standardized interfaces. The entire commentary audio generation process is automated through orchestration tools and automated scripts, from data input to voice output, without human intervention. (2) By integrating ASR, LLM and multimodal TTS technology, many problems in related video commentary generation solutions are effectively solved, adapting to various complex commentary scenarios, improving the quality and naturalness of commentary audio, and optimizing the viewing experience. Specifically, ASR can convert the audio carried by the video into (recognized) text, and LLM can be used to deeply analyze the recognized text to generate high-quality commentary text (commentary words). Based on the parameters such as emotion and speech rate obtained by LLM analysis, combined with the visual information of the video, multimodal TTS is guided to synthesize natural, emotional and expressive commentary audio that is highly matched with the video content.

[0093] Embodiment 2 Figure 8 The block diagram of the video explanation generating device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 8 As shown, the apparatus 1000 may include: an acquisition module 1100, an identification module 1200, a first generation module 1300 and a second generation module 1400, wherein: An acquisition module 1100 is used to acquire audio and visual information of a target video; A recognition module 1200, configured to obtain a recognition text based on the audio and a pre-trained automatic speech recognition model; A first generating module 1300 is used to input the recognition text into a pre-trained language model to obtain the explanation text of the target video through the language model; The second generating module 1400 is used to input the visual information and the commentary text into a pre-trained multimodal model to obtain the commentary audio of the target video through the multimodal model.

[0094] As an optional embodiment, obtaining the recognized text based on the audio and a pre-trained automatic speech recognition model includes: Preprocessing the audio until the audio quality meets a preset standard; The preprocessed audio is input into a pre-trained automatic speech recognition model to obtain the recognition text through the automatic speech recognition model.

[0095] As an optional embodiment, the recognized text includes a plurality of text segments, each of which has a corresponding timestamp; Correspondingly, the language model obtains the commentary text of the target video through the following operations: Based on the timestamps corresponding to the multiple text segments, the recognized text is semantically analyzed to obtain semantic information; Performing topic modeling on the identified text to obtain topic information; Based on the semantic information and the subject information, the recognized text is processed to obtain an enhanced text; Based on the enhanced text and a preset narration style, a narration text of the target video is generated.

[0096] As an optional embodiment, the recognized text is processed based on the semantic information and the subject information to obtain an enhanced text, including: Determine the key part, the non-key part and the deviation of each text segment according to the semantic information and the topic information; For each text segment, the non-critical part is deleted and the deviation is corrected to obtain the enhanced text.

[0097] As an optional embodiment, the target video includes a plurality of video frames, each video frame having a corresponding timestamp; Correspondingly, obtain the visual information of the target video, including: Get the picture content of each video frame; Based on the picture content of each video frame, obtaining a visual feature corresponding to each video frame, wherein the visual feature is associated with a timestamp of the corresponding video frame; The visual information of the target video is determined according to the visual features of the multiple video frames.

[0098] As an optional embodiment, the multimodal model obtains the narration audio of the target video through the following operations: Generating a target spectrogram according to the visual information and the explanation text; Generate a narration audio of the target video according to the target spectrogram.

[0099] As an optional embodiment, the explanation text includes a plurality of explanation segments, each of which has a corresponding timestamp and emotion tag; Correspondingly, generating a target spectrogram according to the visual information and the explanation text includes: Get the timestamp and sentiment label of each narration segment; Match the corresponding video frame according to the timestamp of each commentary segment to obtain the visual features corresponding to each commentary segment; Generating an initial spectrogram according to the multiple commentary segments and the corresponding emotion labels; Based on the visual features corresponding to the multiple commentary segments, the initial spectrogram is adjusted to obtain a target spectrogram.

[0100] Embodiment 3 Fig. 9 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the video commentary generation method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig. 9 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the video explanation generation method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0101] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0102] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0103] It should be pointed out that Fig. 9 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0104] In this embodiment, the video commentary generation method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0105] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the video commentary generating method in the embodiment are implemented.

[0106] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video explanation generation method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0107] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0108] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0109] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a video description, characterized in that: The method comprises: Obtain audio and visual information of the target video; Based on the audio and a pre-trained automatic speech recognition model, obtaining a recognized text; Inputting the recognized text into a pre-trained language model to obtain the explanation text of the target video through the language model; The visual information and the commentary text are input into a pre-trained multimodal model to obtain the commentary audio of the target video through the multimodal model.

2. The method according to claim 1, characterized in that Based on the audio and a pre-trained automatic speech recognition model, a recognition text is obtained, including: Preprocessing the audio until the audio quality meets a preset standard; The preprocessed audio is input into a pre-trained automatic speech recognition model to obtain the recognition text through the automatic speech recognition model.

3. The method according to claim 1, characterized in that The recognized text includes a plurality of text segments, each of which has a corresponding timestamp; Correspondingly, the language model obtains the commentary text of the target video through the following operations: Based on the timestamps corresponding to the multiple text segments, the recognized text is semantically analyzed to obtain semantic information; Performing topic modeling on the identified text to obtain topic information; Based on the semantic information and the subject information, the recognized text is processed to obtain an enhanced text; Based on the enhanced text and a preset narration style, a narration text of the target video is generated.

4. The method according to claim 3, characterized in that: Based on the semantic information and the subject information, the recognized text is processed to obtain an enhanced text, including: Determine the key part, the non-key part and the deviation of each text segment according to the semantic information and the topic information; For each text segment, the non-critical part is deleted and the deviation is corrected to obtain the enhanced text.

5. The method according to claim 1, characterized in that The target video includes a plurality of video frames, each of which has a corresponding timestamp; Correspondingly, obtain the visual information of the target video, including: Get the picture content of each video frame; Based on the picture content of each video frame, obtaining a visual feature corresponding to each video frame, wherein the visual feature is associated with a timestamp of the corresponding video frame; The visual information of the target video is determined according to the visual features of the multiple video frames.

6. The method according to claim 5, characterized in that The multimodal model obtains the narration audio of the target video through the following operations: Generating a target spectrogram according to the visual information and the explanation text; Generate a narration audio of the target video according to the target spectrogram.

7. The method according to claim 6, characterized in that The explanation text includes a plurality of explanation segments, each of which has a corresponding timestamp and emotion label; Correspondingly, generating a target spectrogram according to the visual information and the explanation text includes: Get the timestamp and sentiment label of each narration segment; Match the corresponding video frame according to the timestamp of each commentary segment to obtain the visual features corresponding to each commentary segment; Generating an initial spectrogram according to the multiple commentary segments and the corresponding emotion labels; Based on the visual features corresponding to the multiple commentary segments, the initial spectrogram is adjusted to obtain a target spectrogram.

8. A video description generating device, characterized in that: The device comprises: An acquisition module, used to acquire audio and visual information of a target video; A recognition module, used to obtain recognition text based on the audio and a pre-trained automatic speech recognition model; A first generating module, used for inputting the recognition text into a pre-trained language model to obtain the explanation text of the target video through the language model; The second generating module is used to input the visual information and the commentary text into a pre-trained multimodal model to obtain the commentary audio of the target video through the multimodal model.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images