Audio and video processing methods, apparatus, computer equipment, readable storage media and program products
By optimizing and rewriting speech recognition text and merging sentiment classification labels through a pre-trained text processing model, combined with variable speed processing, the problems of low efficiency and high cost in video content production in existing technologies are solved, and precise synchronization and automated processing of audio and video are achieved.
Patent Information
- Application Number
- CN202610194197.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-10
- Publication Date
- 2026-11-17
AI Technical Summary
Current technologies rely heavily on manual labor for video content production, resulting in low production efficiency and high costs. In particular, labor costs and production cycles become significant constraints when dealing with multi-disciplinary and multi-version courses.
The pre-trained text processing model optimizes and rewrites the speech recognition text and assigns it sentiment classification labels. Adjacent sentences with the same sentiment classification labels are merged into paragraphs, generating paragraph audio segments. The speed is then adjusted based on the speech recognition results to ensure accurate audio-visual synchronization.
It achieves more natural and coherent audio explanations, reduces audio-visual separation and stuttering after speed changes, improves the automation level of video processing, reduces labor costs, and adapts to the rapid production needs of different course versions.
Smart Images

Figure CN123123622A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multimedia data processing technology, and in particular to an audio and video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology
[0002] In current fields such as online education and content creation, video content production typically relies on a multi-stage post-production process. Taking the production of a typical educational video as an example, the process generally includes: recording the original video, transcribing the audio content using manual or automated speech recognition, manually refining and rewriting the transcribed text, regenerating the audio using voice actors or text-to-speech systems, manually editing and adjusting the video footage to match the newly generated audio, and finally synthesizing and outputting the final product.
[0003] This process suffers from low production efficiency and high costs, primarily due to its reliance on extensive manual labor. Specifically, video editing requires frame-by-frame adjustments to align with new audio, text needs manual modification to suit conversational language for reading aloud, and overall audio-visual synchronization requires meticulous manual proofreading. This process is time-consuming; for example, processing a 2-hour video often requires 3 to 4 hours or even more of manual processing time. As the scale of processing expands, such as in the case of multi-disciplinary and multi-version courses, the required labor costs and production cycle become significant constraints. Summary of the Invention
[0004] Therefore, it is necessary to provide an audio and video processing method, device, computer equipment, computer-readable storage medium, and computer program product that can achieve more natural and coherent audio narration and precise audio-visual synchronization, in response to the above-mentioned technical problems.
[0005] Firstly, this application provides an audio / video processing method, including:
[0006] In response to receiving a processing request for the target audio and video, the speech recognition text of the target audio and video and multiple video segments included in the target audio and video are obtained, wherein the multiple video segments include video segments of statements corresponding to each statement in the speech recognition text;
[0007] The speech recognition text is input into a pre-trained text processing model so that the text processing model outputs the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on the set rules.
[0008] The target sentences with the same sentiment classification label in the rewritten speech recognition text are merged to obtain the target paragraph, and the paragraph audio segment is generated based on the target paragraph and the corresponding sentiment classification label.
[0009] Obtain the speech recognition result of the audio segment, and based on the speech recognition result, segment the audio segment into multiple first statement audio segments;
[0010] Based on the non-target sentences other than the target sentence in the rewritten speech recognition text and their corresponding sentiment classification tags, a second sentence audio segment is generated.
[0011] Generate a target audio stream based on the audio segments of the first and second statements;
[0012] Based on the duration of the first and second statement audio segments, the corresponding statement video segments in the plurality of video segments are subjected to speed-changing processing to generate the target video stream;
[0013] Based on the target audio stream and the target video stream, the processed target audio and video are generated.
[0014] In one embodiment, the speech recognition result includes a timestamp; the step of segmenting the paragraph audio segment into multiple first statement audio segments based on the speech recognition result includes:
[0015] Based on the timestamp, the audio segment of the paragraph is divided into multiple first statement audio segments that correspond one-to-one with the target statement.
[0016] In one embodiment, the step of inputting the speech-recognized text into a pre-trained text processing model, so that the text processing model outputs a rewritten speech-recognized text and sentiment classification labels for each sentence in the rewritten speech-recognized text based on set rules, includes:
[0017] The speech recognition text is input into a pre-trained text processing model, which then calls the large language model service interface to send the processing instructions containing the set rules and the speech recognition text to the large language model. The large language model returns the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text, and outputs them. The set rules include scene adaptation rules and sentiment classification dimensions.
[0018] In one embodiment, the step of speed-changing the corresponding statement video segments among the plurality of video segments based on the duration of the first and second statement audio segments to generate the target video stream includes:
[0019] The multiple video segments are traversed, and the duration ratio of the current statement video segment to the corresponding first statement audio segment or second statement audio segment is calculated. The current statement video segment is then subjected to speed-changing processing based on the duration ratio.
[0020] The target video stream is generated based on all the video segments that have undergone speed-changing processing.
[0021] In one embodiment, after generating the processed target audio and video based on the target audio stream and the target video stream, the process includes:
[0022] Based on the rewritten speech recognition text or the target audio stream, drive the digital human model to generate lip-sync animation of the digital human.
[0023] Based on the emotion classification labels, the digital human model is driven to generate facial expression animations of the digital human;
[0024] The digital human is superimposed onto the target area in the target audio-visual background.
[0025] In one embodiment, the process of merging adjacent target sentences with the same sentiment classification label in the rewritten speech recognition text to obtain a target paragraph, and generating a paragraph audio segment based on the target paragraph and its corresponding sentiment classification label, includes the following steps:
[0026] Export the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text to a table file;
[0027] In response to receiving the updated table file, the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text are updated based on the updated table file.
[0028] Secondly, this application also provides an audio / video processing apparatus, comprising:
[0029] The request response module is used to respond to a received processing request for the target audio and video, and to obtain the speech recognition text of the target audio and video and multiple video segments included in the target audio and video, wherein the multiple video segments include video segments of sentences corresponding to each sentence in the speech recognition text;
[0030] The model processing module is used to input the speech recognition text into a pre-trained text processing model, so that the text processing model outputs the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on the set rules.
[0031] The audio generation module is used to merge adjacent target sentences with the same sentiment classification label in the rewritten speech recognition text to obtain target paragraphs, and generate paragraph audio segments based on the target paragraphs and their corresponding sentiment classification labels.
[0032] An audio segmentation module is used to obtain the speech recognition result of the audio segment of the paragraph, and based on the speech recognition result, segment the audio segment of the paragraph into multiple first sentence audio segments;
[0033] The audio generation module is also used to generate a second sentence audio segment based on the non-target sentences other than the target sentence in the rewritten speech recognition text and the corresponding sentiment classification tags;
[0034] An audio stream generation module is used to generate a target audio stream based on the audio segments of the first statement and the second statement;
[0035] The video stream generation module is used to perform speed-changing processing on the corresponding statement video segments among the plurality of video segments based on the duration of the first statement audio segment and the second statement audio segment, and generate a target video stream.
[0036] The synthesis module is used to generate processed target audio and video based on the target audio stream and the target video stream.
[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above-mentioned embodiments.
[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.
[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above claims.
[0040] The aforementioned audio and video processing methods, devices, computer equipment, computer-readable storage media, and computer program products optimize and rewrite speech recognition text using a pre-trained text processing model and assign emotion classification labels. This enables the adaptation of audio generation to text semantics and emotion, making the rewritten text conform to the expression habits of the target scene, and the audio explanation more natural and coherent. Furthermore, by merging adjacent target sentences with the same emotion classification labels into target paragraphs and generating paragraph audio segments, the emotional coherence of the speech can be preserved to the greatest extent. Then, by performing speech recognition again on the paragraph audio segments, accurate internal timestamps are obtained and the first sentence audio segments are segmented. At the same time, second sentence audio segments are generated for non-target sentences, ensuring that all sentences have corresponding accurate audio segments. By combining the duration of each audio segment with the corresponding sentence video segments for speed adjustment, the accurate synchronization of audio and video after speed adjustment can be further guaranteed, reducing the problems of audio-visual separation and stuttering after speed adjustment. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is an application environment diagram of an audio / video processing method in one embodiment;
[0043] Figure 2 This is a flowchart illustrating an audio / video processing method in one embodiment;
[0044] Figure 3 This is a flowchart illustrating step S207 in one embodiment;
[0045] Figure 4 This is a flowchart illustrating the audio and video processing method in another embodiment;
[0046] Figure 5 This is a flowchart illustrating the audio and video processing method in yet another embodiment;
[0047] Figure 6 This is a flowchart illustrating the audio and video processing method in another embodiment;
[0048] Figure 7 This is a structural block diagram of an audio / video processing device in one embodiment;
[0049] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0051] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0052] The audio and video processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0053] In one exemplary embodiment, such as Figure 2 As shown, an audio and video processing method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S201 to S206. Wherein:
[0054] S201, in response to receiving a processing request for the target audio and video, obtain the speech recognition text of the target audio and video and multiple video segments included in the target audio and video, wherein the multiple video segments include video segments of sentences corresponding to each sentence in the speech recognition text.
[0055] The target audio / video file can be the original audio / video file to be processed, including video frame data and the corresponding audio data stream. In one possible implementation, the target audio / video file can originate from online education course recordings, content creation materials, etc., and the format can be mainstream audio / video formats such as MP4, AVI, and MOV. The speech recognition text of the target audio / video file can be text data obtained by converting the audio data stream of the target audio / video file using Automatic Speech Recognition (ASR) technology. It contains text content that corresponds one-to-one with each sentence in the audio, and includes the start and end timestamps of each sentence in the audio.
[0056] For example, the ASR module can be invoked to submit the audio data stream of the target audio and video through the speech recognition service interface, poll and parse the recognition results, extract the text content of each sentence and the corresponding start and end timestamps to form speech recognition text; based on the timestamps, the target audio and video can be physically divided into multiple independent sentence video segments by calling an external command-line tool (such as FFmpeg), stored in a temporary directory, and the correspondence between each segment and the sentences in the speech recognition text can be preserved.
[0057] S202, input the speech recognition text into the pre-trained text processing model so that the text processing model outputs the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on the set rules.
[0058] The text processing model can be implemented based on a large language model. The rules can be associated with the target scenario or content type, audience characteristics, and expression style requirements. In one possible implementation, the text processing model can parse the metadata of the target audio / video (such as content types like math teaching and workplace skills training, and audience types like students and new employees) and match the corresponding rules from a mapping library.
[0059] S203, merge adjacent target sentences with the same sentiment classification label in the rewritten speech recognition text to obtain target paragraphs, and generate paragraph audio segments based on the target paragraphs and their corresponding sentiment classification labels.
[0060] Understandably, all sentences within the same target paragraph should have the same sentiment classification label. Based on the text content and corresponding sentiment classification labels of the target paragraph, a multi-sensory text-to-speech service can be invoked to generate audio segments of the paragraph, whose emotional tone matches the sentiment classification label of that paragraph.
[0061] For example, all sentences in the rewritten speech recognition text can be traversed, and the sentiment classification labels of adjacent sentences can be detected in the order of sentences. Continuous sentences with the same label can be merged into a target paragraph. The text processing model maintains the mapping relationship between sentiment classification labels and TTS (Text-to-Speech) timbre labels. Based on the sentiment classification label of the target paragraph, the matching timbre is selected, and the target paragraph text is submitted by calling the TTS service interface to generate a continuous paragraph audio segment, ensuring that the audio sentiment is consistent with the paragraph content.
[0062] S204, obtain the speech recognition result of the audio segment, and based on the speech recognition result, divide the audio segment into multiple first sentence audio segments.
[0063] For example, the speech recognition results of the audio segment can be obtained, which include the text content of each sentence, part-of-speech tagging, and pause interval features; based on the text content in the speech recognition results, semantic matching is performed with the target sentences of the target segment to determine the corresponding text segment of each target sentence in the audio segment; combining part-of-speech tagging (such as punctuation marks and the position of modal particles) and pause interval features, the sentence boundaries after semantic matching are corrected to form the audio interval corresponding to each target sentence; an audio processing tool is called to segment the audio segment of the segment according to the corrected audio interval to generate the first sentence audio segment that corresponds one-to-one with the target sentence.
[0064] S205, based on the non-target sentences other than the target sentence in the rewritten speech recognition text and their corresponding sentiment classification labels, generate the second sentence audio segment.
[0065] It is understandable that there are no adjacent sentences with the same sentiment classification label among the non-target sentences.
[0066] For example, non-target sentences that are not merged into the target paragraph in the rewritten speech recognition text can be filtered out. Based on the sentiment classification label corresponding to each non-target sentence, the matching timbre is selected through the TTS timbre mapping relationship of the text processing model. The TTS service interface is called to generate independent second sentence audio segments for each non-target sentence, ensuring that the emotional tone of each segment is consistent with the sentiment classification label of the corresponding sentence.
[0067] S206, Generate the target audio stream based on the audio segments of the first and second statements.
[0068] For example, the corresponding first or second statement audio segments can be retrieved sequentially according to the original order of each statement in the rewritten speech recognition text, and all segments can be connected in sequence without loss using an audio splicing tool. Before this, an audio post-processing tool (such as Sox) can be called to normalize the volume of all first and second statement audio segments to ensure that the volume of each segment is uniform and the listening experience is coherent, ultimately forming a complete target audio stream.
[0069] S207, based on the duration of the first and second statement audio segments, the corresponding statement video segments in multiple video segments are subjected to speed-changing processing to generate the target video stream.
[0070] For example, a mapping relationship between video segments and corresponding audio segments can be established first, and their respective durations recorded; the duration difference between audio and video can be calculated, and smooth transition frames can be inserted when the video needs to be extended, and redundant and invalid frames can be removed when it needs to be shortened; then all segments after speed change can be calibrated according to the original audio and video timing to ensure that there are no misalignments between segments; finally, all calibrated segments can be spliced together in order to generate the target video stream.
[0071] S208 generates processed target audio and video based on the target audio stream and the target video stream.
[0072] For example, a video compositing tool (such as FFmpeg) can be called to synchronously combine the target audio stream and the target video stream to ensure accurate matching of audio and video. During the compositing process, a cleanup connection strategy can be used to eliminate video stuttering and audio gaps at the splicing points, and finally output a complete processed target audio and video file.
[0073] In the aforementioned audio and video processing method, the pre-trained text processing model optimizes and rewrites the speech recognition text and assigns emotion classification labels, enabling the audio generation to adapt to the semantics and emotion of the text. This makes the rewritten text conform to the expression habits of the target scenario (such as a one-on-one teaching scenario), and the audio explanation is more natural and coherent. Furthermore, by merging adjacent target sentences with the same emotion classification labels into target paragraphs and generating paragraph audio segments, the emotional coherence of the speech can be preserved to the greatest extent. Then, by performing speech recognition again on the paragraph audio segments (the original text is obtained from a single speech recognition), accurate internal timestamps are obtained and the first sentence audio segments are segmented. At the same time, second sentence audio segments are generated for non-target sentences, ensuring that all sentences have corresponding accurate audio segments. By combining the duration of each audio segment with the corresponding sentence video segments for speed adjustment, the accurate synchronization of audio and video after speed adjustment can be further guaranteed, reducing the problems of audio-visual separation and stuttering after speed adjustment.
[0074] In an exemplary embodiment, the speech recognition result may include a timestamp, and the step of dividing the paragraph audio segment into multiple first statement audio segments based on the speech recognition result may include: dividing the paragraph audio segment into multiple first statement audio segments that correspond one-to-one with the target statement based on the timestamp.
[0075] It is understandable that the speech recognition result of the audio segment is a secondary speech recognition result. The audio segment is a continuous audio generated based on the unified emotion classification label, without the environmental noise interference of the original audio, and the speech intonation is coherent and consistent. The ASR module can capture the sentence boundary more accurately, including the more accurate timestamps of each sentence corresponding to the audio segment.
[0076] For example, the audio segment can be submitted to cloud storage through the data upload unit, and the ASR module can be called again to perform speech recognition on the audio segment to obtain the precise timestamp of each sentence in the audio segment (i.e., the secondary ASR recognition result). Based on the precise timestamp, an external audio processing tool (such as FFmpeg) can be called to accurately divide the audio segment into multiple independent first sentence audio segments, and each first sentence audio segment corresponds one-to-one with a target sentence in the target segment.
[0077] In this embodiment, by using the timestamps in the secondary speech recognition results to segment audio segments, it is possible to achieve a precise one-to-one correspondence between the first statement audio segment and the target statement, thereby improving the accuracy of duration matching and speed synchronization of subsequent audio and video segments, and further enhancing the accuracy of audio-visual synchronization.
[0078] In an exemplary embodiment, step S202 may include: inputting the speech recognition text into a pre-trained text processing model, causing the text processing model to call the large language model service interface, sending the processing instructions containing the set rules and the speech recognition text to the large language model, receiving the rewritten speech recognition text returned by the large language model and the sentiment classification labels of each sentence in the rewritten speech recognition text and outputting them; wherein, the set rules include scene adaptation rules and sentiment classification dimensions.
[0079] The text processing model can be used to semantically optimize the original speech recognition text based on scene adaptation rules, solving problems such as scene mismatch and semantic incoherence in the original speech recognition text. The text processing model can also be used to determine the intent and sentiment of each sentence in the rewritten text based on the sentiment classification dimension, outputting a sentiment classification label, with each sentence corresponding to a unique sentiment classification label.
[0080] For example, the text processing model can construct processing instructions containing scene adaptation rules and sentiment classification dimensions through the large language model service interface, and send the speech recognition text to the large language model service in batches; the large language model performs semantic understanding on the original text based on the processing instructions, completes the rewriting of the speech between scenes (such as from a large class to a one-on-one teaching scene), and performs sentiment judgment on each sentence and assigns a classification label (such as teaching, encouragement, emphasis). The text processing model parses the returned structured data and outputs the rewritten speech recognition text and the corresponding sentiment classification label.
[0081] Optionally, the rules can also include rules for content simplification, style consistency, or audience adaptation.
[0082] In this embodiment, by calling the large language model service interface and sending the processing instructions, which include scene adaptation rules and sentiment classification dimensions, along with the speech recognition text to the large language model, it is possible to achieve accurate adaptation of text rewriting to the target scene, while ensuring the standardization and consistency of sentiment classification, and improving the reliability and adaptability of text processing results.
[0083] In one exemplary embodiment, such as Figure 3 As shown, step S207 above may include:
[0084] S2071, traverse multiple video segments, calculate the duration ratio of the current statement video segment to the corresponding first statement audio segment or second statement audio segment, and perform speed-changing processing on the current statement video segment based on the duration ratio.
[0085] S2072 generates the target video stream based on all the video segments that have undergone speed-changing processing.
[0086] For example, each statement video segment can be traversed to find its corresponding first or second statement audio segment and calculate the duration ratio between the two. By calling an external video processing tool (such as FFmpeg's setpts filter), the statement video segments can be precisely speed-changed according to the duration ratio to ensure that the duration of the speed-changed video segments is completely consistent with the duration of the corresponding audio segments. All speed-changed statement video segments are then losslessly spliced together in the original target audio and video frame order to form the target video stream. The video data is not loaded into the main program memory to avoid memory overflow.
[0087] In one possible implementation, all video-related operations are performed by calling external, independent command-line tools. These operations include video segmentation, speed adjustment, and splicing. Video data is not directly loaded into the main program's memory; instead, video files are stored in a temporary directory during processing, and these temporary files are cleaned up after processing, achieving low-memory video processing.
[0088] In this embodiment, by traversing video segments and calculating the duration ratio of the current sentence video segment to the corresponding audio segment, targeted speed adjustment is performed based on this ratio. This enables precise audio-visual adaptation at the sentence level, avoids local asynchrony problems caused by speed deviation, and further improves the synchronization consistency between the target video stream and the audio stream.
[0089] In one exemplary embodiment, such as Figure 4 As shown, the above audio and video processing method may further include:
[0090] S209, based on the rewritten speech recognition text or target audio stream, drives the digital human model to generate lip-sync animation of the digital human.
[0091] For example, the rewritten speech recognition text, the first sentence audio segment, the second sentence audio segment, the sentiment classification tags corresponding to each sentence, and the precise timestamp information can be integrated to form a complete resource package. The rewritten speech recognition text and the corresponding precise timestamps can be extracted from the previously generated complete asset package, and the rewritten speech recognition text can be transmitted to the digital human rendering engine in segments according to the timestamp order. The digital human rendering engine can then decompose each segment of text into pronunciations, generate lip-syncing instructions that match the rhythm of the syllables, and drive the digital human model to generate synchronized lip-syncing animations.
[0092] S210, based on emotion classification tags, drives the digital human model to generate digital human facial expression animations.
[0093] For example, the sentiment classification tags of each statement in the complete asset package can be extracted; the digital human rendering engine can match the facial muscle movement parameters (including the movement amplitude and rate of eyebrows, eyes, facial contours, etc.) corresponding to each tag based on the preset mapping table of sentiment tags and expression parameters, and drive the digital human model to generate facial expression animations consistent with the sentiment tags, and the start and stop times of the facial expression animations are aligned with the audio duration of the corresponding statement.
[0094] S211, overlays a digital human onto a target area within a target audio / video background.
[0095] For example, a digital human motion video with a transparent background can be output using a digital human rendering engine. An external video processing tool (such as FFmpeg) can then be used to overlay the digital human motion video onto a preset target area of the processed target audio-visual content. The coordinates and size of the preset target area can be pre-defined using a configuration file, and the timing of the digital human video and the target audio-visual content is kept synchronized during the overlay process.
[0096] In this embodiment, the virtual digital human is rendered by generating rewritten speech recognition text, target audio stream, and emotion classification tags. This enables precise synchronization of lip movements and speech, as well as matching of facial expressions with the emotions in the content. This solves the problem of the limited expressiveness of digital humans and enhances the interactivity and appeal of the processed audio and video.
[0097] In one exemplary embodiment, such as Figure 5 As shown, the above audio and video processing method may further include:
[0098] S212, export the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text to a table file.
[0099] For example, the rewritten speech recognition text and the corresponding sentiment classification labels for each sentence can be exported to a spreadsheet file (such as Excel format) in a preset format. The columns and fields of the spreadsheet file include, but are not limited to, sentence number, original speech recognition text, rewritten speech recognition text, sentiment classification labels, timestamp (start time / end time), and manual modification notes.
[0100] Optionally, the table file is used to output the text content and sentiment classification tags after manual review and optimization.
[0101] S213, in response to receiving the updated table file, update the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on the updated table file.
[0102] For example, after exporting the table file, it can wait for manual review and optimization; in response to receiving the updated table file uploaded after manual modification, the table file is read, and the differences between the original exported data and the updated data are compared one by one according to the statement number. Based on the rewritten speech recognition text column and sentiment classification label column in the updated table file, the corresponding stored data is updated synchronously. After the update is completed, the steps of merging adjacent target statements with the same sentiment classification label in the rewritten speech recognition text to obtain the target paragraph are continued, and the paragraph audio segment is generated based on the target paragraph and the corresponding sentiment classification label.
[0103] In this embodiment, by exporting the rewritten speech recognition text and sentiment classification labels to a table file for manual optimization, and updating the relevant data based on the updated table file, human-computer collaborative calibration of the text processing results can be achieved, improving the accuracy and adaptability of the text content and sentiment labels, and avoiding deviations caused by automatic model processing.
[0104] In one exemplary embodiment, an audio / video processing system is provided, such as... Figure 6 As shown, it includes an ASR module, an LLM text processing module, a multi-emotion TTS synthesis module, an audio-visual synchronization processing module, a video synthesis module, and a virtual digital human synthesis module, among which:
[0105] The ASR module is used to respond to the received processing request of the target audio and video, perform automatic speech recognition on the audio data stream of the target audio and video, extract the speech recognition text, which contains text content corresponding to each sentence in the audio and the corresponding start and end timestamps; and based on the timestamps, call the external command line tool to segment the target audio and video into sentence video segments corresponding to each sentence, and store them in a temporary directory.
[0106] The LLM text processing module receives the speech recognition text output by the ASR module. Based on preset rules (including scene adaptation rules and sentiment classification dimensions), it calls the large language model service interface to semantically optimize and rewrite the original text, outputting the rewritten speech recognition text. Simultaneously, it performs sentiment determination on each sentence in the rewritten text and outputs sentiment classification labels corresponding to each sentence. In addition, this module also supports exporting the rewritten speech recognition text and sentiment classification labels to a spreadsheet file, and in response to receiving an updated spreadsheet file, it synchronously updates the corresponding data stored in the system.
[0107] The multi-emotion TTS synthesis module receives the rewritten speech recognition text and emotion classification tags output by the LLM text processing module. It merges adjacent target sentences with the same emotion classification tags into target paragraphs. Based on the text content of the target paragraphs and their corresponding emotion classification tags, it calls the multi-emotion TTS service to generate paragraph audio segments. Simultaneously, it generates independent second sentence audio segments based on non-target sentences and their corresponding emotion classification tags. It performs secondary ASR recognition on the paragraph audio segments to obtain accurate timestamps and divides the paragraph audio segments into first sentence audio segments that correspond one-to-one with the target sentences. It performs volume normalization processing on all first sentence audio segments and second sentence audio segments and concatenates them in the order of sentences to generate the target audio stream.
[0108] The audio-visual synchronization processing module is used to establish a mapping relationship between the video segments of sentences and the corresponding first / second audio segments of sentences, calculate the duration ratio between the two, and call external tools to speed up the video segments of sentences based on the duration ratio to ensure that the duration of the speed-up video segments is completely consistent with the duration of the corresponding audio segments. Then, a two-step clean connection strategy is used to splice all the speed-up video segments of sentences to generate the target video stream. Finally, the target audio stream and the target video stream are synchronously synthesized to output the processed target audio and video.
[0109] The video compositing module receives the processed target audio and video output from the audio-visual synchronization processing module, combines it with the digital human dynamic video output from the virtual digital human compositing module, and calls external tools to overlay the digital human video onto a preset target area (such as the upper right corner of the screen) of the target audio and video, and outputs a video file with the overlaid digital human.
[0110] The virtual digital human synthesis module receives the rewritten speech recognition text, emotion classification labels, and precise timestamps output by the LLM text processing module. Based on the rewritten speech recognition text or target audio stream, it drives the digital human model to generate lip-sync animation; based on the emotion classification labels, it drives the digital human model to generate facial expression animation; and outputs a dynamic video of the digital human with a transparent background.
[0111] In one possible implementation, the above audio and video processing method may include: the audio and video processing system receives the raw audio and video file to be processed (formatted as MP4 / AVI / MOV, etc.) and triggers a processing request. The ASR module recognizes the audio data stream of the raw audio and video, generating speech-recognized text with timestamps; based on the timestamps, it calls FFmpeg to segment the raw audio and video into sentence video segments and stores them in a temporary directory. The LLM text processing module inputs the speech-recognized text into a large language model, completes text rewriting based on scene adaptation rules (such as from large class to one-on-one teaching scenarios), and simultaneously labels each sentence with emotion classification dimensions (such as teaching, encouragement, emphasis); it supports exporting the results to an Excel spreadsheet for manual review, receiving the updated spreadsheet and synchronizing the data. The multi-emotion TTS synthesis module merges adjacent sentences with the same emotion tags into target paragraphs, calls the TTS service to generate paragraph audio segments; performs secondary ASR recognition on the paragraph audio segments to obtain accurate timestamps, and segments them into first sentence audio segments; simultaneously generates second sentence audio segments for non-target sentences; performs volume normalization processing on all audio segments, and splices them to generate the target audio stream. The audio-visual synchronization processing module calculates the duration ratio of the video segment to the corresponding audio segment, and uses FFmpeg's setpts filter to speed up the video segment. A two-step cleanup and concatenation strategy is used to stitch the speed-up video segments together to generate the target video stream. The target audio stream is then combined with the target video stream to output the processed target audio-visual content. Simultaneously, a complete asset package containing rewritten text, audio segments, emotion tags, and timestamps is generated. The virtual digital human synthesis module, based on the data in the asset package, drives the digital human model to generate lip-sync and facial animations, outputting a dynamic video of the digital human with a transparent background. The video synthesis module overlays the dynamic video of the digital human onto a preset area of the processed target audio-visual content, outputting the final audio-visual file. After processing, the system automatically cleans up files in all temporary directories to release memory.
[0112] In summary, the aforementioned audio and video processing methods and systems can improve the automation level of video processing, shortening the traditional manual video re-creation process that requires hours or even tens of hours to minutes, reducing labor costs, enabling large-scale production of video content, and quickly responding to the video production needs of different course versions and teaching scenarios. By optimizing and rewriting the speech recognition text through a pre-trained text processing model and assigning emotion classification labels, the audio generation can be adapted to the semantics and emotion of the text, making the rewritten text conform to the expression habits of the target scenario (such as one-on-one teaching scenarios), and the audio explanation more natural and coherent. Furthermore, by merging adjacent target sentences with the same emotion classification labels into target paragraphs and generating paragraph audio segments, the emotional coherence of the speech can be preserved to the greatest extent. Then, by performing speech recognition again on the paragraph audio segments (the original text is obtained from a first speech recognition), accurate internal timestamps are obtained and the first sentence audio segments are segmented. At the same time, second sentence audio segments are generated for non-target sentences, ensuring that all sentences have corresponding accurate audio segments. Combining the various audio segments The duration of each segment is adjusted to speed up the corresponding video clip, ensuring precise synchronization between audio and video after speed adjustment, reducing audio-visual separation and stuttering issues, and achieving accurate matching of lip movements and voice. By integrating the audio clips of the first and second sentences to form a target audio stream, the auditory jump caused by mixing the original recording and the newly generated audio is avoided. Combined with audio normalization processing, the smoothness and consistency of the overall audio experience are guaranteed. Audio clips generated based on emotion classification tags are rich in emotional variation, getting rid of monotonous machine voice output and improving the expressiveness of the content. By generating rewritten speech recognition text, target audio stream, and emotion classification tags, the rendering of virtual digital humans is driven, achieving precise synchronization between digital human lip movements and voice, and matching of facial expressions with the content's emotions. This solves the problem of monotonous digital human expressiveness and enhances the interactivity and appeal of the processed audio and video. In addition, the entire processing flow does not rely on large-scale memory loading of video data, can stably process ultra-long videos, and the implementation of each functional module is flexible, making it easy to replace or add related services and tools, improving maintainability and scalability.
[0113] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0114] Based on the same inventive concept, this application also provides an audio and video processing apparatus for implementing the audio and video processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more audio and video processing apparatus embodiments provided below can be found in the limitations of the audio and video processing method described above, and will not be repeated here.
[0115] In one exemplary embodiment, such as Figure 7 As shown, an audio and video processing device 300 is provided, including: a request and response module 301, a model processing module 302, an audio generation module 303, an audio segmentation module 304, an audio stream generation module 305, a video stream generation module 306, and a synthesis module 307, wherein:
[0116] The request response module 301 is used to respond to the received processing request for the target audio and video, and to obtain the speech recognition text of the target audio and video and multiple video segments included in the target audio and video, wherein the multiple video segments include video segments of sentences corresponding to each sentence in the speech recognition text.
[0117] The model processing module 302 is used to input the speech recognition text into a pre-trained text processing model, so that the text processing model outputs the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on the set rules.
[0118] The audio generation module 303 is used to merge adjacent target sentences with the same sentiment classification label in the rewritten speech recognition text to obtain target paragraphs, and generate paragraph audio segments based on the target paragraphs and their corresponding sentiment classification labels.
[0119] The audio segmentation module 304 is used to obtain the speech recognition results of the audio segment and, based on the speech recognition results, segment the audio segment into multiple first-statement audio segments.
[0120] The aforementioned audio generation module 303 is also used to generate a second sentence audio segment based on the non-target sentences other than the target sentence and their corresponding sentiment classification labels in the rewritten speech recognition text.
[0121] The audio stream generation module 305 is used to generate a target audio stream based on the first statement audio segment and the second statement audio segment.
[0122] The video stream generation module 306 is used to perform speed-changing processing on the corresponding statement video segments in multiple video segments based on the duration of the first statement audio segment and the second statement audio segment, and generate the target video stream.
[0123] The synthesis module 307 is used to generate processed target audio and video based on the target audio stream and the target video stream.
[0124] In one embodiment, the speech recognition result includes a timestamp, and the audio segmentation module 304 is further configured to:
[0125] Based on timestamps, the audio segments are divided into multiple first-statement audio segments that correspond one-to-one with the target statement.
[0126] In one embodiment, the model processing module 302 is further configured to:
[0127] The speech recognition text is input into a pre-trained text processing model, which then calls the large language model service interface to send the processing instructions containing the set rules and the speech recognition text to the large language model. The large language model returns the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text, and outputs them. The set rules include scene adaptation rules and sentiment classification dimensions.
[0128] In one embodiment, the video stream generation module 306 is further configured to:
[0129] Iterate through multiple video segments, calculate the duration ratio of the current statement video segment to the corresponding first statement audio segment or second statement audio segment, and perform speed adjustment on the current statement video segment based on the duration ratio.
[0130] The target video stream is generated based on all the video segments that have undergone speed-changing processing.
[0131] In one embodiment, the audio / video processing apparatus 300 further includes a digital human synthesis module, used for:
[0132] Based on the rewritten speech recognition text or target audio stream, drive the digital human model to generate lip-sync animation of the digital human;
[0133] Based on emotion classification tags, the digital human model is driven to generate digital human facial animations.
[0134] Overlay a digital human onto a target area within the target audio / video background.
[0135] In one embodiment, the audio / video processing apparatus 300 further includes a verification module, used for:
[0136] Export the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text to a spreadsheet file;
[0137] In response to receiving the updated table file, the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text are updated based on the updated table file.
[0138] Each module in the aforementioned audio and video processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0139] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements an audio / video processing method.
[0140] Those skilled in the art will understand that Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0141] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0142] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.
[0143] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. An audio-video processing method, characterized in that, The method includes: In response to receiving a processing request for the target audio and video, the speech recognition text of the target audio and video and multiple video segments included in the target audio and video are obtained, wherein the multiple video segments include video segments of statements corresponding to each statement in the speech recognition text; The speech recognition text is input into a pre-trained text processing model so that the text processing model outputs the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on the set rules. The target sentences with the same sentiment classification label in the rewritten speech recognition text are merged to obtain the target paragraph, and the paragraph audio segment is generated based on the target paragraph and the corresponding sentiment classification label. Obtain the speech recognition result of the audio segment, and based on the speech recognition result, segment the audio segment into multiple first statement audio segments; Based on the non-target sentences other than the target sentence in the rewritten speech recognition text and their corresponding sentiment classification tags, a second sentence audio segment is generated. Generate a target audio stream based on the audio segments of the first and second statements; Based on the duration of the first and second statement audio segments, the corresponding statement video segments in the plurality of video segments are subjected to speed-changing processing to generate the target video stream; Based on the target audio stream and the target video stream, the processed target audio and video are generated.
2. The method of claim 1, wherein, The speech recognition result includes a timestamp; the step of segmenting the paragraph audio segment into multiple first statement audio segments based on the speech recognition result includes: Based on the timestamp, the audio segment of the paragraph is divided into multiple first statement audio segments that correspond one-to-one with the target statement.
3. The method of claim 1, wherein, The step of inputting the speech recognition text into a pre-trained text processing model, so that the text processing model outputs the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on set rules, includes: The speech recognition text is input into a pre-trained text processing model, which then calls the large language model service interface to send the processing instructions containing the set rules and the speech recognition text to the large language model. The large language model returns the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text, and outputs them. The set rules include scene adaptation rules and sentiment classification dimensions.
4. The method of claim 1, wherein, The step of speed-changing the corresponding statement video segments among the plurality of video segments based on the duration of the first and second statement audio segments to generate the target video stream includes: The multiple video segments are traversed, and the duration ratio of the current statement video segment to the corresponding first statement audio segment or second statement audio segment is calculated. The current statement video segment is then subjected to speed-changing processing based on the duration ratio. The target video stream is generated based on all the video segments that have undergone speed-changing processing.
5. The method of claim 1, wherein, The process of generating the processed target audio and video based on the target audio stream and the target video stream includes: Based on the rewritten speech recognition text or the target audio stream, drive the digital human model to generate lip-sync animation of the digital human. Based on the emotion classification labels, the digital human model is driven to generate facial expression animations of the digital human; The digital human is superimposed onto the target area in the target audio-visual background.
6. The method of claim 1, wherein, The process involves merging adjacent target sentences with the same sentiment classification label in the rewritten speech recognition text to obtain target paragraphs, and generating paragraph audio segments based on the target paragraphs and their corresponding sentiment classification labels. This process includes: Export the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text to a table file; In response to receiving the updated table file, the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text are updated based on the updated table file.
7. An audio-video processing apparatus, characterized by comprising: The device includes: The request response module is used to respond to a received processing request for the target audio and video, and to obtain the speech recognition text of the target audio and video and multiple video segments included in the target audio and video, wherein the multiple video segments include video segments of sentences corresponding to each sentence in the speech recognition text; The model processing module is used to input the speech recognition text into a pre-trained text processing model, so that the text processing model outputs the rewritten speech recognition text and the sentiment classification labels of each sentence in the rewritten speech recognition text based on the set rules. The audio generation module is used to merge adjacent target sentences with the same sentiment classification label in the rewritten speech recognition text to obtain target paragraphs, and generate paragraph audio segments based on the target paragraphs and their corresponding sentiment classification labels. An audio segmentation module is used to obtain the speech recognition result of the audio segment of the paragraph, and based on the speech recognition result, segment the audio segment of the paragraph into multiple first sentence audio segments; The audio generation module is also used to generate a second sentence audio segment based on the non-target sentences other than the target sentence in the rewritten speech recognition text and the corresponding sentiment classification tags; An audio stream generation module is used to generate a target audio stream based on the audio segments of the first statement and the second statement; The video stream generation module is used to perform speed-changing processing on the corresponding statement video segments among the plurality of video segments based on the duration of the first statement audio segment and the second statement audio segment, and generate a target video stream. The synthesis module is used to generate processed target audio and video based on the target audio stream and the target video stream. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
