A method and apparatus for audio processing

Abstract

The embodiment of the specification provides a method and device for audio processing, wherein the method for audio processing comprises the following steps: obtaining initial text corresponding to initial audio, generating initial basic information corresponding to at least two initial text segments, the at least two initial text segments being obtained based on the initial text, the initial basic information comprising a target text segment, a target audio segment, the target text segment being generated by translating the initial text segment based on translation information, and the target audio segment being synthesized based on the corresponding target text segment; in the case of modifying a to-be-modified text segment in the initial text, obtaining to-be-modified basic information corresponding to the to-be-modified text segment and a corrected text segment, updating the target audio segment in the to-be-modified basic information based on the corrected text segment, and obtaining corrected basic information; and generating target audio corresponding to the initial audio according to the corrected basic information and the initial basic information. The efficiency and accuracy of audio modification can be improved, and the quality and naturalness of the target audio can be ensured.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of speech processing technology, and in particular to an audio processing method and apparatus. Background Technology

[0002] With the rapid development of internet technology, cross-language video content has also grown rapidly, finding widespread application in streaming media and short video creation. Cross-language video content involves video translation, which can be understood as translating video content from one language into another, such as translating an English video into a Chinese video.

[0003] Video translation involves text recognition (converting audio from video into text), translation (translating audio from video into text in the target language), and speech synthesis (converting text in the target language into audio in the target language). Currently, video translation often involves independently performing text recognition, translation, and speech synthesis on the initial video's audio. Furthermore, the translated audio needs to be reintroduced into the video, replacing the original audio with the target language audio and adding target language subtitles while preserving the video frame, subtitle timeline, and overall video rhythm. Therefore, if text recognition, translation, and speech synthesis are performed independently, it becomes problematic. In practical applications, users may customize the subtitles, requiring each step of the video translation process to be re-executed separately, increasing the difficulty and efficiency of generating the target video. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an audio processing method. One or more embodiments of this specification also relate to an audio processing method applied to a cloud-side device, an audio processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an audio processing method is provided, comprising: Obtain the initial text corresponding to the initial audio, and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. When modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, and update the target audio segment in the basic information to be modified based on the corrected text segment to obtain the corrected basic information; Based on the corrected basic information and the initial basic information, the target audio corresponding to the initial audio is generated.

[0006] According to a second aspect of the embodiments of this specification, an audio processing method is provided, applied to a cloud-side device, comprising: The receiving end device sends an audio processing request, wherein the audio processing request carries initial audio; Obtain the initial text corresponding to the initial audio, and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. When modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, and update the target audio segment in the basic information to be modified based on the corrected text segment to obtain the corrected basic information; Based on the corrected basic information and the initial basic information, the target audio corresponding to the initial audio is generated; The target audio is sent to the end-side device.

[0007] According to a third aspect of the embodiments of this specification, an audio processing apparatus is provided, comprising: The first acquisition module is configured to acquire the initial text corresponding to the initial audio and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. The first update module is configured to, when modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, update the target audio segment in the basic information to be modified based on the corrected text segment, and obtain the corrected basic information; The first generation module is configured to generate the target audio corresponding to the initial audio based on the corrected basic information and the initial basic information.

[0008] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described audio processing method.

[0009] According to a fifth aspect of the embodiments of this specification, an electronic device is provided, comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described audio processing method.

[0010] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method.

[0011] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method.

[0012] In one embodiment of the audio processing method provided in this specification, initial text corresponding to initial audio is obtained, and initial basic information corresponding to at least two initial text segments contained in the initial text is generated. This unifies audio, text, and audio processing within a structured framework of basic information, providing a unified data foundation for subsequent text translation and speech synthesis, avoiding data inconsistencies caused by independent processing at each stage. Furthermore, when text in the audio needs modification, the aforementioned generated initial basic information can be used directly for modification. Specifically, the target audio segment in the basic information to be modified is updated based on the corrected text segment. Thus, in the case of partial text modification, speech synthesis is performed only on the affected text segment without re-executing the entire process, reducing computational costs and processing time in text modification scenarios and improving audio generation efficiency. Moreover, by generating target audio based on the corrected and initial basic information, the audio continuity between the corrected and uncorrected parts during splicing is ensured, thereby guaranteeing the quality and naturalness of the generated target audio. Attached Figure Description

[0013] Figure 1 This is an architecture diagram of an audio processing system provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating an audio processing method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating an audio processing method applied to a cloud-side device, as provided in one embodiment of this specification. Figure 4 This is a schematic diagram of the structure of an audio processing apparatus provided in one embodiment of this specification; Figure 5 This is a schematic diagram of an audio processing apparatus located on a cloud-side device according to one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification; Figure 7 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] Text-to-Speech (TTS): TTS can be understood as an artificial intelligence technology that converts text information into audible audio signals. Specifically, it performs linguistic analysis and acoustic modeling on text information and generates audio waveforms corresponding to the text. TTS technology is widely used in voice assistants, video dubbing, audiobooks, and other fields. In practical applications, the input information for a TTS model can also include audio, which can generate speech output with a similar timbre based on the timbre characteristics of the input audio, thereby achieving timbre cloning or timbre conversion.

[0022] Automatic Speech Recognition (ASR): ASR, also known as speech-to-text technology, can be understood as an artificial intelligence technology that converts audio signals into text information. Specifically, it can map the acoustic features in an audio waveform into a text sequence through the collaborative work of an acoustic model and a language model.

[0023] Forced Alignment: Forced alignment is an important technique in the field of speech recognition. It can be understood as a technique that, given the audio and its corresponding text content, automatically determines the start and end timestamps of each character or word on the audio timeline using an alignment algorithm. It establishes a temporal correspondence between a text sequence and an audio frame sequence. Simply put, a forced alignment algorithm can pinpoint the start and end times of each character, word, or phoneme in the audio; that is, it specifies when each character, word, or phoneme in the text begins and ends in the audio.

[0024] Speech subject: The speech subject can be understood as the object that produces speech in the audio, i.e., the speaker. It is understood that the speaker can be a natural person or a virtual person designed using computer or artificial intelligence technology. In one or more embodiments of this specification, a speech subject has a subject identifier, which is used to distinguish different speech subjects. The same speech subject has consistent sound characteristics in different segments of the audio. Whether the speech subjects corresponding to different audio segments are consistent can be determined by comparing the similarity of the sound characteristics of each audio segment. If the similarity between the sound characteristics of any two initial audio segments is greater than a preset similarity threshold, it can be determined that the two audio segments correspond to the same speech subject, and the subject identifier of the speech subject can be recorded in the basic information corresponding to the two speech segments.

[0025] Voice features: Voice features can be understood as biological characteristics that reflect the uniqueness of a speaker's voice. They are typically extracted through acoustic analysis of audio signals and can be used to distinguish different speakers. In one or more embodiments of this specification, the similarity between any two voice features can be determined by calculating the cosine similarity or Euclidean distance between the two voice feature vectors.

[0026] Voice Activity Detection (VAD): VAD can be understood as a technique for segmenting audio signals into speech frames and silence frames. A speech frame can be understood as an audio segment in which a speaker is making a sound, while a silence segment can be understood as an audio segment in which no speaker is making a sound. VAD is usually performed in units of fixed-length audio frames, and each frame can be labeled as a speech frame or a silence frame.

[0027] Corresponding to the background technology, video translation typically involves multiple stages such as text recognition, text translation, and speech synthesis. Currently, text recognition, translation, and speech synthesis are often performed independently on the audio of the initial video. However, in real-world video translation scenarios, subtitles can originate from various sources, such as native subtitles from the video, subtitles recorded from the screen, text from speech recognition, or manually corrected subtitles. Furthermore, the adaptability of speech generation varies greatly depending on the language, segment length, and speaking rhythm. Inconsistencies between the translated text length and the original segment length can directly lead to audio-visual asynchrony. In addition, scenarios with multiple speakers require the preservation of speaker segmentation and timbre consistency. In other words, current translation scenarios have many practical needs, rather than simply involving text recognition, text translation, and speech synthesis.

[0028] In such a scenario, because text recognition, translation, and speech synthesis are performed independently, numerous problems can easily arise. For example, the text recognition stage may only output the text corresponding to the audio, lacking consideration for the text structure requirements of downstream translation and speech synthesis. Similarly, the translation stage may only perform text conversion without constraining the duration of the translated text in the video, easily leading to audio-visual asynchrony, thus increasing the difficulty of generating the target video, reducing accuracy, and resulting in low generation efficiency. Furthermore, if modifications to the subtitles in the video are needed, each stage of the translation task needs to be re-executed, leading to inefficient and difficult modifications and wasting computational resources.

[0029] To address the aforementioned technical problems, this specification provides an audio processing method. This specification also relates to an audio processing method applied to cloud-side devices, an audio processing apparatus, an audio processing apparatus applied to cloud-side devices, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0030] See Figure 1 , Figure 1 This specification illustrates an architecture diagram of an audio processing system according to one embodiment of the present specification. The audio processing system may include a client 100 and a server 200. Client 100 is used to send an audio processing request to server 200, wherein the audio processing request carries initial audio; Server 200 is configured to receive an audio processing request sent by client 100, obtain the initial text corresponding to the initial audio, and generate initial basic information corresponding to at least two initial text segments contained in the initial text. The initial basic information includes a target text segment and a target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is a synthesized audio corresponding to the target text segment. When modifying a text segment to be modified in the initial text, server 200 obtains the basic information to be modified and a corrected text segment corresponding to the text segment to be modified. Based on the corrected text segment, server 200 updates the target audio segment in the basic information to be modified to obtain corrected basic information. Server 200 generates the target audio corresponding to the initial audio based on the corrected basic information and each initial basic information segment. Server 200 then sends the target audio to client 100. Client 100 is also used to receive the target audio sent by server 200.

[0031] An audio processing system may include multiple clients 100 and a server 200. Clients 100 can be referred to as end-side devices, and the server 200 can be referred to as cloud-side devices. Multiple clients 100 can establish communication connections through the server 200. In an audio processing scenario, the server 200 is used to provide audio processing services between the multiple clients 100. Each client 100 can act as a sender or receiver, communicating through the server 200.

[0032] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In audio processing scenarios, users can publish data streams to server 200 through client 100, server 200 can generate target audio based on the data stream, and push the target audio to other clients that have established communication.

[0033] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0034] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on a computing device and depends on the device or certain apps on the device to run. The computing device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured on the computing device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0035] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0036] It is worth noting that the audio processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the audio processing methods provided in the embodiments of this specification. In other embodiments, the audio processing methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0037] It should be noted that the audio processing method provided in this specification can be applied to various scenarios. For example, in the cross-language video translation scenario described in the background, it is necessary to convert the initial video from the initial language to the target language. This can be achieved by acquiring the initial audio from the initial video and identifying it as initial text, dividing the initial text into multiple initial text segments, and translating each segment into a target text segment in the target language. Then, a target audio segment is generated for each target text segment. Based on the aforementioned target text segments and target audio segments, basic information corresponding to each initial text segment can be constructed, thereby enabling cross-language video translation. Furthermore, when modifications to certain subtitles in the initial video are required, such as when a user modifies certain subtitle segments, the audio of the affected initial text segments can be resynthesized based on the basic information corresponding to each initial text segment and the subtitle information to be modified, without re-executing the entire translation process, thus improving modification efficiency and accuracy.

[0038] For example, in multilingual dubbing scenarios, where multiple language versions of the initial video need to be generated, the basic information for each language version corresponding to each initial text segment can be generated in the manner described above, improving the efficiency of multilingual dubbing. Furthermore, if modifications to certain subtitles in the initial video are needed, these subtitles can be modified specifically based on the aforementioned basic information, without needing to modify other parts, thus improving both efficiency and accuracy.

[0039] For example, in educational courses, it's necessary to convert the audio content of instructional videos into multiple languages ​​for users in different regions and speaking different languages ​​to watch and learn. This can be achieved by acquiring the initial audio from the initial instructional video and identifying it as initial text. This initial text is then divided into multiple initial text segments, each translated into target text segments in multiple languages. Target audio segments are then generated for each language's target text segment. Based on the aforementioned target text and audio segments for each language, basic information for each initial text segment in each language can be constructed, thus enabling cross-language video translation. Furthermore, if modifications to certain subtitles in the initial instructional video are needed, such as user modifications to certain subtitle segments, the audio of the affected initial text segments can be resynthesized based on the basic information of each language's corresponding initial text segments and the subtitle information to be modified, without re-executing the entire translation process, improving modification efficiency and accuracy.

[0040] In summary, this specification does not limit the practical application scenarios of the audio processing methods. The audio processing methods provided in this specification can be used in any scenario where it is necessary to translate initial audio to obtain target audio, and where it is possible to avoid re-executing the entire processing step when modifying the initial text. Furthermore, for cases where it is necessary to generate target audio and its corresponding subtitles while preserving the main speaker, video character timeline, and video rhythm of the initial audio, the audio processing methods provided in one or more embodiments of this specification can be used.

[0041] See Figure 2 , Figure 2 A flowchart of an audio processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0042] Step 202: Obtain the initial text corresponding to the initial audio, and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment.

[0043] The initial audio can be understood as the raw audio data that requires audio processing. In one or more embodiments of this specification, the initial audio may be a meeting recording containing multiple speakers, a movie clip containing dialogue, or a recording of a solo speech, etc. In practical applications, the initial audio may be derived from an audio track in a video file, a separate audio file, or a real-time acquired audio stream.

[0044] The initial text can be understood as the text content obtained through speech recognition of the initial audio, reflecting the speech content of each speaker in the initial audio. In one or more embodiments of this specification, the initial text can be obtained in various ways. For example, the initial audio can be input into an ASR model to obtain the recognized text output by the ASR model, which is the initial text corresponding to the initial audio. Another example is that it can be obtained by importing an external file. Yet another example is that it can be obtained from the video frame corresponding to the initial video from which the initial audio originates, specifically by performing Optical Character Recognition (OCR) on the subtitle area in the video frame of the initial video to extract the subtitle text burned into the video frame, thereby using the extracted subtitle text as the initial text.

[0045] It should be noted that the above methods for obtaining the initial text can be used individually or in combination to adapt to different video and subtitle sources in actual application scenarios.

[0046] In this context, the initial text segment can be understood as a text segment obtained by dividing the initial text according to preset rules. In one example, the preset division rules could be based on a preset number of characters, a preset number of words, or a preset number of sentences. Specifically, the initial text could be divided by specifying the number of characters, words, or sentences contained, to obtain at least two initial text segments. In another example, the preset division rules could be based on the detection results of VAD (Visual Analog Detection), specifically, based on the start and end times of at least one continuous speech segment detected by VAD in the initial audio, with each detected continuous speech segment corresponding to one initial text segment. In yet another example, the preset division rules could be based on changes in the pronunciation subject, specifically, if the pronunciation subject in the detected initial audio changes, the text before and after the change is divided into two different initial text segments.

[0047] Understandably, each initial text segment corresponds to a continuous speech segment in the initial audio, that is, it corresponds to an initial audio segment.

[0048] As mentioned earlier, the audio processing method provided in this specification can be applied to the generation of cross-language audio in different scenarios. That is, it is necessary to convert audio from one language to another. Therefore, the translation information can contain the target language, and this translation information can be used to convert the text in the initial language corresponding to the initial audio into text in the target language. In other words, it can generate target text fragments in the target language corresponding to the initial text fragments based on the translation information. Specifically, after dividing the initial text into multiple initial text fragments, each initial text fragment and the translation information can be input into a preset language model, which will then generate target text fragments in the target language corresponding to each initial text fragment.

[0049] For example, if the target language of the translation information is Chinese and the initial language of the initial audio is English, that is, if the English audio needs to be translated into Chinese audio, then the Chinese and the initial text fragment can be input into the preset language model, and the preset language model can translate the initial English text fragment into the target Chinese text fragment.

[0050] It should be noted that this manual does not impose specific restrictions on the method of obtaining translation information; it can be set based on the actual scenario and application requirements. Furthermore, the target languages ​​included in the translation information can be one or more. When multiple target languages ​​are included, the initial text fragment can be translated into target text fragments in multiple languages, thereby enabling the construction of basic information corresponding to the target text fragments in multiple languages ​​in subsequent steps.

[0051] In one or more embodiments of this specification, the target text segment can be understood as target language text obtained by translating an initial text segment based on translation information, and the target audio segment is audio data generated based on the target text segment using speech synthesis technology. That is, after obtaining each target text segment, each target text segment can be input into a preset speech synthesis model, and the preset speech synthesis model can output the synthesized audio corresponding to each target text segment, thereby obtaining the target audio segment corresponding to each target text segment.

[0052] It is understandable that the language corresponding to the initial audio segment and the initial text segment is the initial language, and the language corresponding to the target audio segment and the target text segment is the target language.

[0053] In practical applications, during the process of translating an initial text fragment to generate a target text fragment, the translation quality directly affects the accuracy of subsequent speech synthesis and the audibility of the final output target audio. If the translated target text fragment contains errors or is unnaturally expressed, it will lead to semantic deviations or pronunciation abnormalities in the synthesized speech. Therefore, in order to improve the accuracy of the generated target text fragment and thus the accuracy of the generated target audio, quality verification can be performed during the translation process.

[0054] In one or more embodiments of this specification, the target text segment corresponding to the initial text segment is obtained in the following manner: for each initial text segment, the translation information and the initial text segment are input into a preset language model to obtain the intermediate text segment corresponding to the initial text segment generated by the preset language model; If the characters between the initial text segment and the intermediate text segment meet a preset character condition, the intermediate text segment is determined to be the target text segment corresponding to the initial text segment; If the characters between the initial text fragment and the intermediate text fragment do not meet the preset character conditions, the initial text fragment is re-inputted into the preset language model.

[0055] The preset language model can be understood as a neural network model with cross-language translation capabilities. This preset language model can generate translated text (i.e., target text fragments) in the target language from initial text fragments in the initial language. In one or more embodiments of this specification, the preset language model can be a pre-trained language model, such as a multilingual translation model based on the Transformer architecture, or a translation model specifically fine-tuned for video translation scenarios.

[0056] In one or more embodiments of this specification, the preset language model can receive translation information and an initial text fragment as input. The translation information is used to indicate the translation direction, that is, to convert the initial language corresponding to the initial text fragment into the target language. In other words, the initial text fragment and the translation information can be input into the preset language model to obtain the intermediate text fragment corresponding to the initial text fragment.

[0057] It is understandable that the intermediate text fragments are translations generated by the pre-defined language model based on the initial text fragments. Since they have not yet undergone quality verification, the intermediate text fragments may contain translation errors due to model inference errors, insufficient understanding of the context, or ambiguity of the initial text fragments, such as omissions, over-translations, semantic shifts, or grammatical incoherence.

[0058] Therefore, in one or more embodiments of this specification, the quality of the generated intermediate text fragments can be verified according to preset character conditions to obtain the target text fragment. The preset character conditions can be understood as verification rules used to determine whether the translation quality of the intermediate text fragments meets the standards.

[0059] In practical applications, preset character conditions can include various dimensions of verification indicators, such as character quantity ratio conditions, keyword matching conditions, or character line conditions, etc.

[0060] The requirement of meeting the character count ratio can be understood as the difference between the number of characters in the intermediate text segment and the number of characters in the initial text segment falling within a preset range; too many or too few characters are considered abnormal. In other words, if the difference between the number of characters in the intermediate text segment and the initial text segment is within the preset range, the intermediate text segment is considered to meet the preset character count condition and can be used as the target text segment. If the difference between the number of characters in the intermediate text segment and the initial text segment is not within the preset range, the intermediate text segment is considered to not meet the preset character count condition, and its corresponding initial text segment can be re-input into the preset language model to obtain a new intermediate text segment and re-determine whether it meets the preset character count condition. It should be noted that the preset range can be set based on actual needs and application scenarios; this manual does not impose specific restrictions.

[0061] Meeting the keyword matching condition can be understood as ensuring that the keywords in the initial text segment have correct translations in the intermediate text segment. In other words, if all the keywords in the initial text segment are present in the intermediate text segment, the intermediate text segment is deemed to meet the preset character conditions and can be used as the target text segment. If any keyword in the intermediate text segment is not present, the intermediate text segment is deemed not to meet the preset character conditions, and its corresponding initial text segment can be re-input into the preset language model to obtain a new intermediate text segment and re-determine whether it meets the preset character conditions. It should be noted that the keywords can be set based on actual needs and application scenarios; this manual does not impose specific restrictions.

[0062] The "satisfying the character line condition" can be understood as the difference between the number of lines in the intermediate text segment and the number of lines in the initial text segment being within a preset line number range, and the difference between the number of characters in the intermediate text segment and the number of characters in the initial text segment being within a preset line character range. In other words, the line number and character number of the initial text segment and the intermediate text segment can be obtained separately. If the difference between their line numbers and character numbers is within the preset line number range, and the difference between their character numbers is within the preset line character range, then the preset character condition is satisfied between the initial text segment and the intermediate text segment. It should be noted that the preset line character range and the preset line number range can be set based on actual needs and application scenarios; this manual does not impose specific restrictions, such as allowing them to be zero.

[0063] Correspondingly, not meeting the preset character conditions may include: the difference between the number of characters in the intermediate text segment and the number of characters in the initial text segment does not fall within the preset number range; or the keywords in the initial text segment have corresponding correct translations in the intermediate text segment; or the difference between the number of lines in the intermediate text segment and the number of lines in the initial text segment does not fall within the preset number of lines and the difference between the number of characters in the intermediate text segment and the number of characters in the initial text segment does not fall within the preset number of line characters.

[0064] In one or more embodiments of this specification, the preset character conditions may further include contextual or semantic consistency verification, that is, checking whether the context between the intermediate text segment and the target text segment corresponding to the adjacent initial text segment is coherent. Specifically, for each initial text segment, the intermediate text segment corresponding to the initial text segment and the target text segment corresponding to the adjacent initial text segment can be input into a large language model to evaluate the contextual or semantic coherence score between the intermediate text segment corresponding to the initial text segment and the target text segment corresponding to the adjacent initial text segment. If the score is greater than a preset score threshold, the intermediate text segment corresponding to the initial text segment can be determined as the target text segment corresponding to the initial text segment. The preset score threshold can be set based on actual needs and application scenarios, and this specification does not impose specific limitations. By scoring the context and semantics between adjacent text segments, it can be ensured that adjacent text segments are consistent in context and semantics, such as person address, tense, and tone, avoiding inconsistencies.

[0065] In one or more embodiments of this specification, during the process of re-inputting the initial text segment corresponding to the intermediate text segment that does not meet the preset character conditions into the preset language model to regenerate the intermediate text segment, the number of times the intermediate text segment is regenerated can also be obtained. If the number is greater than the preset number threshold, the intermediate text segment with the highest matching degree with the initial text segment can be selected from each historical intermediate text segment as the target text segment. Alternatively, it can be determined that the translation has failed to trigger human translation or other translation engines. In other words, the intermediate text segment corresponding to the initial text segment can be generated by human translation or by other translation engines and used as the target text segment.

[0066] In practical applications, a numbered input method can be used during the translation process. That is, each initial text fragment can be numbered sequentially and then input into the preset language model along with the translation information. The preset language model can output intermediate text fragments and perform preset character condition checks.

[0067] It is understood that any one of the preset character conditions of the above multiple dimensions can be used, or they can be used in combination. This manual does not impose any specific restrictions.

[0068] Based on the above method, intermediate text segments are generated by a preset language model, and the intermediate text segments are validated using preset character conditions. If the validation fails, a re-translation is triggered, which can ensure the quality of the target text segments obtained by translation, improve the accuracy and stability of translation, and thus improve the accuracy and coherence of the target audio generated subsequently.

[0069] In one or more embodiments of this specification, after obtaining the target text segment and target audio segment corresponding to each initial text segment, the initial basic information corresponding to each initial text segment can be constructed. Specifically, the target text segment and target audio segment corresponding to the initial text segment can be used as its corresponding initial basic information. By constructing the initial text segment, the corresponding translated text (i.e., the target text segment), and the synthesized audio (i.e., the target audio segment) as basic information for unified management, a data foundation is provided for subsequent modifications to the audio. Furthermore, the data format corresponding to each audio segment is a unified format in the basic information, avoiding problems such as data format re-alignment caused by independently performing each translation stage, thus facilitating maintenance and management.

[0070] In one or more embodiments of this specification, without modifying the initial text, the generated target audio segments can be directly concatenated according to their corresponding timestamps to obtain the target audio. Each target audio segment, according to its corresponding timestamp, represents the pronunciation time of the initial audio segment corresponding to that target audio segment in the initial audio. The pronunciation time of the initial audio segment corresponding to the initial text segment in the initial audio can be obtained based on forced alignment technology, and thus used as the pronunciation time of the target audio segment corresponding to the initial text segment in the initial audio. Then, the target audio segments can be concatenated in chronological order to obtain the target audio.

[0071] Step 204: When modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, and update the target audio segment in the basic information to be modified based on the corrected text segment to obtain the corrected basic information.

[0072] Here, the text fragment to be modified can be understood as the initial text fragment that needs to be corrected among multiple initial text fragments corresponding to the initial text. The corrected text fragment can be understood as the new text content obtained after modification, used to replace the original target text fragment. The basic information to be modified is the initial basic information corresponding to the text fragment to be modified. It can be understood that this basic information to be modified includes at least the target text fragment and the target audio fragment corresponding to the text fragment to be modified.

[0073] In practical applications, modification operations can be triggered by user-initiated editing, automatic quality detection identifying errors, or automatic modification based on contextual semantics. In one specific example, a user can actively select the initial text segment to be modified through the user editing interface corresponding to the initial audio, and can then input the corrected text content—the corrected text segment. In another specific example, quality detection can be automatically performed on each initial text segment, and the identified low-quality, erroneous segments can be used as the text segments to be modified. Furthermore, a large language model can be used to automatically generate corrected text segments based on contextual semantics.

[0074] In one or more embodiments of this specification, when updating the target audio segment in the basic information to be modified based on the corrected text segment, a corrected audio segment corresponding to the corrected text segment can be generated. This corrected audio segment is audio data obtained by re-synthesizing the corrected text segment, and can be used to replace the target audio segment in the basic information to be modified. Specifically, after obtaining the corrected text segment, it can be input into a preset speech synthesis model to regenerate the corresponding synthesized audio to obtain the corrected audio segment. Then, the corrected audio segment can be used to replace the target audio segment in the basic information to be modified, thereby obtaining the corrected basic information corresponding to the text segment to be processed, so as to realize the local update of the text segment to be modified in the speech synthesis stage.

[0075] In one or more embodiments of this specification, during the process of updating the target audio segment in the basic information to be modified based on the corrected text segment, different speech synthesis strategies can be selected according to the duration of the target audio segment in the basic information to be modified. For example, when the duration of the target audio segment is greater than a preset duration threshold, a fine-tuned speech synthesis model can be used for synthesis, that is, the corrected text segment can be input into the fine-tuned speech synthesis model to obtain a better timbre matching effect. When the duration of the target audio segment is less than the preset duration threshold, a preset speech synthesis model can be used directly for synthesis, that is, the corrected text segment can be directly input into the preset speech synthesis model to avoid the overfitting problem introduced by fine-tuning.

[0076] In the above method, only the audio segments corresponding to the text segments affected by the modification are updated by using the initial basic information corresponding to each initial text segment, while the audio segments of other unmodified text segments remain unchanged. This can avoid the computational overhead of full modification and redo, improve modification efficiency, and also ensure the overall coherence and naturalness of the audio.

[0077] Step 206: Generate the target audio corresponding to the initial audio based on the corrected basic information and the initial basic information.

[0078] In one or more embodiments of this specification, the target audio can be obtained by splicing the corrected audio segments in the corrected basic information and the target audio segments in each initial basic information (excluding the initial basic information corresponding to the corrected basic information) according to their pronunciation times in the initial audio. Furthermore, during the splicing process, audio smoothing can be performed at the boundaries of adjacent audio segments, for example, by fading in and out audio of a certain duration before and after the splicing point to ensure the continuity and naturalness of the sound quality at the splicing point. It is understood that since the initial basic information corresponding to the text segment to be modified has been updated to the corrected basic information, the initial basic information used when generating the target audio is each initial basic information excluding the initial basic information corresponding to the corrected basic information. The target audio can be obtained by splicing these initial basic information and the audio segments in the corrected basic information according to their chronological order in the initial audio.

[0079] In practical applications, after generating the target audio, the target audio can be combined with the video footage of the initial video. Specifically, the target audio and its corresponding text can be integrated into the initial video from which the initial audio originated, in order to achieve cross-language video generation.

[0080] In one or more embodiments of this specification, after obtaining the target video, the method further includes: Obtain the initial video corresponding to the initial audio and obtain the target text corresponding to the target audio; The target text and the target audio are updated into the initial video to obtain the target video.

[0081] The initial video can be understood as the original video file containing the initial audio. The initial video can contain a video frame track, an original audio track, and possibly an original subtitle track. The initial video is the input for video translation, requiring the audio content to be converted from the original language to the target language.

[0082] In this context, target text can be understood as the text in the target language obtained by translating the initial text. Target text comprises a collection of target text fragments corresponding to each initial text fragment, constituting complete target language text content. Target text can be used to generate subtitles for the target video.

[0083] After audio translation and speech synthesis, simply replacing the target audio with the initial audio may lead to problems such as mismatch between subtitles and audio, and desynchronization between video and sound. Therefore, to ensure the consistency between the translated text and audio and the video, updating the initial video with the target text and audio involves replacing the initial audio in the initial video with the target audio as a new audio track, and overlaying the target text as subtitles onto the video. During subtitle overlay, the display time and position of the subtitles in the video can be determined based on the time position information of each target text segment in the audio, ensuring that the subtitles appear synchronously with the synthesized audio.

[0084] In one or more embodiments of this specification, after synthesis, the target audio and its corresponding text can be pasted back into the video layout of the initial video, and four sets of products can be determined. When pasting the text corresponding to the target audio into the video layout of the initial video, the subtitle boxes can first be normalized to a preset boundary interval, such as [bx, by, bw, bh]. Specifically, the subtitle region in the initial video can be detected to obtain each subtitle detection box, and each subtitle detection box can be normalized to the preset boundary interval [bx, by, bw, bh], where bx is the normalized left boundary, by is the normalized upper boundary, bw is the normalized width, and bh is the normalized height. Then, based on the pixel width and height of the initial video, the normalized coordinates can be converted into pixel coordinate regions, i.e., pixel regions of interest (ROI).

[0085] In one or more embodiments of this specification, the standard deviations of the left, right, and center lines are calculated for all single-line subtitle detection frames. The line corresponding to the minimum standard deviation among the three lines is determined as the horizontal alignment method. If the standard deviation of the left boundary is the smallest, it is determined to be left-aligned; if the standard deviation of the right boundary is the smallest, it is determined to be right-aligned; and if the standard deviation of the center line is the smallest, it is determined to be centered. In cases of parallel alignment or multiple minimum values, the default center alignment is uniformly applied. The horizontal alignment method (left-aligned / right-aligned / centered) is determined by taking the minimum standard deviation of the three boundaries (left, right, and center) of all single-line subtitle detection frames. In cases of parallel alignment or multiple minimum values, the default center alignment is applied. The vertical coordinate is taken as the average of the upper boundary, and the font size is estimated from the minimum height of the detection frame to avoid individual tall frames increasing the font size. Based on the determined horizontal alignment method, vertical anchor point, and font size, the translated subtitles corresponding to the target text are pasted back into the initial video to maintain the original layout of the translated subtitles. Specifically, determine the horizontal position of the translated subtitles according to the horizontal alignment method, the vertical position according to the vertical anchor point, and the font size according to the font size. After replying, produce four sets of outputs, including the translated audio, bilingual subtitles, revised subtitles, intermediate package, and voice track. Among them, the bilingual subtitles have four arrangement options, with the source line first and the translated line second; the revised subtitles are for manual editing and then returned to the post-translation revision stage.

[0086] In a specific example, subtitle overlay can use a subtitle container format such as ASS or SRT, generating subtitle files based on the pronunciation time or start and end timestamps of each target text segment. The display time of the subtitles is aligned with the start and end timestamp positions of each target text segment in the target audio. The display position can be determined based on the position information of the original subtitles, or a preset default position such as centering at the bottom of the video can be used.

[0087] In a specific example, the target audio can also be kept in frame synchronization with the video frame. Specifically, the sampling rate of the target audio can be consistent with the audio track configuration of the initial video, and the total duration of the target audio can be matched with the total duration of the initial video. If the difference between the total duration of the target audio and the total duration of the initial video is greater than a preset difference, a silent segment can be added at the end of the target audio or appropriate adjustments can be made.

[0088] By updating the target text and target audio into the initial video using the above method, cross-language video generation is achieved, ensuring the synchronization and coordination of the translated and modified audio, subtitles and video footage, and improving the audiovisual consistency of the video.

[0089] The aforementioned audio processing method obtains the initial text corresponding to the initial audio and generates initial basic information corresponding to at least two initial text segments contained in the initial text. This unifies audio, text, and audio processing within a structured framework of basic information, providing a unified data foundation for subsequent text translation and speech synthesis, and avoiding data inconsistencies caused by independent processing at each stage. When text in the audio needs modification, the aforementioned generated initial basic information can be used directly for modification. Specifically, the target audio segment in the basic information to be modified is updated based on the corrected text segment. Therefore, in the case of partial text modification, speech synthesis is performed only on the affected text segment without re-executing the entire process, reducing computational costs and processing time in text modification scenarios and improving audio generation efficiency. By generating target audio based on the corrected audio segment and each target audio segment, the audio continuity between the corrected and uncorrected parts during splicing is ensured, avoiding discontinuities or abnormalities in sound quality at splicing points, thereby guaranteeing the quality and naturalness of the generated target audio.

[0090] Furthermore, in one or more embodiments of this specification, before synthesizing the target audio segments corresponding to each target text segment, the method further includes: for each target text segment, generating placeholder audio based on the initial audio segment corresponding to the target text segment, the placeholder audio being used as a substitute audio for the target text segment when the synthesis of the target text segment fails; when synthesizing the target audio segment corresponding to the target text segment, if the target text segment is marked as a copy sentinel, directly using the placeholder output corresponding to the target text segment as the target audio segment and skipping the synthesis; if the synthesis of the target text segment fails, using the placeholder output corresponding to the target text segment as the target audio segment; if the synthesis failure is caused by a network anomaly, retrying the current synthesis request; if the retry still fails, falling back to the backup backend for the current synthesis request.

[0091] In one optional embodiment, the initial audio segment corresponding to the target text segment can be used as placeholder audio, the initial audio segment corresponding to the target text segment can be silenced to obtain placeholder audio, or the initial audio segment corresponding to the target text segment can be copied verbatim after duration alignment to obtain placeholder audio. Before synthesizing the target text segment, it is first determined whether the target text segment is marked as a verbatim copy sentinel.

[0092] In one or more embodiments of this specification, upon receiving a copy sentinel marked by a user for the target text fragment, the initial audio fragment corresponding to the target text fragment is directly used as the target audio fragment, and the copy sentinel is used to indicate that the target text fragment does not participate in translation and synthesis.

[0093] In one or more embodiments of this specification, if the synthesis of the target text segment fails, the failure type is determined. If the failure type is a segment-level anomaly, the placeholder output corresponding to the target text segment is used as the target audio segment, and the failure event corresponding to the target text segment is recorded. The failure event is used for the calculation of the proportion of lost speech and the proportion of abnormal silence in subsequent quality statistics.

[0094] In one or more embodiments of this specification, when a synthesis failure is caused by a network anomaly, retrying the current synthesis request can specifically involve retrying the current synthesis request a preset number of retries and a backoff interval if the synthesis of the target text fragment fails and the failure type is a network anomaly. If the retry is successful, the target audio fragment obtained during the retry is used as the target audio fragment corresponding to the target text fragment. If the retry still fails, the current synthesis request is rolled back to a backup backend.

[0095] In practical applications, when generating the initial basic information corresponding to the initial text segment in step 202 above, simply including the target text segment and the target audio segment is insufficient for operations such as speech duration matching, speaker consistency judgment, and speech synthesis model selection. Therefore, in one or more embodiments of this specification, the initial basic information also includes a first duration, a second duration, and a speaker corresponding to the initial text segment. The first duration is the pronunciation duration of the initial audio segment corresponding to the initial text segment in the initial audio, and the second duration is the duration of the target audio segment corresponding to the initial text segment.

[0096] The first duration can be understood as the pronunciation duration of the initial audio segment corresponding to the initial text segment in the initial audio, or the actual length of time it occupies, or the time consumed by the speaker to pronounce the initial text segment. The first duration can be determined by performing Voice Activity Detection (VAD) on the initial audio, specifically by detecting the start and end times of the audio corresponding to the initial text segment; the difference between the two is the first duration. Alternatively, it can be determined by forcibly aligning the initial audio and initial text. Specifically, by using forced alignment techniques to obtain the timestamps of each initial text segment in the initial audio, the pronunciation duration corresponding to each initial text segment can be obtained.

[0097] The second duration can be understood as the length of the synthesized audio corresponding to the target text segment generated after translating the initial text segment. Due to differences in expression habits and speech rates across languages, the audio duration of text segments with the same semantic content may differ in different languages. For example, English expressions may be more concise, while the corresponding Chinese translation may require more syllables, resulting in a discrepancy between the second and first durations. Precise control of the second duration is crucial for ensuring the synchronization between the translated audio and the original video footage.

[0098] Corresponding to the aforementioned description of the speech subject, the speech subject can be used to distinguish speech segments from different speakers. When multiple speech subjects exist in the initial audio, each speech subject has unique timbre characteristics. Therefore, to ensure that a matching preset speech synthesis model can be selected for each speech subject during subsequent speech synthesis, the speech subject corresponding to each generated initial text segment can be labeled. In practical applications, the speech subject can be identified using a subject identifier. Specifically, a unique subject identifier can be set for each speech subject identified in the initial video, or a unique subject identifier can be set for each speech subject identified in the initial audio. This subject identifier can be used throughout the audio and video processing flow to ensure that the same speech subject in the same video segment always uses the same subject identifier. The assignment of subject identifiers can be based on the clustering results of sound features, grouping audio segments with sound feature similarity exceeding a feature similarity threshold into the same speech subject.

[0099] The initial basic information constructed in the above manner includes the first duration, the second duration, and the speaker, enriching the dimensions of the initial basic information. This provides a data foundation for subsequent speech synthesis, duration adjustment, and speaker consistency maintenance, which can improve the quality and smoothness of the generated target audio and target video.

[0100] In one or more embodiments of this specification, when segmenting the initial text to obtain initial text fragments, directly dividing it according to a fixed length or punctuation marks may result in a large number of excessively short text fragments. These excessively short text fragments are prone to inaccurate translation due to insufficient context, and also make it difficult to generate natural and fluent audio during speech synthesis due to the short duration of the synthesized audio. Therefore, it is necessary to avoid creating excessively short text fragments while ensuring semantic integrity.

[0101] In one or more embodiments of this specification, the initial text fragments in the initial text are obtained in the following manner: The initial text is divided into at least two baseline text segments, and the pronunciation subject corresponding to each baseline text segment is determined; If the pronunciation duration of the reference text segment to be processed in the initial audio is less than the first preset duration, an associated reference text segment with the same pronunciation subject as the reference text segment to be processed is obtained, and the reference text segment to be processed and the associated reference text segment are merged to obtain a merged text segment, wherein the reference text segment to be processed is any one reference text segment; The merged text segments and the baseline text segments with a pronunciation duration greater than or equal to the first preset duration are used as the initial text segments.

[0102] It should be noted that the benchmark text segment can be understood as a text segment obtained after preliminary segmentation of the initial text. Preliminary segmentation can be performed using the aforementioned preset segmentation rules, or it can be based on punctuation marks, sentence boundaries, or speech pauses. In practical applications, the preliminary segmented benchmark text segments are usually numerous and finely granular. Therefore, there are benchmark text segments with very short pronunciation durations, such as only one or two words of interjection or short responses. Thus, benchmark text segments with the same pronunciation subject can be merged to improve the accuracy of subsequent translation and speech synthesis.

[0103] The first preset duration can be understood as a duration threshold used to determine whether the baseline text segment is too short. The first preset duration can be set according to the actual scenario and actual needs, such as 1 second, 2 seconds, or 3 seconds, etc.

[0104] The associated reference text segment can be understood as another reference text segment with the same pronunciation subject as the reference text segment to be processed. Further, the associated reference text segment can be understood as a reference text segment with the same pronunciation subject as the reference text segment to be processed and adjacent in text order. That is, if the pronunciation duration of the reference text to be processed in the initial audio is less than a first preset duration, other reference text segments with the same pronunciation subject and adjacent in text order as the reference text to be processed can be merged with the reference text to be processed. The merged reference text segment can be used as the initial text segment, thus ensuring that the merged text segment still belongs to the same pronunciation subject and avoiding the erroneous merging of speech segments with different pronunciation subjects.

[0105] Understandably, if the duration of the reference text segment in the initial audio is greater than or equal to the first preset duration, the reference text segment can be directly used as the initial text segment.

[0106] In one or more embodiments of this specification, when merging the base text fragment to be processed and the associated base text fragment to obtain the merged text fragment, the process can be carried out sequentially from left to right according to the text order, or it can be merged in chronological order according to the start and end timestamps of each base text fragment in the initial audio.

[0107] In one or more embodiments of this specification, for each reference text segment to be processed, the reference text segment to be processed can be merged with the reference text segment with the same pronunciation subject adjacent to its right. If there is no reference text segment with the same pronunciation subject to the right of the reference text segment to be processed, the reference text segment to be processed can be merged with the reference text segment with the same pronunciation subject adjacent to its left. If there are no adjacent reference text segments with the same pronunciation subject to the left or right of the reference text segment to be processed, the reference text segment to be processed can be retained as an independent initial text segment.

[0108] It is understandable that the initial text segment obtained by merging two or more reference text segments has its first duration in the initial audio as the sum of the pronunciation durations of the corresponding reference text segments in the initial audio.

[0109] The above method can merge baseline text segments with excessively short pronunciation durations with other baseline text segments of the same pronunciation subject, thereby obtaining initial text segments of appropriate length and complete semantics. This ensures the semantic integrity of the initial text segments and avoids the problem of poor translation and speech synthesis quality caused by excessively short initial text segments, thus improving the quality of subsequent translation and speech synthesis.

[0110] In one or more embodiments of this specification, the pronunciation subject corresponding to each reference text segment can be determined based on the sound features corresponding to each initial audio segment. Specifically, determining the pronunciation subject corresponding to each reference text segment includes: Determine the acoustic features of the initial audio segment in the initial audio corresponding to each benchmark text segment; If the similarity between the sound features corresponding to any two reference text segments is greater than a preset similarity threshold, it is determined that the two reference text segments correspond to the same pronunciation subject, and the pronunciation subject corresponding to each reference text segment is generated.

[0111] In this context, sound features can be understood as vectors extracted from audio signals that represent the unique timbre of the speaker. Sound features can reflect the speaker's vocal tract structure, pronunciation habits, and other physiological and behavioral characteristics, and can be used to distinguish different speakers. In one or more embodiments of this specification, sound features can be extracted from the initial audio segments corresponding to each benchmark text segment using a pre-trained sound coding model. That is, each initial audio segment can be input into the pre-trained sound coding model to obtain the sound features corresponding to each initial audio segment output by the pre-trained sound coding model.

[0112] It should be noted that the pre-trained sound coding model can be selected based on the actual application scenario and actual needs, such as a model trained based on a deep neural network.

[0113] The preset similarity threshold can be understood as a critical similarity value used to determine whether two sound features belong to the same speaker. This preset similarity threshold can be set according to the actual application scenario and actual accuracy requirements; this manual does not impose specific restrictions.

[0114] In practical applications, the similarity between sound features can be calculated using cosine similarity, Euclidean distance, or other distance metrics. In a specific example, the cosine similarity between the sound features corresponding to any two reference text segments can be calculated. If the cosine similarity is greater than a preset similarity threshold, it is determined that the two reference text segments correspond to the same speaker.

[0115] In one or more embodiments of this specification, any two reference text segments whose corresponding sound features have a similarity greater than a preset similarity threshold can be marked as having the same pronunciation subject. Furthermore, the same subject identifier can be written into the basic information corresponding to the two reference text segments. Different pronunciation subject identifiers can be assigned to two or more reference text segments whose corresponding sound features have different similarities.

[0116] In a specific example, VAD can be used to detect valid speech segments first, and then sound features can be extracted for each valid speech segment. Next, a clustering method based on a preset similarity threshold can be used to group speech segments whose cosine similarity between sound features is greater than the preset similarity threshold into the same pronunciation subject cluster. Finally, the same subject identifier can be assigned to each pronunciation subject cluster to generate pronunciation subject identifiers corresponding to each benchmark text segment.

[0117] In one or more embodiments of this specification, the sound features of the initial audio segment in the initial audio corresponding to each reference text segment are determined; clustering is performed based on the sound features of each reference text segment to obtain at least one pronunciation subject cluster, wherein if the similarity between the sound features corresponding to any two reference text segments is greater than a preset similarity threshold, the two reference text segments are determined to belong to the same pronunciation subject cluster; a pronunciation subject identifier is assigned to each pronunciation subject cluster, and the pronunciation subjects corresponding to each reference text segment contained in the pronunciation subject cluster are determined as the pronunciation subject identifier.

[0118] The above method identifies the speaker and text segments with similar speakers by extracting sound features and comparing similarities, providing a data foundation for speech synthesis and timbre consistency among multiple speakers.

[0119] In practical applications, when generating target audio segments for target text segments, a suitable speech synthesis strategy can be selected based on the amount of audio data available to different speakers. If the same synthesis method is used for all speakers, the synthesis quality of speakers with sufficient data may not be fully utilized, or the synthesis effect of speakers with insufficient data may be poor due to forced fine-tuning. Therefore, in one or more embodiments of this specification, the target audio segment corresponding to the target text segment is obtained through the following method: For each speaker, calculate the sum of the first durations of all target text segments corresponding to that speaker; When the total is greater than the second preset duration, the preset speech synthesis model is fine-tuned based on the initial audio segments of the target text segments corresponding to the speech subject in the initial audio to obtain the target speech synthesis model. The target text segments corresponding to the speech subject are then input into the target speech synthesis model to obtain the target audio segments of the target text segments corresponding to the speech subject output by the target speech synthesis model. If the total duration is less than or equal to the second preset duration, each target text segment corresponding to the pronunciation subject and the initial audio segment of each target text segment in the initial audio are input into the preset speech synthesis model to obtain the target audio segments of each target text segment corresponding to the pronunciation subject output by the preset speech synthesis model.

[0120] The second preset duration can be understood as a threshold used to determine whether the amount of audio data corresponding to a speaker meets the preset duration threshold for fine-tuning the speech synthesis model. The second preset duration can be set according to the fine-tuning requirements of the preset speech synthesis model, for example, it can be set to 300 seconds, 600 seconds, or 1200 seconds, etc. If the total amount of audio data corresponding to a certain speaker (i.e., the sum of the first durations corresponding to each target text segment) is greater than the second preset duration, it indicates that there is sufficient data to model the speaker's voice, thus allowing for fine-tuning of the preset speech synthesis model before speech synthesis.

[0121] The preset speech synthesis model can be understood as a pre-trained general speech synthesis model that has the basic ability to convert text into speech. In one or more embodiments of this specification, the preset speech synthesis model can support various speech synthesis functions, such as direct text-to-speech synthesis, or voice cloning generation based on reference audio.

[0122] The target speech synthesis model can be understood as a speech synthesis model fine-tuned for a specific speaker. By using the initial audio segment corresponding to the speaker to fine-tune the preset speech synthesis model, the fine-tuned target speech synthesis model can more accurately capture the speaker's timbre characteristics, pronunciation habits, and prosodic patterns, thereby generating a more natural synthesized audio segment that better matches the speaker's timbre.

[0123] In practical applications, the initial audio segment used in the fine-tuning process can be preprocessed. Preprocessing operations can include removing background noise, removing silent segments, normalizing volume, etc. The number of training epochs and the learning rate for fine-tuning can be adjusted according to the amount of audio data corresponding to a specific speaker; the larger the amount of data, the more training epochs can be increased.

[0124] In one or more embodiments of this specification, for a zero-sample synthesis strategy, that is, without fine-tuning the speech synthesis model, an initial audio segment corresponding to a specific speaker can be used as a reference audio and input into a preset speech synthesis model. The preset speech synthesis model can extract timbre features from the reference audio and generate a target audio segment with a timbre similar to that of the reference audio based on the extracted timbre features.

[0125] By selecting either a fine-tuning model and synthesis strategy or a zero-sample synthesis strategy based on the amount of audio data from the speaker, the synthesis quality can be improved by fully utilizing personalized fine-tuning when there is sufficient audio data, and the feasibility of synthesis can be ensured by utilizing zero-sample capability when there is insufficient audio data, thus achieving a balance between synthesis effect and data utilization efficiency.

[0126] In one or more embodiments of this specification, when synthesizing a corrected text segment into a corrected audio segment, different speech synthesis models exhibit differences in synthesis quality and efficiency. For longer audio segments, a finely tuned target speech synthesis model can achieve better synthesis results, while for shorter audio segments, a preset speech synthesis model is more efficient. Therefore, in one or more embodiments of this specification, updating the target audio segment in the basic information to be modified based on the corrected text segment to obtain corrected basic information includes: When the second duration corresponding to the target audio segment is greater than the fourth preset duration, the target speech synthesis model corresponding to the speaker of the target audio segment is obtained, and the corrected text segment is input into the target speech synthesis model to obtain the corrected audio segment corresponding to the corrected text segment generated by the target speech synthesis model. If the second duration corresponding to the target audio segment is less than or equal to the fourth preset duration, the corrected text segment is input into the preset speech synthesis model to obtain the corrected audio segment corresponding to the corrected text segment generated by the preset speech synthesis model. Replace the target audio segment in the basic information to be modified with the corrected audio segment to obtain the corrected basic information.

[0127] The fourth preset duration can be understood as a duration threshold used to select the speech synthesis model. This fourth preset duration can be set according to the actual application scenario and actual needs; this manual does not impose specific restrictions. If the second duration of the target audio segment is greater than the fourth preset duration, it indicates that the target audio segment is more important or has richer content; therefore, a fine-tuned target speech synthesis model can be used for correction to improve the correction quality. If the second duration is less than or equal to the fourth preset duration, it indicates that the target audio segment is shorter; therefore, a preset speech synthesis model can be used to meet the quality requirements while saving computational resources.

[0128] The target speech synthesis model can be understood as a speech synthesis model fine-tuned for a specific speaker, capable of generating high-quality synthesized speech that better matches the timbre characteristics of that speaker. The preset speech synthesis model can be understood as a general-purpose speech synthesis model with zero-sample timbre cloning capability, applicable to the rapid synthesis of various speakers.

[0129] In one or more embodiments of this specification, the setting of the fourth preset duration can be associated with the second preset duration. For example, the second preset duration is used to determine whether the amount of audio data corresponding to the speech subject is sufficient for model fine-tuning, and the fourth preset duration is used to further determine whether a specific audio segment is worth using the fine-tuning model when the fine-tuning speech synthesis model is available. Therefore, the fourth preset duration can be set to be less than or equal to the second preset duration.

[0130] In one or more embodiments of this specification, if a certain speech subject does not have a corresponding target speech synthesis model, that is, the amount of audio data of the speech subject is insufficient for model fine-tuning, then regardless of whether the second duration is greater than the fourth preset duration, the preset speech synthesis model is required to synthesize the corrected audio segment.

[0131] In practical applications, after modifying the text segment to be modified and obtaining the corrected audio segment, it is necessary to re-synthesize the corrected audio segment corresponding to the corrected text segment. In this process, it can be checked whether the second duration of the target audio segment corresponding to the text segment to be modified is greater than a preset threshold. If so, the fine-tuned target speech synthesis model corresponding to the speaker of the text segment to be modified is called to regenerate and synthesize the corrected audio segment corresponding to the corrected text segment. If not, the preset zero-sample speech synthesis model can be called to quickly synthesize the corrected audio segment corresponding to the corrected text segment.

[0132] The above method allows for the selection of either the target speech synthesis model or a preset speech synthesis model for synthesizing corrected audio based on the second duration, thus balancing synthesis quality and computational efficiency.

[0133] In one or more embodiments of this specification, after generating the target audio segment, due to the expression differences between the initial language of the initial audio segment and the target language of the target audio segment, the second duration of the target audio segment may deviate significantly from the first duration of its corresponding initial audio segment. If the duration deviation is too large, the translated audio will be out of sync with the original video. Therefore, the duration of the target audio segment can be adapted and aligned with the duration of the initial audio segment. Thus, in one or more embodiments of this specification, before obtaining the basic information to be modified corresponding to the text segment to be modified and the corrected text segment, the method further includes: For each initial text segment, if the difference between the first duration and the second duration corresponding to the initial text segment is greater than the third preset duration, the duration of the target audio segment corresponding to the initial text segment is adjusted with the goal of aligning the first duration and the second duration corresponding to the initial text segment.

[0134] The third preset duration can be understood as a duration threshold used to determine whether the deviation between the first and second durations needs adjustment. This third preset duration can be set according to the actual application scenario and actual needs, such as 0.5 seconds, 1 second, or 2 seconds; this manual does not impose specific restrictions. If the difference between the first and second durations is greater than the third preset duration, it indicates that there is a significant deviation between the pronunciation duration of the synthesized audio obtained after translation (i.e., the target audio segment) and the pronunciation duration of the initial audio segment, and therefore adjustment is required.

[0135] In one or more embodiments of this specification, aligning the first and second durations corresponding to the initial text segment can be understood as making the first and second durations the same or approximately the same. Correspondingly, duration adjustment can be understood as processing the second duration of the target audio segment to make it closer to the first duration. Duration adjustment can be achieved in various ways, such as audio acceleration or deceleration, insertion or deletion of silence segments, or trimming or lengthening of audio segments. In a specific example, duration adjustment can be implemented based on a time-stretching algorithm, specifically by changing the playback speed of the target audio segment without changing its pitch. This time-stretching algorithm can specifically take the form of a phase vocoder.

[0136] In one or more embodiments of this specification, the duration adjustment can be implemented in different ways depending on whether the difference is positive or negative. Specifically, when the second duration is longer than the first duration, the target audio segment can be moderately sped up; when the second duration is shorter than the first duration, a silence segment can be inserted or the target audio segment can be moderately slowed down at a designated position.

[0137] By detecting and adjusting the deviation between the first and second durations using the above method, we can ensure the consistency of the pronunciation duration between the synthesized audio (i.e., the target audio segment) and the initial audio segment. This provides audio data with consistent duration for subsequent text modification operations and target audio synthesis operations, avoiding audio-visual desynchronization problems caused by duration mismatch.

[0138] In practical applications, after completing audio translation and speech synthesis, it is necessary to evaluate the quality of the generated target audio to determine whether the translated target audio meets the deliverable quality standards. Therefore, in one or more embodiments of this specification, the method further includes: Before obtaining the basic information to be modified and the corrected text fragment corresponding to the text fragment to be modified, the first reference information corresponding to the initial audio is determined. The reference information includes a fourth duration, a first quantity, a second quantity, and a third quantity. The fourth duration is the sum of the second durations of each target audio fragment corresponding to the initial audio. The first quantity is the number of the same first duration and second duration. The second quantity is the number of target audio fragments that are missing a specified voice frame compared to the corresponding initial audio fragment. The third quantity is the number of target audio fragments whose number of silent voice frames corresponding to the initial audio is greater than the preset number of voice frames. After obtaining the corrected audio segment, the second reference information corresponding to the initial audio is re-determined based on the corrected audio segment; If the difference between the second reference information and the first reference information meets a preset difference condition, the target audio is determined to have been successfully generated.

[0139] The first reference information can be understood as quality assessment metrics for each target audio segment corresponding to the initial audio segment before text modification. The fourth duration reflects the total duration of all target audio segments, used to assess whether the duration of the translated target audio segments matches the original audio. The first quantity reflects the number of identified speech entities, used to assess the completeness of the speech entities. The second quantity reflects the number of target text segments, used to assess translation coverage. The third quantity reflects the number of initial text segments, used to assess the completeness of text segmentation.

[0140] The second reference information can be understood as the quality evaluation index data corresponding to each synthesized audio segment (including the corrected audio segment and other target audio segments excluding the target audio segment corresponding to the corrected audio segment) re-statistically after the text modification to obtain the corrected audio segment. It is understood that the second reference information contains the same dimensional indicators as the first reference information, only representing reference information corresponding to audio segments at different stages.

[0141] The preset difference conditions can be understood as rules used to determine whether the difference between two sets of reference information is within an acceptable range. These preset difference conditions may include differences in various dimensional indicators not exceeding corresponding thresholds, such as the fourth duration changing by no more than 5%, the first quantity remaining unchanged, the second quantity remaining unchanged, and the third quantity remaining unchanged, etc. Understandably, by comparing the differences between the second and first reference information, it is possible to assess whether any abnormal changes were introduced after the text modification.

[0142] In one or more embodiments of this specification, if the difference between the second reference information and the first reference information meets a preset difference condition, it can be determined that the target audio was successfully generated. After determining that the target audio was successfully generated, a quality statistics report can also be output. The quality statistics report can include the indicator values ​​of each dimension of the first reference information and the second reference information and their differences, so that relevant personnel can quickly obtain the quality status of the target audio.

[0143] In one or more embodiments of this specification, if the difference between the first reference information and the second reference information does not meet the preset difference condition, it can be determined that the target audio generation has failed, and the text modification operation can be reverted to the state before the correction and re-executed. Specifically, the corrected text fragment is re-input into the preset language synthesis model to regenerate the corrected audio fragment, and the target audio is regenerated.

[0144] The above method involves statistically analyzing and comparing reference information before and after correction. If the difference between the corrected and uncorrected reference information meets a preset difference condition, it indicates that the text modification operation did not introduce any anomalies, and the target audio was successfully generated. If the difference does not meet the preset difference condition, it indicates that the text modification operation may have caused problems such as audio data loss, abnormal duration, or changes in the speaker's information, requiring further inspection or rollback. Therefore, by statistically analyzing and comparing reference information before and after correction, translation quality can be evaluated and anomaly detection can be achieved, ensuring the integrity and reliability of the generated target audio.

[0145] Based on the aforementioned audio processing methods, video translation tasks can be modeled as a two-stage task model. Specifically, the first stage is the translation stage, which sequentially executes operations such as text acquisition, translation, audio preprocessing, speech reconstruction, duration adaptation, mixing, and output write-back, thereby generating initial basic information for each initial text segment, including the target text segment and the target audio segment. The second stage is the correction stage, which receives the modified text segment for the text segment to be modified and reuses the initial basic information from the first stage. This allows skipping video parsing, voice separation, speaker analysis, and translation, and only resynthesizing the text segments affected by the modification. The initial basic information can include a segmentation boundary table, a speaker merging table, references based on the speaker's vocal model, an accompaniment track, a timeline table, and quality statistics. Through this two-stage reuse mechanism, the correction stage does not need to redo segmentation and translation, reducing computational overhead and improving correction efficiency in interactive correction scenarios.

[0146] The following example, using the audio processing methods provided in this manual in a video translation scenario, further illustrates the audio processing methods.

[0147] First, the server can receive the initial video to be translated and extract the initial audio from it. Specifically, the user uploads the initial video to be translated via the client; this initial video is an English instructional video. After receiving the initial video, the server extracts the audio track from the video file as the initial audio. The initial audio is in English.

[0148] Then, speech recognition can be performed on the initial audio to obtain the initial text. Specifically, an ASR model is used to perform speech recognition on the initial audio, converting the speech content in the audio into initial text. The ASR model can be an end-to-end speech recognition model, such as a model based on the Transformer architecture.

[0149] Next, speech activity detection can be performed on the initial audio to determine the temporal boundaries of each speech segment. Specifically, VAD can be used to perform endpoint detection on the initial audio with a frame accuracy of 10ms to determine the start and end times of each speech segment in the audio. The VAD results are used for subsequent text segmentation and duration calculation.

[0150] Next, the initial text can be divided into multiple baseline text segments, and the speaker of each baseline text segment can be determined. Specifically, the initial text is divided into multiple baseline text segments based on the speech pauses detected by VAD. The sound features of the corresponding audio segment are extracted for each baseline text segment, and the speaker of each baseline text segment is determined by voiceprint similarity clustering. The sound features are extracted using a speaker coding model, and segments with a similarity greater than a preset threshold, such as 0.75, are grouped as belonging to the same speaker.

[0151] Next, reference text segments with pronunciation durations shorter than a first preset duration can be merged to obtain the initial text segment. Specifically, the pronunciation duration of each reference text segment in the initial audio is checked. For reference text segments to be processed with pronunciation durations shorter than the first preset duration, such as 2 seconds, adjacent reference text segments with the same pronunciation subject are obtained and merged. After merging, the pronunciation duration is recalculated. If it is still shorter than the first preset duration, merging continues until the condition is met or merging can no longer continue. The merged text segment and the reference text segments with pronunciation durations greater than or equal to the first preset duration are used as the initial text segment.

[0152] Furthermore, for each speaker, the sum of the first durations of its corresponding initial text segments can be calculated, and a speech synthesis strategy can be selected based on this sum. Specifically, for each speaker, the sum of the first durations of all its initial text segments—that is, the total duration of their pronunciation in the initial audio—is calculated. If the sum is greater than a second preset duration, such as 300 seconds, the preset speech synthesis model is fine-tuned based on the initial audio segments of that speaker to obtain a target speech synthesis model specific to that speaker. If the sum is less than or equal to the second preset duration, a zero-sample synthesis strategy is adopted, and the preset speech synthesis model is used directly.

[0153] Furthermore, for each initial text fragment, the translation information and the initial text fragment can be input into a preset language model to obtain intermediate text fragments. Specifically, a numbered input method can be used, where each initial text fragment is sequentially numbered and then input into the preset language model along with the translation information. The preset language model then generates the corresponding intermediate text fragment for each initial text fragment based on the translation information, such as English to Chinese translation.

[0154] Furthermore, the intermediate text segment can be quality-checked to determine the target text segment. Specifically, it checks whether the characters between the intermediate and initial text segments meet preset character conditions, including line count consistency checks and character quantity ratio checks. If the preset character conditions are met, the intermediate text segment is determined as the target text segment. If not, the initial text segment is re-input into the preset language model for retry. If the number of retries exceeds a threshold, it is downgraded to a machine translation interface.

[0155] Then, basic information corresponding to each initial text segment can be generated. Specifically, structured basic information is generated for each initial text segment, including the target text segment, the target audio segment, the first duration, the second duration, and the speaker identifier. The target audio segment is obtained by speech synthesis from the target text segment.

[0156] Next, the difference between the first and second durations can be detected, and the duration of target audio segments with a deviation exceeding a third preset duration can be adjusted. Specifically, for each initial text segment, the difference between the first and second durations is calculated. If the absolute value of the difference is greater than the third preset duration, such as 0.5 seconds, the duration of the target audio segment is adjusted. When the second duration is longer than the first duration, acceleration is performed; when the second duration is shorter than the first duration, deceleration or additional silence is added.

[0157] Next, the first reference information corresponding to the initial audio can be determined. Then, if the text segment to be modified in the initial text is modified, the corrected text segment is obtained and the audio is resynthesized to obtain the corrected audio segment. Specifically, the first reference information before correction is statistically analyzed, including the fourth duration (the sum of the second durations of all target audio segments), the first quantity (the number of speech subjects), the second quantity (the number of target text segments), and the third quantity (the number of initial text segments). The first reference information serves as the baseline data for quality assessment. The user modifies a translated text (the text segment to be modified) through the client, inputting the corrected text segment. The server obtains the basic information to be modified and resynthesizes the audio based on the corrected text segment. If the second duration of the target audio segment is greater than the fourth preset duration, the target speech synthesis model corresponding to the speech subject is used for synthesis; otherwise, the preset speech synthesis model is used. The synthesis result is the corrected audio segment.

[0158] Finally, target audio can be generated based on the corrected audio segments and each target audio segment, and its quality can be assessed. Specifically, the corrected audio segments can replace the original target audio segments and be sequentially concatenated with other unmodified target audio segments to generate complete target audio. Smoothing is performed between adjacent segments during concatenation to ensure audio continuity. Then, the second reference information is re-statistically analyzed and compared with the first reference information. If the differences meet preset conditions (changes in each indicator are within threshold ranges), the target audio is considered successfully generated. The target audio and target text can then be updated into the initial video to obtain the target video.

[0159] The method described above enables a complete processing flow in video translation scenarios, from audio extraction, speech recognition, text translation, speech synthesis to post-translation correction. By managing audio, text, and speaker information through a unified structured intermediate format, post-translation correction only requires resynthesizing the audio of the affected segment, eliminating the need to redo the entire process, thus significantly improving translation efficiency and audio quality.

[0160] Corresponding to the above method embodiments, this specification also provides method embodiments for audio processing applied to cloud-side devices. Figure 3 A flowchart of an audio processing method for a cloud-side device according to an embodiment of this specification is shown, which specifically includes the following steps.

[0161] Step 302: Receive an audio processing request sent by the receiving end device, wherein the audio processing request carries initial audio; It is understandable that edge devices have limited computing resources and storage capabilities, making it difficult for them to independently complete large-scale processing tasks such as speech recognition, translation, and speech synthesis. Therefore, audio processing tasks can be deployed on cloud-side devices to leverage their computing resources and model service capabilities to provide audio processing capabilities to edge devices.

[0162] Among them, the audio processing request is a processing request sent by the end device to the cloud device. It can carry the initial audio to be processed. In practical applications, the audio processing request can also carry translation information, speech subject configuration information, etc.

[0163] Step 304: Obtain the initial text corresponding to the initial audio, and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on the translation information, and the target audio segment is synthesized based on the corresponding target text segment.

[0164] Step 306: When modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, and update the target audio segment in the basic information to be modified based on the corrected text segment to obtain the corrected basic information.

[0165] Step 308: Generate the target audio corresponding to the initial audio based on the corrected basic information and the initial basic information.

[0166] Step 310: Send the target audio to the end-side device.

[0167] In the aforementioned audio processing method applied to cloud-side devices, the cloud-side device obtains the initial text corresponding to the initial audio and generates basic information corresponding to at least two initial text segments contained in the initial text. This unifies audio, text, and audio processing within a structured framework of basic information, providing a unified data foundation for subsequent text translation and speech synthesis, avoiding data inconsistencies caused by independent processing at each stage. When text in the audio needs modification, the aforementioned generated basic information can be used directly for modification. Specifically, the target audio segment in the basic information to be modified is updated based on the corrected text segment. Thus, in the case of partial text modification, speech synthesis is performed only on the affected text segment without re-executing the entire process, reducing computational costs and processing time in text modification scenarios and improving audio generation efficiency. Furthermore, by generating target audio based on the corrected audio segment and each target audio segment, the audio continuity between the corrected and uncorrected parts during splicing is ensured, avoiding discontinuous or abnormal sound quality at splicing points, thereby guaranteeing the quality and naturalness of the generated target audio.

[0168] The above is an illustrative scheme of an audio processing method applied to a cloud-side device according to this embodiment. It should be noted that the technical solution of this audio processing method applied to a cloud-side device belongs to the same concept as the audio processing method described above. For details not described in detail in the technical solution of the audio processing method applied to a cloud-side device, please refer to the description of the technical solution of the audio processing method described above.

[0169] Corresponding to the above method embodiments, this specification also provides embodiments of audio processing apparatus. Figure 4 A schematic diagram of an audio processing apparatus according to one embodiment of this specification is shown. Figure 4 As shown, the device includes: The first acquisition module 402 is configured to acquire the initial text corresponding to the initial audio and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. The first update module 404 is configured to, when modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, update the target audio segment in the basic information to be modified based on the corrected text segment, and obtain the corrected basic information. The first generation module 406 is configured to generate the target audio corresponding to the initial audio based on the corrected basic information and the initial basic information.

[0170] Optionally, the first acquisition module 402 is further configured to, for each initial text segment, input the translation information and the initial text segment into a preset language model to obtain an intermediate text segment corresponding to the initial text segment generated by the preset language model; if the characters between the initial text segment and the intermediate text segment meet preset character conditions, determine the intermediate text segment as the target text segment corresponding to the initial text segment; if the characters between the initial text segment and the intermediate text segment do not meet the preset character conditions, re-input the initial text segment into the preset language model.

[0171] Optionally, the initial basic information also includes a first duration, a second duration, and a pronunciation subject corresponding to the initial text segment. The first duration is the pronunciation duration of the initial audio segment corresponding to the initial text segment in the initial audio, and the second duration is the duration of the target audio segment corresponding to the initial text segment.

[0172] Optionally, the first acquisition module 402 is further configured to divide the initial text into at least two reference text segments and determine the pronunciation subject corresponding to each reference text segment; if the pronunciation duration of the reference text segment to be processed in the initial audio is less than a first preset duration, acquire an associated reference text segment with the same pronunciation subject as the reference text segment to be processed, merge the reference text segment to be processed and the associated reference text segment to obtain a merged text segment, wherein the reference text segment to be processed is any one reference text segment; and use each merged text segment and the reference text segment with a pronunciation duration greater than or equal to the first preset duration as the initial text segment.

[0173] Optionally, the first acquisition module 402 is further configured to determine the sound features of the initial audio segment in the initial audio corresponding to each reference text segment; if the similarity between the sound features corresponding to any two reference text segments is greater than a preset similarity threshold, determine that the two reference text segments correspond to the same pronunciation subject, and generate the pronunciation subject corresponding to each reference text segment.

[0174] Optionally, the first acquisition module 402 is further configured to: calculate the sum of the first durations of each target text segment corresponding to each pronunciation subject; if the sum is greater than a second preset duration, fine-tune a preset speech synthesis model based on the initial audio segments of each target text segment corresponding to the pronunciation subject in the initial audio to obtain a target speech synthesis model, and input each target text segment corresponding to the pronunciation subject into the target speech synthesis model to obtain the target audio segments of each target text segment corresponding to the pronunciation subject output by the target speech synthesis model; if the sum is less than or equal to the second preset duration, input each target text segment corresponding to the pronunciation subject and the initial audio segments of each target text segment in the initial audio into the preset speech synthesis model to obtain the target audio segments of each target text segment corresponding to the pronunciation subject output by the preset speech synthesis model.

[0175] Optionally, the first update module 404 is further configured to: when the second duration corresponding to the target audio segment is greater than the fourth preset duration, obtain the target speech synthesis model corresponding to the speaker of the target audio segment, and input the corrected text segment into the target speech synthesis model to obtain the corrected audio segment corresponding to the corrected text segment generated by the target speech synthesis model; when the second duration corresponding to the target audio segment is less than or equal to the fourth preset duration, input the corrected text segment into the preset speech synthesis model to obtain the corrected audio segment corresponding to the corrected text segment generated by the preset speech synthesis model; and replace the target audio segment in the basic information to be modified with the corrected audio segment to obtain the corrected basic information.

[0176] Optionally, the first acquisition module 402 is further configured to, for each initial text segment, adjust the duration of the target audio segment corresponding to the initial text segment with the goal of aligning the first duration and the second duration corresponding to the initial text segment, if the difference between the first duration and the second duration corresponding to the initial text segment is greater than a third preset duration.

[0177] Optionally, the first generation module 406 is further configured to obtain the initial video corresponding to the initial audio and the target text corresponding to the target audio; update the target text and the target audio into the initial video to obtain the target video.

[0178] Optionally, the first generation module 406 is further configured to, before obtaining the basic information to be modified corresponding to the text segment to be modified and the corrected text segment, determine the first reference information corresponding to the initial audio, wherein the reference information includes a fourth duration, a first quantity, a second quantity, and a third quantity, the fourth duration being the sum of the second durations of each target audio segment corresponding to the initial audio, the first quantity being the number of the same first duration and second duration, the second quantity being the number of target audio segments missing a specified voice frame compared to the corresponding initial audio segment, and the third quantity being the number of target audio segments whose number of silent voice frames corresponding to the initial audio is greater than a preset number of voice frames; after obtaining the corrected audio segment, redetermine the second reference information corresponding to the initial audio based on the corrected audio segment; and determine that the target audio generation is successful if the difference between the second reference information and the first reference information satisfies a preset difference condition.

[0179] The aforementioned audio processing apparatus acquires the initial text corresponding to the initial audio and generates basic information corresponding to at least two initial text segments contained in the initial text. This unifies audio, text, and audio processing within a structured framework of basic information, providing a unified data foundation for subsequent text translation and speech synthesis, avoiding data inconsistencies caused by independent processing at each stage. When text in the audio needs modification, the aforementioned generated basic information can be used directly for modification. Specifically, the target audio segment in the basic information to be modified is updated based on the corrected text segment. Thus, in the case of partial text modification, speech synthesis is performed only on the affected text segment, without re-executing the entire process, reducing computational costs and processing time in text modification scenarios and improving audio generation efficiency. Furthermore, by generating target audio based on the corrected audio segment and each target audio segment, the audio continuity between the corrected and uncorrected parts during splicing is ensured, avoiding discontinuities or abnormalities in sound quality at splicing points, thereby guaranteeing the quality and naturalness of the generated target audio.

[0180] The above is an illustrative embodiment of an audio processing apparatus according to this invention. It should be noted that the technical solution of this audio processing apparatus and the technical solution of the aforementioned audio processing method belong to the same concept. Details not described in detail in the technical solution of the audio processing apparatus can be found in the description of the technical solution of the aforementioned audio processing method.

[0181] Corresponding to the above method embodiments, this specification also provides an embodiment of an audio processing apparatus located on a cloud-side device. Figure 5 A schematic diagram of an audio processing apparatus located on a cloud-side device according to one embodiment of this specification is shown. Figure 5 As shown, the device includes: The receiving module 502 is configured to receive an audio processing request sent by the end-side device, wherein the audio processing request carries initial audio. The second acquisition module 504 is configured to acquire the initial text corresponding to the initial audio and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. The second update module 506 is configured to, when modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, update the target audio segment in the basic information to be modified based on the corrected text segment, and obtain the corrected basic information. The second generation module 508 is configured to generate the target audio corresponding to the initial audio based on the corrected basic information and the initial basic information; The sending module 510 is configured to send the target audio to the end-side device.

[0182] The aforementioned audio processing device located on the cloud side acquires the initial text corresponding to the initial audio and generates basic information corresponding to at least two initial text segments contained in the initial text. This unifies audio, text, and audio processing within a structured framework of basic information, providing a unified data foundation for subsequent text translation and speech synthesis, avoiding data inconsistencies caused by independent processing at each stage. When text in the audio needs modification, the aforementioned generated basic information can be used directly for modification. Specifically, the target audio segment in the basic information to be modified is updated based on the corrected text segment. Thus, in the case of partial text modification, speech synthesis is performed only on the affected text segment, without re-executing the entire process, reducing computational costs and processing time in text modification scenarios and improving audio generation efficiency. Furthermore, by generating target audio based on the corrected audio segment and each target audio segment, the audio continuity between the corrected and uncorrected parts during splicing is ensured, avoiding discontinuities or abnormalities in sound quality at splicing points, thereby guaranteeing the quality and naturalness of the generated target audio.

[0183] The above is an illustrative scheme of an audio processing apparatus located on a cloud-side device according to this embodiment. It should be noted that the technical solution of this audio processing apparatus located on a cloud-side device and the technical solution of the audio processing method applied to a cloud-side device described above belong to the same concept. For details not described in detail in the technical solution of the audio processing apparatus located on a cloud-side device, please refer to the description of the technical solution of the audio processing method applied to a cloud-side device described above.

[0184] Figure 6 A structural block diagram of a computing device 600 provided according to one embodiment of this specification is shown.

[0185] The computing device 600 includes: Memory 610 and processor 620; The memory 610 is used to store computer programs / instructions, and the processor 620 is used to execute the computer programs / instructions, which, when executed by the processor 620, implement the steps of the audio processing method.

[0186] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0187] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0188] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0189] Figure 7 A structural block diagram of an electronic device 700 provided according to one embodiment of this specification is shown.

[0190] A memory 710 and a processor 720 are connected via a bus 730; The memory 710 is used to store computer programs / instructions, and the processor 720 is used to execute the computer programs / instructions, which, when executed by the processor 720, implement the steps of the method.

[0191] Specifically, the components of the electronic device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0192] Electronic device 700 also includes access device 740, which enables electronic device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 740 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0193] In one embodiment of this specification, the above-described components of the electronic device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0194] Electronic device 700 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 700 can also be a mobile or stationary server.

[0195] The above is an illustrative embodiment of an electronic device according to this invention. It should be noted that the technical solution of this electronic device and the technical solution of the audio processing method described above belong to the same concept. Details not described in detail in the technical solution of the electronic device can be found in the description of the technical solution of the audio processing method described above.

[0196] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method.

[0197] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the audio processing method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the audio processing method described above.

[0198] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method.

[0199] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the audio processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the audio processing method described above.

[0200] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0201] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0202] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0203] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0204] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An audio processing method, comprising: Obtain the initial text corresponding to the initial audio, and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. When modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, and update the target audio segment in the basic information to be modified based on the corrected text segment to obtain the corrected basic information; Based on the corrected basic information and the initial basic information, the target audio corresponding to the initial audio is generated.

2. The method as described in claim 1, wherein the target text segment corresponding to the initial text segment is obtained in the following manner: For each initial text fragment, the translation information and the initial text fragment are input into a preset language model to obtain the intermediate text fragment corresponding to the initial text fragment generated by the preset language model; If the characters between the initial text segment and the intermediate text segment meet a preset character condition, the intermediate text segment is determined to be the target text segment corresponding to the initial text segment; If the characters between the initial text fragment and the intermediate text fragment do not meet the preset character conditions, the initial text fragment is re-inputted into the preset language model.

3. The method as described in claim 1, wherein the initial basic information further includes a first duration, a second duration, and a pronunciation subject corresponding to the initial text segment, wherein the first duration is the pronunciation duration of the initial audio segment corresponding to the initial text segment in the initial audio, and the second duration is the duration of the target audio segment corresponding to the initial text segment.

4. The method of claim 3, wherein each initial text segment in the initial text is obtained in the following manner: The initial text is divided into at least two baseline text segments, and the pronunciation subject corresponding to each baseline text segment is determined; If the pronunciation duration of the reference text segment to be processed in the initial audio is less than a first preset duration, an associated reference text segment with the same pronunciation subject as the reference text segment to be processed is obtained. The reference text segment to be processed and the associated reference text segment are then merged to obtain a merged text segment. The reference text segment to be processed can be any reference text segment; Each merged text segment and the baseline text segment with a pronunciation duration greater than or equal to the first preset duration are used as the initial text segments.

5. The method as described in claim 4, wherein determining the pronunciation subject corresponding to each reference text segment includes: Determine the acoustic features of the initial audio segment in the initial audio corresponding to each benchmark text segment; If the similarity between the sound features corresponding to any two reference text segments is greater than a preset similarity threshold, it is determined that the two reference text segments correspond to the same pronunciation subject, and the pronunciation subject corresponding to each reference text segment is generated.

6. The method as described in claim 3, wherein the target audio segment corresponding to the target text segment is obtained by the following method: For each speaker, calculate the sum of the first durations of all target text segments corresponding to that speaker; When the total is greater than the second preset duration, the preset speech synthesis model is fine-tuned based on the initial audio segments of the target text segments corresponding to the speech subject in the initial audio to obtain the target speech synthesis model. The target text segments corresponding to the speech subject are then input into the target speech synthesis model to obtain the target audio segments of the target text segments corresponding to the speech subject output by the target speech synthesis model. If the total duration is less than or equal to the second preset duration, each target text segment corresponding to the pronunciation subject and the initial audio segment of each target text segment in the initial audio are input into the preset speech synthesis model to obtain the target audio segments of each target text segment corresponding to the pronunciation subject output by the preset speech synthesis model.

7. The method of claim 6, wherein updating the target audio segment in the basic information to be modified based on the modified text segment to obtain the modified basic information includes: When the second duration corresponding to the target audio segment is greater than the fourth preset duration, the target speech synthesis model corresponding to the speaker of the target audio segment is obtained, and the corrected text segment is input into the target speech synthesis model to obtain the corrected audio segment corresponding to the corrected text segment generated by the target speech synthesis model. If the second duration corresponding to the target audio segment is less than or equal to the fourth preset duration, the corrected text segment is input into the preset speech synthesis model to obtain the corrected audio segment corresponding to the corrected text segment generated by the preset speech synthesis model. Replace the target audio segment in the basic information to be modified with the corrected audio segment to obtain the corrected basic information.

8. The method as described in claim 3, further comprising, before obtaining the basic information to be modified and the corrected text fragment corresponding to the text fragment to be modified, the method further comprising: For each initial text segment, if the difference between the first duration and the second duration corresponding to the initial text segment is greater than the third preset duration, the duration of the target audio segment corresponding to the initial text segment is adjusted with the goal of aligning the first duration and the second duration corresponding to the initial text segment.

9. The method according to any one of claims 1 to 8, wherein the method further comprises: Obtain the initial video corresponding to the initial audio and obtain the target text corresponding to the target audio; The target text and the target audio are updated into the initial video to obtain the target video.

10. The method of claim 7, further comprising: Before obtaining the basic information to be modified and the corrected text fragment corresponding to the text fragment to be modified, the first reference information corresponding to the initial audio is determined. The reference information includes a fourth duration, a first quantity, a second quantity, and a third quantity. The fourth duration is the sum of the second durations of each target audio fragment corresponding to the initial audio. The first quantity is the number of the same first duration and second duration. The second quantity is the number of target audio fragments that are missing a specified voice frame compared to the corresponding initial audio fragment. The third quantity is the number of target audio fragments whose number of silent voice frames corresponding to the initial audio is greater than the preset number of voice frames. After obtaining the corrected audio segment, the second reference information corresponding to the initial audio is re-determined based on the corrected audio segment; If the difference between the second reference information and the first reference information meets a preset difference condition, the target audio is determined to have been successfully generated.

11. An audio processing method applied to a cloud-side device, comprising: The receiving end device sends an audio processing request, wherein the audio processing request carries initial audio; Obtain the initial text corresponding to the initial audio, and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. When modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, and update the target audio segment in the basic information to be modified based on the corrected text segment to obtain the corrected basic information; Based on the corrected basic information and the initial basic information, the target audio corresponding to the initial audio is generated; The target audio is sent to the end-side device.

12. An audio processing apparatus, comprising: The first acquisition module is configured to acquire the initial text corresponding to the initial audio and generate initial basic information corresponding to at least two initial text segments. The at least two initial text segments are obtained based on the initial text. The initial basic information includes the target text segment and the target audio segment corresponding to the initial text segment. The target text segment is generated by translating the initial text segment based on translation information, and the target audio segment is synthesized based on the corresponding target text segment. The first update module is configured to, when modifying the text segment to be modified in the initial text, obtain the basic information to be modified and the corrected text segment corresponding to the text segment to be modified, update the target audio segment in the basic information to be modified based on the corrected text segment, and obtain the corrected basic information; The first generation module is configured to generate the target audio corresponding to the initial audio based on the corrected basic information and the initial basic information.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

14. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

16. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images