A method and apparatus for audio processing

Abstract

Embodiments of the present specification provide a method and apparatus for audio processing, wherein the method for audio processing comprises: obtaining an initial audio containing at least one difference audio segment, wherein the difference audio segment represents an audio segment of the initial audio corresponding to a difference in text relative to a standard text; constructing a to-be-processed word sequence based on word sequences corresponding to each difference audio segment and at least one non-difference audio segment; inputting the to-be-processed word sequence into a target audio processing model to obtain a corrected audio segment corresponding to each difference audio segment output by the target audio processing model; and updating the initial audio according to each corrected audio segment to obtain a target audio corresponding to the initial audio. This method can improve the efficiency of audio correction and save computing resources, improve the accuracy of correction, ensure that the generated corrected audio segment can be consistent with the non-difference audio segment in terms of timbre, prosody, etc., and improve the overall naturalness and coherence of the target audio.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to an audio processing method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence technology, various models have emerged and are widely used in different fields. For example, in the field of video generation, video generation models can be used to generate high-quality video footage. In particular, the process of generating video footage often requires the use of Text-to-Speech (TTS) technology to synthesize the audio corresponding to the video, thus forming complete audiovisual content. However, due to the limitations of TTS models' stability, misreadings and omissions are inevitable in the synthesized audio, resulting in low-quality generated video and affecting user experience. Therefore, it is necessary to repair the erroneous audio.

[0003] Currently, the process of repairing erroneous audio typically involves re-speech synthesis, specifically using a TTS model to re-synthesize the text corresponding to the erroneous audio. This can be achieved by changing the random seed to try and obtain error-free synthesized audio. However, the regenerated audio requires manual listening to determine if the repair was successful, which is wasteful of human resources and inefficient. Furthermore, the regenerated audio often exhibits uncontrollable differences in timbre, rhythm, and speech rate compared to other parts of the original audio, resulting in an inconsistent overall listening experience after repair. Additionally, the duration of the resynthesized audio is difficult to control precisely, especially in applications requiring synchronization with video footage, such as video dubbing. Changes in the duration of the repaired audio can lead to audio-visual desynchronization. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an audio processing method. One or more embodiments of this specification also relate to an audio processing method applied to a cloud-side device, a method for training an audio processing model, an audio processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an audio processing method is provided, comprising: Obtain initial audio containing at least one differential audio segment, wherein the differential audio segment represents an audio segment whose text corresponds to the initial audio and differs from the standard text; Based on the word sequences corresponding to each differential audio segment and at least one non-differential audio segment, construct the word sequence to be processed; The sequence of words to be processed is input into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model; The initial audio is updated based on each corrected audio segment to obtain the target audio corresponding to the initial audio.

[0006] According to a second aspect of the embodiments of this specification, an audio processing method is provided, applied to a cloud-side device, comprising: Obtain an audio processing request sent by the end device, wherein the audio processing request carries an initial audio containing at least one differential audio segment, the differential audio segment representing an audio segment whose text corresponding to the initial audio differs from the standard text; Based on the word sequences corresponding to each differential audio segment and at least one non-differential audio segment, construct the word sequence to be processed; The sequence of words to be processed is input into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model; The initial audio is updated according to each corrected audio segment to obtain the target audio corresponding to the initial audio. The target audio is sent to the end-side device.

[0007] According to a third aspect of the embodiments of this specification, a method for training an audio processing model is provided, comprising: Obtain a sample word sequence, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio. The sample word sequence is input into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model. Each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence. The initial audio processing model is trained based on the masked lexical subsequences of each sample and the standard lexical subsequences of each sample masked lexical subsequence in the global lexical sequence of the sample, thereby obtaining the target audio processing model.

[0008] According to a fourth aspect of the embodiments of this specification, a method for training an audio processing model is provided, applied to a cloud-side device, comprising: Obtain the model training request sent by the end device, wherein the model training request carries the model identifier of the initial audio processing model; Obtain the sample word sequence corresponding to the initial audio processing model, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio. The sample word sequence is input into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model. Each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence. The initial audio processing model is trained based on the masked word subsequences of each sample and the standard word subsequences of each sample masked word subsequence in the global word sequence of the sample, thereby obtaining the target audio processing model and the model parameters of the target audio processing model. The model parameters of the target audio processing model are sent to the end-side device.

[0009] According to a fifth aspect of the embodiments of this specification, an audio processing apparatus is provided, comprising: The first acquisition module is configured to acquire initial audio containing at least one differential audio segment, wherein the differential audio segment represents an audio segment whose text corresponds to the initial audio and differs from the standard text. The first construction module is configured to construct the word sequence to be processed based on the word sequence corresponding to each differential audio segment and at least one non-differential audio segment. The first input module is configured to input the word sequence to be processed into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model. The first update module is configured to update the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio.

[0010] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned audio processing method and audio processing model training method.

[0011] According to a seventh aspect of the embodiments of this specification, an electronic device is provided, comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned audio processing method and audio processing model training method.

[0012] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method and the training method for the audio processing model.

[0013] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method and the training method for the audio processing model.

[0014] The audio processing method provided in one embodiment of this specification can correct discrepancies in the initial audio, avoiding the need for indiscriminate regeneration of the entire audio, thus improving generation efficiency and saving computational resources. By constructing a sequence of words to be processed based on the word sequences corresponding to each discrepancy and non-discrepancy audio segment, the target audio processing model can utilize the contextual information of the non-discrepancy audio segments when correcting the audio, thereby providing a reference for correcting the discrepancy audio segments. This improves the accuracy of the correction and allows control over the duration of the generated corrected audio segments, preventing audio-visual asynchrony caused by duration variations in practical applications. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-discrepancy audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-discrepancy audio segments in terms of timbre and rhythm, improving the overall naturalness and coherence of the target audio. Attached Figure Description

[0015] Figure 1 This is an architecture diagram of an audio processing system provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating an audio processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the structure of an audio processing model provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating an audio processing method applied to a cloud-side device, as provided in one embodiment of this specification. Figure 5 This is a flowchart illustrating a training method for an audio processing model provided in one embodiment of this specification; Figure 6This is a flowchart illustrating a training method for an audio processing model applied to a cloud-side device, as provided in one embodiment of this specification. Figure 7 This is a schematic diagram of the structure of an audio processing apparatus provided in one embodiment of this specification; Figure 8 This is a schematic diagram of an audio processing apparatus located on a cloud-side device according to one embodiment of this specification; Figure 9 This is a schematic diagram of the structure of a training device for an audio processing model provided in one embodiment of this specification; Figure 10 This is a schematic diagram of the structure of a training device for an audio processing model located on a cloud-side device, provided in one embodiment of this specification. Figure 11 This is a structural block diagram of a computing device provided in one embodiment of this specification; Figure 12 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0016] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0017] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0018] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0019] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0020] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0021] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0022] Text-to-Speech (TTS): TTS can be understood as an artificial intelligence technology that converts text information into audible audio signals. Specifically, it performs linguistic analysis and acoustic modeling on text information and generates audio waveforms corresponding to the text. TTS technology is widely used in voice assistants, video dubbing, audiobooks, and other fields.

[0023] Automatic Speech Recognition (ASR): ASR, also known as speech-to-text technology, can be understood as an artificial intelligence technology that converts audio signals into text information. Specifically, it can map the acoustic features in an audio waveform into a text sequence through the collaborative work of an acoustic model and a language model.

[0024] Forced Alignment: Forced alignment is an important technique in the field of speech recognition. It can be understood as a technique that, given the audio and its corresponding text content, automatically determines the start and end timestamps of each character or word on the audio timeline using an alignment algorithm. It establishes a temporal correspondence between a text sequence and an audio frame sequence. Simply put, a forced alignment algorithm can pinpoint the start and end times of each character, word, or phoneme in the audio; that is, it specifies when each character, word, or phoneme in the text begins and ends in the audio.

[0025] Token: A token can be understood as a discrete representation unit obtained after audio signals, text, or images have been processed by a specific encoder. In the speech domain, a token typically corresponds to one or more frames of acoustic features in an audio signal. In the text domain, a token typically corresponds to a character, subword, punctuation mark, or special symbol.

[0026] Non-autoregressive (NAR) generation: NAR can be understood as a sequence generation paradigm, specifically a parallel sequence generation method. Unlike autoregressive token-by-token serial generation, NAR predicts the output of multiple target positions in parallel through one or a few forward propagations. In one or more embodiments of this specification, the tokens at each mask position can be generated in parallel, that is, mask prediction can be performed on the tokens at the mask positions. This mask prediction can be understood as replacing the mask markers at the positions to be predicted in the input token sequence and predicting the token content at all mask positions.

[0027] Corresponding to the background technology, in practical applications, during the process of converting text into audio using TTS technology, there are problems such as misreading, omission, or inaccurate pronunciation of words in the text, resulting in poor quality of synthesized audio that does not meet actual requirements. In such cases, it is necessary to correct the poor quality audio that does not meet actual requirements.

[0028] Current repair methods include re-sampling, which involves using a TTS model to re-synthesize the text corresponding to the erroneous audio segment. Specifically, the random seed used in the TTS model can be changed to attempt to obtain an audio synthesis result that meets the actual requirements. However, whether the audio synthesis result obtained through re-sampling meets the actual requirements usually relies on manual listening judgment, which is wasteful of human resources and inefficient. Furthermore, due to the randomness in generating the audio synthesis result—meaning that the success of the repair depends on probability—there are situations where repeated attempts still fail to obtain an audio synthesis result that meets the actual requirements, wasting computational resources. Moreover, the regenerated audio synthesis result has uncontrollable differences from other audio parts of the original audio in terms of timbre, rhythm, and speech rate, resulting in an inconsistent overall listening experience.

[0029] Furthermore, current repair methods also include sentence-by-sentence resynthesis, which involves completely resynthesizing the entire sentence containing the error. However, this results in a significant difference in the timbre, rhythm, and speech rate of the entire sentence compared to adjacent sentences. Additionally, the duration of the regenerated audio is uncontrollable; therefore, in situations where the synthesized audio needs to be synchronized with the video, such as in video dubbing scenarios, variations in duration can easily lead to audio-visual desynchronization.

[0030] To address the aforementioned technical problems, this specification provides an audio processing method, and also relates to an audio processing method applied to cloud-side devices, a training method for an audio processing model, an audio processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0031] See Figure 1 , Figure 1 This specification illustrates an architecture diagram of an audio processing system according to one embodiment of the present specification. The audio processing system may include a client 100 and a server 200. Client 100 is used to send an audio processing request to server 200, wherein the audio processing request carries an initial audio containing at least one differential audio segment, the differential audio segment representing an audio segment whose text corresponding to the initial audio differs from the standard text. Server 200 is used to receive audio processing requests sent by client 100, construct a word sequence to be processed based on the word sequences corresponding to each differing audio segment and at least one non-dissimilar audio segment; input the word sequence to be processed into a target audio processing model to obtain the corrected audio segments corresponding to each differing audio segment output by the target audio processing model; update the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio; and send the target audio to client 100. Client 100 is also used to receive the target audio sent by server 200.

[0032] The server 200 in the audio processing system provided in one embodiment of this specification can correct the differing audio segments in the initial audio, avoiding the need for indiscriminate regeneration of the entire audio, thus improving generation efficiency and saving computational resources. By constructing a sequence of words to be processed based on the word sequences corresponding to each differing and non-distinguishing audio segment, the target audio processing model can utilize the contextual information of the non-distinguishing audio segments when correcting the audio, thereby providing a reference for correcting the differing audio segments. This improves the accuracy of the correction and allows control over the duration of the generated corrected audio segments, avoiding audio-visual asynchrony caused by duration changes in practical applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-distinguishing audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-distinguishing audio segments in terms of timbre, rhythm, etc., improving the overall naturalness and coherence of the target audio.

[0033] An audio processing system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In an audio processing scenario, server 200 is used to provide audio processing services between multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0034] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In audio processing scenarios, users can publish data streams to server 200 through client 100, server 200 can generate target audio corresponding to the initial audio based on the data stream, and push the target audio corresponding to the initial audio to other clients that have established communication.

[0035] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0036] Client 100 can be a browser, application (APP), or web application such as HyperText Markup Language 5 (H5) application, or a lightweight application (also known as a mini-program), or cloud application, etc. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as based on the Real-Time Communication (RTC) SDK. Client 100 can be deployed in a computing device and depends on the device to run or on certain APPs on the device. The computing device may have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, or personal computer. Various other types of applications can also be configured in the computing device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0037] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0038] It is worth noting that the audio processing methods and audio processing model training methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the audio processing methods and audio processing model training methods provided in the embodiments of this specification. In other embodiments, the audio processing methods and audio processing model training methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0039] See Figure 2 , Figure 2A flowchart of an audio processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0040] Step 202: Obtain an initial audio containing at least one differential audio segment, wherein the differential audio segment represents an audio segment whose text corresponds to the initial audio and differs from the standard text.

[0041] In one or more embodiments of this specification, the initial audio can be understood as audio that needs audio repair, or audio that does not meet user needs and contains erroneous audio segments. In some optional embodiments, the initial audio is audio synthesized by a TTS model based on standard text; in other optional embodiments, the initial audio is audio obtained by recording, reading aloud, or responding to standard text using robotic intelligent voice. This specification does not specifically limit the source of the initial audio.

[0042] Understandably, the initial audio corresponds to a standard text. This standard text can be understood as a benchmark reference, the text content that the initial audio is expected to express correctly. Alternatively, it can be said that the standard text specifies the expected text content corresponding to the initial audio. Therefore, this standard text can be used to determine whether the initial audio meets the requirements. The specific form of the standard text corresponding to the initial audio varies in different scenarios. For example, in a video dubbing scenario, the standard text can be the dialogue text. The dialogue text can be input into a TTS model to obtain the synthesized audio corresponding to the dialogue text, and the initial audio can be obtained based on this synthesized audio. Similarly, in an audiobook scenario, the standard text is the original text of the book. The original text can be converted into synthesized audio using actual speech synthesis tools in the audiobook scenario, such as a TTS model, and the initial audio can be obtained based on this synthesized audio.

[0043] For example, the standard text is "The weather is so nice today, shall we go for a walk in the park together?". This standard text can be input into a TTS model to obtain the synthesized audio, which is the audio corresponding to "The weather is so nice today, shall we go for a walk in the park together?". As mentioned earlier, due to the limitations of the TTS model's performance, errors are unavoidable in the synthesized audio, such as misreading, omissions, and extra readings. Specifically, this can manifest as the actual text corresponding to the synthesized audio (referred to as the initial text in this specification) being "The weather is so nice today, shall we go for a run in the park together?", meaning that the TTS model incorrectly processes "walk" as "run" during the generation of the synthesized audio.

[0044] It should be noted that in some optional embodiments, the initial audio is the complete audio that needs to be repaired in different scenarios. In other optional embodiments, the initial audio is a portion of the audio extracted from the complete audio that needs to be repaired. This specification does not specifically limit the length of the initial audio, as long as the initial audio contains at least one different audio segment and at least one non-different audio segment. In other words, the text corresponding to the initial audio consists of non-different text segments that are consistent with the standard text and different text segments that are inconsistent with the standard text.

[0045] Using the previous example, the standard text is "The weather is so nice today, we can go for a walk in the park," and the actual text corresponding to the synthesized audio is "The weather is so nice today, we can go for a run in the park, shall we go together?" The initial audio can be the complete synthesized audio corresponding to "The weather is so nice today, we can go for a run in the park, shall we go together?" or a part of the complete synthesized audio, such as the synthesized audio segment corresponding to "Shall we go for a run in the park, shall we go together?"

[0046] In one or more embodiments of this specification, a differing audio segment can be understood as an audio segment in the initial audio where the corresponding actual text content differs from the standard text. In other words, if the text content corresponding to a certain segment of the initial audio is inconsistent with the text content at the corresponding position in the standard text, then that segment of audio can be considered a differing audio segment. Conversely, a non-differential audio segment can be understood as an audio segment in the initial audio where the corresponding actual text content does not differ from the standard text.

[0047] Understandably, the presence of discrepancies in the initial audio indicates that the corresponding text information deviates from the requirements of the standard text. This could manifest as a mispronunciation of a word, omission of a word, the presence of extra words not present in the standard text, or misreading of numbers. Therefore, it is necessary to perform audio repair on the discrepancies in the initial audio to ensure that the repaired audio segments, i.e., the corrected audio segments, conform to the corresponding text content in the standard text. The corrected audio segments are then used to update the initial audio to obtain the target audio, ensuring that the text content corresponding to the target audio is consistent with the standard text.

[0048] It should be noted that this specification does not limit the specific granularity of the differing and non-disagreeing text segments, which can be set based on the actual application scenario and actual needs. For example, one or more differing word segments, or one or more sentences, can be considered as differing text segments. Correspondingly, non-disagreeing text segments can also be one or more word segments, or one or more sentences, etc., where a word segment can include one or more characters. Therefore, it can be understood that the length of the time interval corresponding to the differing audio segment in the initial audio is also not specifically limited; its specific length can correspond to the granularity of the differing text segment. For example, if the differing text segment can be a single differing word, then the time interval corresponding to the differing audio segment is the pronunciation duration of that differing word in the initial audio.

[0049] In one or more embodiments of this specification, the initial audio corresponds to a global time interval. The specific form of the global time interval can be an audio timeline corresponding to the initial audio. The global time interval or audio timeline can be used to characterize the time position information of each audio sampling point or audio frame in the initial audio.

[0050] Understandably, each differing audio segment corresponds to a time interval on the audio timeline; that is, a differing audio segment is an audio fragment within the initial audio. Furthermore, when the initial audio contains multiple differing audio segments, the time intervals of each differing audio segment on the audio timeline do not overlap, and they are distributed in chronological order within the initial audio. Correspondingly, the same applies to the non-dissimilar audio segments in the initial audio; each non-dissimilar audio segment also corresponds to a time interval on the audio timeline, and the time intervals of each non-dissimilar audio segment do not overlap, nor do the non-dissimilar audio segments overlap with the differing audio segments on the audio timeline. These audio segments collectively constitute the initial audio; in other words, the initial audio is divided into at least one differing audio segment and at least one non-dissimilar audio segment, and the time intervals corresponding to these audio segments collectively constitute the audio timeline of the initial audio.

[0051] For example, the initial audio timeline is from 0 seconds to 5 seconds. The initial audio contains two different audio segments and one non-different audio segment. Specifically, the time interval corresponding to the first different audio segment is [0 seconds, 2 seconds), the time interval corresponding to the non-different audio segment is [2 seconds, 3 seconds), and the time interval corresponding to the second different audio segment is [3 seconds, 5 seconds].

[0052] Furthermore, in practical applications, time intervals can be represented by a start timestamp and an end timestamp, such as [0.5 seconds, 0.8 seconds]. Alternatively, time intervals can also be represented by a start audio frame number and an end audio frame number, such as [frame 12, frame 20]. This specification does not limit the specific representation of time intervals.

[0053] In one or more embodiments of this specification, the differing audio segments can be determined in various ways. For example, ASR technology can be used to identify the initial text corresponding to the initial audio. Then, the initial text can be compared with the standard text in a normalized space to identify the differing text segments. The audio segments corresponding to these differing text segments are the differing audio segments. When determining the differing audio segments corresponding to the differing text segments, a forced alignment technique can be used to map each word in the standard text onto the audio timeline, obtaining the time interval of each word in the standard text on the audio timeline. This allows the differing audio segments in the initial audio corresponding to the differing audio segments to be determined based on the words contained in the differing text segments. Alternatively, the differing audio segments in the initial audio can be manually labeled.

[0054] In practical applications, the initial audio can be obtained first, and then the initial text corresponding to the initial audio can be obtained using ASR technology. The initial text and the standard text can be compared, and the text content in the standard text that differs from the initial text can be taken as the difference text segment. Forced alignment technology can be used to determine the time interval corresponding to each word in the standard text, thereby determining the audio segment corresponding to the difference text segment in the initial audio, which can be taken as the difference audio segment.

[0055] In one or more embodiments of this specification, after obtaining the initial audio, when determining the difference audio segments in the initial audio, the initial text corresponding to the initial audio can be obtained first, and the initial text and the standard text can be compared to obtain at least one difference text segment in the standard text. Then, based on the time interval of each difference text segment in the initial audio, the difference audio segment corresponding to each difference text segment can be determined.

[0056] In practical applications, when obtaining the initial text corresponding to the initial audio, ASR (Automatic Sounding) technology can be used. Specifically, the initial audio can be input into an ASR model, and the text content corresponding to the initial audio output by the ASR model can be used as the initial text. Alternatively, the initial text corresponding to the initial audio can be determined manually. This specification does not specify a particular method for obtaining the initial text corresponding to the initial audio.

[0057] Then, the initial text and the standard text can be compared character by character, and the character segments in the standard text that are inconsistent with the initial text can be taken as the difference text segments. For example, if the standard text is "The weather is so nice today, we can go for a walk in the park," and the initial text is "The weather is so nice today, we can go for a run in the park, shall we go together?", then "walk" can be taken as a difference text segment. It should be noted that the range of characters corresponding to the difference text segment can be set according to actual needs. For example, only the different characters can be taken as the difference text segment, or the words or sentences containing the different characters can be taken as the difference text segment. This manual does not impose specific restrictions. Continuing with the previous example, the difference text segment could also be "We can go for a walk in the park."

[0058] In one or more embodiments of this specification, the time interval of each differing text segment in the initial audio can be understood as the pronunciation time or start and end time range of each word in the differing text segment in the initial audio. Correspondingly, in practical applications, the time interval of each differing text segment in the initial audio can be obtained based on forced alignment technology.

[0059] The above method first transcribes the initial audio to obtain the initial text, then compares the initial text with the standard text to find the text segments where the differences are located, and finally uses alignment information to map the difference text segments onto the audio timeline of the initial audio to determine the difference audio segments. This transforms the problem of locating audio domains into a problem of comparing text domains, thereby improving the efficiency and accuracy of difference location. At the same time, it also allows the subsequent audio correction process to accurately target the difference audio segments with problems, avoiding the modification of correct non-difference audio segments.

[0060] Furthermore, after determining the initial text corresponding to the initial audio, during the comparison of the initial text and the standard text, if the similarity between the two is less than a preset similarity threshold, it can be determined that the initial audio quality is too low, and that the initial audio does not match the standard text. In this case, processing can be rejected or a re-synthesis can be instructed. Specifically, the initial audio can be discarded directly without performing subsequent repair processes, or the audio can be re-synthesized based on the standard text. If the similarity between the initial text and the standard text is greater than or equal to the preset similarity threshold, the text segments with differences between the two can be identified, and subsequent processes can be performed. Specifically, when comparing the similarity between the two, the text feature vectors of both can be obtained based on a preset text encoder. Then, the similarity between the text feature vectors of the two can be calculated using similarity calculation methods such as cosine similarity and Euclidean distance, and compared with the preset similarity threshold. The preset similarity threshold can be set based on actual needs.

[0061] Step 204: Construct the word sequence to be processed based on the word sequences corresponding to each differing audio segment and at least one non-dissimilar audio segment.

[0062] In one or more embodiments of this specification, each differing audio segment and each non-dissimilar audio segment can be input into a preset audio encoder, thereby obtaining the word sequences corresponding to each audio segment (i.e., each differing audio segment and each non-dissimilar audio segment) in the initial audio output by the preset audio encoder. Then, a word sequence to be processed can be constructed based on the word sequences corresponding to each differing audio segment and at least one non-dissimilar audio segment. Specifically, the word sequences corresponding to each differing audio segment in the word sequence to be processed are masked word sequences.

[0063] In an optional embodiment, a word sequence to be processed can be constructed based on the word sequences corresponding to each differing audio segment and at least one non-dissimilar audio segment. The word sequences of each differing audio segment in the word sequence to be processed are masked, while the word sequences corresponding to the non-dissimilar audio segments are not masked. Specifically, the word sequences corresponding to each differing audio segment and the at least one non-dissimilar audio segment can be concatenated according to their temporal order in the initial audio. The word sequences corresponding to each differing audio segment in the concatenated word sequence can then be masked, i.e., mask markers are used to replace the words in the word sequences corresponding to each differing audio segment to obtain the word sequence to be processed.

[0064] In one or more embodiments of this specification, a sequence of words to be processed is constructed based on the word sequences corresponding to each differing audio segment and at least one non-disagreeing audio segment, including: Obtain the masked word sequence corresponding to each differential audio segment and the non-masked word sequence corresponding to at least one non-differential audio segment; Based on each masked word sequence and at least one non-masked word sequence, construct the word sequence to be processed.

[0065] In one or more embodiments of this specification, a masked word sequence corresponding to each differing audio segment and a non-masked word sequence corresponding to at least one non-dissimilar audio segment can be obtained. In an optional embodiment, the word sequence corresponding to each differing audio segment output by a preset audio encoder can be masked to obtain a masked word sequence corresponding to each differing audio segment. Furthermore, the word sequence corresponding to each non-dissimilar audio segment output by the preset audio encoder can be used as the non-masked word sequence corresponding to each non-dissimilar audio segment.

[0066] In one or more embodiments of this specification, a masked lexical sequence can be understood as a lexical sequence obtained by lexically encoding differing audio segments using a preset audio encoder and then applying masking tags to the lexical units therein. The masked lexical sequence can be used to identify which lexical positions need to be regenerated; or, in other words, the masking tag is equivalent to sending a signal to the target audio processing model to instruct it to predict and fill in the lexical encoding value at the location of the masking tag. Correspondingly, a non-masked lexical sequence can be understood as a lexical sequence obtained by lexically encoding non-disagreeing audio segments using a preset audio encoder.

[0067] In one or more embodiments of this specification, the word sequence to be processed can be understood as a word sequence obtained by combining each masked word sequence and at least one non-masked word sequence in the temporal order of the initial audio, which can be used as input information for the target audio processing model.

[0068] In one or more embodiments of this specification, the masked word sequence corresponding to the differing audio segment and the non-masked word sequence corresponding to the non-disagreeing audio segment are obtained in the following manner: Obtain the initial word sequence corresponding to the initial audio; In the initial lexical sequence, determine the differential lexical subsequences corresponding to each differential audio segment, and in the initial lexical sequence, determine the non-differential lexical subsequences corresponding to each non-differential audio segment; Based on the differential word subsequences corresponding to each differential audio segment, determine the mask word sequence corresponding to each differential audio segment; and based on the non-differential word subsequences corresponding to each non-differential audio segment, determine the non-mask word sequence corresponding to each non-differential audio segment.

[0069] In practical applications, after obtaining the initial audio, the initial word sequence corresponding to the initial audio can be determined. For example, the initial audio can be input into a preset audio encoder to obtain the initial word sequence corresponding to the initial audio.

[0070] In one or more embodiments of this specification, at least one differing lexical corresponding to each differing audio segment in the initial lexical sequence can be masked using a masking mechanism to obtain a masked lexical sequence corresponding to each differing audio segment. The masked lexical sequence represents the lexical portion that needs to be regenerated. Alternatively, for at least one lexical corresponding to each non-discrete audio segment in the initial lexical sequence, the lexical encoding value of each non-discrete lexical in the initial lexical sequence can be retained; that is, no masking processing is required to obtain a non-masked lexical sequence corresponding to each non-discrete audio segment. The non-masked lexical sequence represents the lexical portion that does not contain errors and therefore does not need to be regenerated.

[0071] In one or more embodiments of this specification, after encoding the initial audio into an initial word sequence, the time interval of each word in the initial word sequence on the audio timeline of the initial audio can be obtained. This allows for the determination of the word sequences corresponding to each differing and non-dissimilar audio segment in the initial word sequence based on the corresponding time intervals on the audio timeline. Specifically, for each differing audio segment, the words covered by the time interval corresponding to that differing audio segment on the audio timeline can be determined as the differing word subsequence corresponding to that differing audio segment in the initial word sequence. This differing word subsequence can then be masked to obtain the masked word sequence corresponding to that differing audio segment. Correspondingly, for each non-dissimilar audio segment, the words covered by the time interval corresponding to that non-dissimilar audio segment on the audio timeline can be determined as the non-dissimilar word subsequence corresponding to that non-dissimilar audio segment in the initial word sequence. This non-dissimilar word subsequence can then be used as the non-masked word sequence corresponding to that non-dissimilar audio segment.

[0072] Then, based on the masked word sequence corresponding to each differing audio segment and the non-masked word sequence corresponding to the non-dissimilar audio segment, a word sequence to be processed can be constructed. Specifically, the masked word sequence corresponding to each differing audio segment and the non-masked word sequence corresponding to at least one non-dissimilar audio segment can be concatenated according to the time order of each differing audio segment and at least one non-dissimilar audio segment in the initial audio to obtain the word sequence to be processed.

[0073] For example, the initial audio, in chronological order, may include differential audio segment 1, non-differential audio segment 1, differential audio segment 2, and non-differential audio segment 2. These four audio segments correspond to masked word sequence 1, non-masked word sequence 1, masked word sequence 2, and non-masked word sequence 2, respectively. The word sequence to be processed can be obtained by concatenating the masked word sequence 1, non-masked word sequence 1, and masked word sequence 2 in that order. Alternatively, the word sequence to be processed can be obtained by concatenating the masked word sequence 1, non-masked word sequence 1, masked word sequence 2, and non-masked word sequence 2 in that order. That is, as long as the word sequence to be processed contains at least one masked word sequence and at least one non-masked word sequence, and each word sequence is concatenated according to its instantaneous sequence in the initial audio, the target audio processing model can generate the predicted word sequence corresponding to the masked word sequence by referring to the non-masked word sequence.

[0074] Preferably, the masked word sequences and the unmasked word sequences can be concatenated according to the temporal order of the differing and non-disagreeing audio segments in the initial audio to obtain the word sequence to be processed. This allows the target processing model to obtain sufficient morning / afternoon information, such as the acoustic and prosodic features corresponding to each unmasked word sequence, and to predict the predicted word sequences corresponding to each masked word sequence in parallel. This not only improves the repair efficiency but also the repair accuracy, that is, improving the accuracy of the generated predicted word sequences, thereby improving the accuracy of the generated corrected audio segments.

[0075] It should be noted that the specific selection of the preset audio encoder specified in one or more embodiments of this specification can be set based on the actual application scenario and actual needs. For example, the preset audio encoder can be a residual vector quantization encoder, a neural audio codec, a Transformer-based audio encoder, or a Mel-spectrum-based encoder, etc., as long as it can encode audio into a discrete word sequence.

[0076] Correspondingly, based on the non-differential word subsequences corresponding to each non-differential audio segment, the non-masked word sequence corresponding to each non-differential audio segment is determined. Specifically, the non-differential word subsequence in the initial word sequence that corresponds to at least one non-differential audio segment can be used as the non-masked word sequence.

[0077] In an optional embodiment, after obtaining the differential word subsequences corresponding to each differential audio segment, each word subsequence other than the differential word subsequences in the initial word sequence can be used as the non-differential word subsequences.

[0078] In one or more embodiments of this specification, a sequence of terms to be processed is constructed based on each masked term sequence and at least one non-masked term sequence, including: According to the temporal order of each differing audio segment and at least one non-discrete audio segment in the initial audio, the masked word sequence corresponding to each differing audio segment and the non-masked word sequence corresponding to at least one non-discrete audio segment are concatenated to obtain the word sequence to be processed.

[0079] Corresponding to the above, non-differential audio segments are the audio fragments in the initial audio excluding the differential audio segments, i.e., the correct audio portion without problems. Non-differential lexical subsequences refer to the lexical units in the initial lexical sequence that correspond temporally to the non-differential audio segments. These lexical units possess the original acoustic features of the initial audio and are correct, requiring no modification. Furthermore, the time intervals of each audio segment on the audio timeline corresponding to the initial audio can be obtained, thus revealing the temporal order of each audio segment in the initial audio. Therefore, according to the temporal order of each audio segment in the initial audio, the masked lexical sequences corresponding to each differential audio segment and the non-masked lexical sequences corresponding to at least one non-differential audio segment can be concatenated to obtain the lexical sequence to be processed. In other words, the masked lexical sequences and at least one non-masked lexical subsequence can be arranged sequentially according to their temporal order in the initial audio to form the lexical sequence to be processed.

[0080] It should be noted that this specification does not restrict the specific structural form of the word sequence to be processed, as long as it contains at least one masked word sequence and at least one non-masked word subsequence. In a specific example, the structure of the word sequence to be processed can be an alternating arrangement of multiple segments such as prefix non-masked word sequence, masked word sequence, middle non-masked segment word sequence, masked segment word sequence, and suffix non-masked word sequence.

[0081] Based on the above method, the masked word sequence and the non-masked word sequence are concatenated in chronological order to form the word sequence to be processed. The word sequence to be processed can simultaneously represent the audio content that needs to be corrected and that does not need to be corrected. It can provide the target audio processing model with complete contextual information and prediction generation instructions, ensuring the accuracy of local audio correction and the consistency of timbre and rhythm with the audio segments that do not need to be corrected.

[0082] Step 206: Input the word sequence to be processed into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model.

[0083] In one or more embodiments of this specification, the target audio processing model has the ability to correct audio segments that differ in quality. It can receive a sequence of words to be processed, constructed from the word sequences corresponding to the differing and non-dissimilar audio segments, and use the word sequences corresponding to the non-dissimilar audio segments as context information to perform masking prediction on the word sequences of the differing audio segments, thereby outputting the corrected audio segments corresponding to each differing audio segment. In practical applications, the target audio processing model can be trained based on the sample word sequences corresponding to the masked sample audio.

[0084] Specifically, the target audio processing model has the ability to predict and generate words at masked positions in a masked word sequence. Specifically, it can regenerate the difference audio segments corresponding to each masked word sequence based on the context information provided by the non-masked word sequences in the word sequence to be processed, so as to output the corrected audio segments.

[0085] The target audio processing model can learn acoustic and linguistic rules based on training data, thereby generating semantically correct and acoustically natural audio content corresponding to masked word sequences based on the contextual information provided by the non-masked word sequences. In practical applications, the target audio processing model can adopt a non-autoregressive architecture to predict the mask positions corresponding to each masked word sequence in parallel. Specifically, the target audio processing model can first use each non-masked word sequence in the word sequence to be processed as contextual information, predict the words at each mask position in each masked word sequence, obtain the predicted word sequence corresponding to each masked word sequence, and then generate the corrected audio segment corresponding to each predicted word sequence based on each predicted word sequence.

[0086] In some optional embodiments, after constructing the word sequence to be processed, the word sequence to be processed can be input into the target audio processing model. During the processing, the target audio processing model can retain the word encoding values ​​corresponding to the non-masked word sequence, and can use the non-masked word sequence as context information to generate and fill the word encoding values ​​corresponding to each mask position in each masked word sequence, thereby obtaining the predicted word sequence corresponding to each masked word sequence, and generating the corrected audio segment corresponding to each predicted word sequence, thus obtaining the corrected audio segment corresponding to each difference audio segment.

[0087] Since the generation of the predicted word sequence references the non-masked word sequence corresponding to the non-differential audio segment, it can be guaranteed that the corrected audio segment corresponding to the predicted word sequence maintains the same acoustic features as the non-differential audio segment. This ensures the overall coherence and naturalness of the target audio obtained based on the corrected audio segment. Furthermore, the predicted word sequence is generated based on the masked word sequence corresponding to the differential audio segment. Therefore, the lengths of the predicted word sequence and the masked word sequence are consistent, ensuring the consistency of the duration between the corrected audio segment and the differential audio segment generated based on the predicted word sequence. This not only guarantees the naturalness and smoothness of the corrected audio segment's update in the initial audio but also avoids audio-visual desynchronization issues caused by inconsistent durations between the corrected and differential audio segments in practical application scenarios such as video dubbing.

[0088] In one or more embodiments of the present specification, in the case of word segmentation replacement, the lengths of the predicted token sequence and the masked token sequence can be kept consistent, so as to achieve a fixed duration. In the case of missing word segments and redundant word segments, the duration of the generated corrected audio can be controlled by adjusting the audio frames.

[0089] As described above, when encoding the initial audio into the initial token sequence, the differential audio segment can be tokenized and encoded by a preset audio encoder. The present specification does not limit the specific model selection and encoding frame rate of the preset audio encoder. In a specific example, the preset audio encoder may be a neural audio codec, and its corresponding encoding frame rate may be 25 frames per second, that is, each token corresponds to an audio duration of 40 milliseconds. In this case, each token corresponds to a fixed duration at the preset encoding frame rate, and the lengths of the predicted token sequence and the masked token sequence are consistent, thereby ensuring the duration consistency between the corrected audio segment and the differential audio segment.

[0090] In one or more embodiments of the present specification, in order to further ensure the duration of the generated corrected audio segment, in addition to the aforementioned method of controlling the duration of the generated corrected audio segment through the length of the token sequence, the duration of the synthesized corrected audio segment can also be explicitly controlled based on the text difference type corresponding to the corrected audio segment, so as to avoid excessive duration difference between the generated corrected audio segment and its corresponding differential audio segment that causes the generated target audio to be unnatural and incoherent.

[0091] In an example, when the text difference type corresponding to the corrected audio segment is a word segmentation replacement type, the duration of the modified corrected audio segment can be kept consistent with the duration of its corresponding differential audio segment. For example, "today" is mispronounced as "tomorrow", and the duration of "tomorrow" in the initial audio is 400 milliseconds, then the duration of the differential audio segment can be determined as the duration of the corrected audio segment, that is, the duration of the corrected audio segment is also 400 milliseconds, keeping the duration unchanged.

[0092] In another example, when the text difference type corresponding to the corrected audio segment is missing word segmentation, the duration corresponding to the corrected audio segment can be determined based on the number of characters of the missing word segmentation, the encoding frame rate of the preset audio encoder, and the preset character pronunciation duration. Specifically, the product of the number of characters of the missing word segmentation, the preset character pronunciation duration, and the encoding frame rate of the preset audio encoder may be calculated, and the product is added to the duration of the differential audio segment corresponding to the corrected audio segment to obtain the duration corresponding to the corrected audio segment. For example, "today" is mispronounced as "to", the encoding frame rate of the preset audio encoder is 25 frames per second, and the preset character pronunciation duration is 0.2 seconds, then the duration corresponding to the missing word segmentation can be determined as 25×0.2×1 equal to 5 frames, that is, 200 milliseconds, the duration of the differential audio segment corresponding to the corrected audio segment is 150 milliseconds, then the duration corresponding to the corrected audio segment is 350 milliseconds.

[0093] Furthermore, in one or more embodiments of this specification, when multiple differing audio segments exist in the initial audio, the sum of the net increment frames in the corrected audio segments corresponding to each differing audio segment can be determined. If the sum of the net increment frames is less than a preset frame threshold, the generated corrected audio segments are deemed to meet the requirements and can be used for the generation of subsequent target audio. If the sum of the net increment frames is greater than or equal to the preset frame threshold, the generated corrected audio segments are deemed to not meet the requirements. In this case, the sum of the net increment frames corresponding to at least one of the corrected audio segments can be reduced so that the sum of the net increment frames in the reduced corrected audio segments meets the preset frame threshold, thereby allowing the reduced corrected audio segments to be used for the generation of subsequent target audio.

[0094] The net increase in frames refers to the number of audio frames added to the corrected audio segment relative to its corresponding difference audio segment when the duration of the corrected audio segment is constrained by the text difference type corresponding to the corrected audio segment. It is understood that for a replacement-type corrected audio segment, since its duration is fixed, the number of net increase in frames is zero. However, for a corrected audio segment lacking word segmentation, since it requires adding previously non-existent audio content to the original audio, the number of net increase in frames is greater than zero. In one or more embodiments of this specification, the duration of the corrected audio segment can be constrained to within 300 milliseconds.

[0095] By controlling the duration of each corrected audio segment in the above manner, the difference between the duration of the corrected audio segment and the difference audio segment can be kept within an acceptable range, avoiding problems such as the target audio being unsmooth and disjointed due to excessive differences.

[0096] Step 208: Update the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio.

[0097] In one or more embodiments of this specification, updating the initial audio based on each corrected audio segment to obtain the target audio corresponding to the initial audio includes: Replace each differing audio segment in the initial audio with the corresponding corrected audio segment to obtain the target audio corresponding to the initial audio.

[0098] In one or more embodiments of this specification, the target audio consists of the correct non-differential audio segments in the initial audio and the corrected audio segments corresponding to the generated differential audio segments. The corrected audio segments corresponding to the masked word sequences in the output word sequence of the target audio processing model, i.e., after obtaining the corrected audio segments corresponding to each differential audio segment, can replace the corresponding differential audio segments in the initial audio with the corrected audio segments, thereby updating the initial audio to obtain the target audio.

[0099] In practical applications, each differing audio segment in the initial audio can be replaced with its corresponding corrected audio segment, while non-disagreeing audio segments are retained. A smooth transition can be applied to the splicing boundaries between audio segments to eliminate splicing artifacts, thus obtaining the target audio. Alternatively, crossfade-in / fade-out processing can be performed at each replacement boundary to eliminate discontinuities that may be introduced by the replacement operation. This manual does not impose specific limitations on the specific crossfade-in / fade-out method; any method can be adopted based on actual needs. For example, a preset duration overlap window can be set at the boundary, and the waveforms of the differing and corrected audio segments can be blended within the overlap window using a linear gradient weight. For instance, a 15-millisecond overlap window can be set at the boundary for crossfade-in / fade-out processing. Crossfade-in / fade-out processing ensures a smooth transition at the splicing boundaries, resulting in a more natural and coherent listening experience for the target audio.

[0100] The aforementioned audio processing method can correct discrepancies in the initial audio, avoiding the need for indiscriminate regeneration of the entire audio, thus improving generation efficiency and saving computational resources. By constructing a sequence of terms to be processed based on the masked term sequences corresponding to each discrepancy audio segment and the non-masked term sequences corresponding to non-discrepancy audio segments, the discrepancy audio segments requiring correction are distinguished from the non-discrepancy audio segments at the term level. This allows the target audio processing model to utilize the contextual information of the non-discrepancy audio segments when correcting the audio, providing a reference for correcting the discrepancy audio segments. This improves the accuracy of the correction while controlling the duration of the generated corrected audio segments, avoiding audio-visual asynchrony caused by duration variations in practical applications, specifically misalignment between the target audio and its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-discrepancy audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-discrepancy audio segments in terms of timbre and rhythm, improving the overall naturalness and coherence of the target audio.

[0101] In one or more embodiments of this specification, in step 202 above, when comparing the initial text and the standard text to obtain the difference text segment, it is necessary to consider the actual boundaries of the difference text segment. For example, if the difference is a single character, marking only that character as a difference text segment would lead to insufficient information and a decrease in correction quality during subsequent audio correction. Therefore, in one or more embodiments of this specification, the segmentation coverage of the difference text segment can be adjusted according to the text difference type to include sufficient contextual information for more accurate audio correction.

[0102] Accordingly, in one or more embodiments of the present specification, comparing the initial text with the standard text to obtain at least one differential text segment in the standard text includes: comparing the initial text with the standard text, and taking the text segment having a difference between the standard text and the initial text as a to-be-processed text segment; adjusting the word segmentation coverage range of each to-be-processed text segment in the standard text according to the corresponding text difference type of each to-be-processed text segment, so as to obtain the differential text segment corresponding to each to-be-processed text segment.

[0103] Herein, the to-be-processed text segment may be understood as an original text fragment that is determined by comparison and has a difference with the initial text in the standard text, and the original text fragment has not been adjusted in the word segmentation coverage range. For example, if the standard text is "The weather is really nice today, we can go to the park for a walk" and the initial text is "The weather is really nice today, we can go to the park for a run, shall we go together", the to-be-processed text segment is exactly "walk".

[0104] Herein, the text difference type may also be understood as the classification of the text difference reflected by the to-be-processed text segment. For example, the text difference type may include a missing word segment, which may be understood as a situation where the text difference is caused by a word segment that exists in the standard text but does not exist in the initial text. The text difference type may further include a mispronounced word segment, which may also be referred to as word segment replacement, and the word segment replacement may be understood as a situation where a certain word segment in the standard text is replaced by another character in the initial text to cause a text difference, for example, "walk" in the foregoing example is replaced by "run". The text difference type may further include an extra read word segment, which may be understood as a situation where the text difference is caused by a word segment that exists in the initial text but does not exist in the standard text. The text difference type may further include a digital word segmentation error, which may be understood as a text difference caused by mispronouncing a digital word segment in the standard text. For example, the standard text includes the digital word segment "2.05", which is misread as two point five, that is, "two point five" in the initial text.

[0105] Herein, the word segmentation coverage range may be understood as the word segmentation range covered by the to-be-processed text segment in the standard text, and the word segmentation range may be characterized by the number of word segments or the number of characters. For example, the word segmentation coverage range corresponding to the to-be-processed text segment includes the single character "walk", and after adjustment, the word segmentation coverage range may include the five characters "go to the park for a walk".

[0106] It is understandable that the text segment to be processed is based on the comparison between the standard text and the initial text, identifying the differences or discrepancies between them. Specifically, the text segment to be processed can originate from either the standard text or the initial text, depending on the type of text difference. If the text difference type is missing word segmentation, the missing word segmentation does not exist in the initial text; therefore, the text segment to be processed originates from the standard text, meaning the missing word segmentation can be located from the standard text. If the text difference type is multiple-reading word segmentation, the multiple-reading word segmentation does not exist in the standard text; therefore, the text segment to be processed originates from the initial text, meaning the multiple-reading word segmentation can be located from the initial text. If the text difference type is misreading word segmentation, the text segment to be processed has corresponding text segments on both the standard text and the initial text. Since the standard text contains a correct text segment and the initial text contains a corresponding multiple-reading misreading text segment, the correct text segment from the standard text can be used as the text segment to be processed.

[0107] In one or more embodiments of this specification, the word segmentation coverage of each text segment to be processed in the standard text is adjusted according to the text difference type corresponding to each text segment to be processed, to obtain the difference text segment corresponding to each text segment to be processed, including: If the text difference type corresponding to the target text segment is determined to be any one of missing segmentation, multiple-read segmentation, or misread segmentation, then the text segment composed of related segments in the standard text that are related to the missing segmentation is obtained, and the text segment composed of the related segments and the target text segment is taken as the difference text segment corresponding to the target text segment; or, If the text difference type corresponding to the target text segment to be processed is determined to be a numeric word segmentation error, the text segment consisting of the full numeric word segmentation corresponding to the numeric region containing the erroneous numeric word segmentation in the standard text is taken as the difference text segment corresponding to the target text segment to be processed. The target text segment to be processed is any one of the text segments to be processed.

[0108] Specifically, for each text segment to be processed, the text difference type corresponding to that segment can be determined. If the text difference type is determined to be missing a segment, the position of the missing segment can be located in the standard text, and related segments can be found. More specifically, a specified number of segments in the standard text that are located before and / or after the missing segment can be used as related segments. It can be understood that, based on the aforementioned method for determining the text segment to be processed in the case of missing segments—that is, the text segment to be processed originates from the standard text—after determining the related segments from the standard text, the text segment formed by the related segments and the text segment to be processed can be directly used as the difference text segment. For example, if the standard text is "The weather is really nice today, can we go for a walk in the park together?", and the initial text is "The weather is really nice today, can we go for a walk together?", then the text segment to be processed is the missing segment "park". The three characters before "park" ("can go") and the two characters after "park" ("walk") can be used as related segments associated with "park". Then, the missing word segment and the text segment composed of its corresponding related word segment can be used as the difference text segment. That is, the difference text segment corresponding to the text segment "park" to be processed is "you can go to the park for a walk".

[0109] In particular, in one or more embodiments of this specification, for the case of missing word segments, that is, when the word segment at the beginning of the sentence in the standard text is missing, since there are no time-aligned word segments before the missing word segment at the beginning of the sentence, the starting position of the difference text segment can be set to the beginning position of the standard text, and the ending position of the difference text segment can be set to the position of a preset number of word segments after the missing word segment, so that the difference text segment can be obtained in this way.

[0110] In practical applications, when segmented words with discrepancies are short or contain few characters, their corresponding time intervals or pronunciation durations in the initial audio are also short. This can easily lead to distortion in the corrected pronunciation during generation, thus affecting the quality of the correction. Adjusting the corresponding segmentation coverage can improve the accuracy of audio correction. Specifically, when the mask area is short, such as 240 milliseconds, the generation process is affected by the acoustic anchoring effect of adjacent original erroneous audio, causing the corrected pronunciation to still carry the acoustic characteristics of the original erroneous pronunciation. Therefore, the acoustic anchoring effect can be reduced by expanding the segmentation coverage, i.e., expanding the mask area.

[0111] When it is determined that the text difference type corresponding to the to-be-processed text segment is an extra-read word segmentation, the associated word segment related to the extra-read word segmentation in the standard text can be acquired. Correspondingly, a specified number of word segments whose positions in the standard text are before and / or after the position of the extra-read word segmentation can be taken as the associated word segment. Then, the text segment composed of the associated word segment and the to-be-processed text segment can be used as the difference text segment corresponding to the to-be-processed text segment. It can be understood that, based on the aforementioned determination manner of the to-be-processed text segment in the case of the extra-read word segmentation, that is, the to-be-processed text segment originates from the initial text side, after the associated word segment is determined from the standard text side, it is also necessary to delete the extra-read word segmentation in the to-be-processed text segment to obtain the deleted to-be-processed text segment, and then the text segment composed of the deleted to-be-processed text segment and the associated word segment can be used as the difference text segment. It can also be understood that, when the to-be-processed text segment is the extra-read word segmentation itself, deleting the extra-read word segmentation in the to-be-processed text segment results in an empty content, that is, the text segment composed of the associated word segment is the difference text segment. For example, the standard text is "The weather is really nice today, we can go for a walk in the park, shall we go together", and the initial text is "The weather is really nice today, we we can go for a walk in the park, shall we go together", then the to-be-processed text segment is the extra-read word segmentation "we", and the five characters after "we", that is "can go for a walk in the park" can be taken as the associated word segment related to "we". Then, the extra-read word segmentation in the to-be-processed text segment is deleted, that is, an empty text is obtained, so the text segment composed of the deleted to-be-processed text segment and its corresponding associated word segment is the text segment composed of the associated word segment itself, that is, "can go for a walk in the park" is the difference text segment.

[0112] When it is determined that the text difference type corresponding to the to-be-processed text segment is a misread word segmentation, the position of the misread word segmentation can be located in the standard text, and the associated word segment related to the misread word segmentation can be searched. More specifically, a specified number of word segments whose positions in the standard text are before and / or after the position of the misread word segmentation can be taken as the associated word segment. It can be understood that, based on the aforementioned determination manner of the to-be-processed text segment in the case of the misread word segmentation, that is, the misread text segment originates from the standard text side, after the associated word segment is determined from the standard text side, the text segment composed of the associated word segment and the to-be-processed text segment can be directly used as the difference text segment. For example, the standard text is "The weather is really nice today, we can go for a walk in the park, shall we go together", and the initial text is "The weather is really nice today, we can go running, shall we go together", then the to-be-processed text segment is the misread word segmentation "walk", the five characters before "walk", that is "can go to the park", and one character "step" after "walk" can be taken as the associated word segment related to "walk". Then, the text segment composed of the misread word segmentation and its corresponding associated word segment can be used as the difference text segment, that is, "can go for a walk in the park".

[0113] When it is determined that the text difference type corresponding to the to-be-processed text segment is a misreading of a digital word segmentation, a complete digital region containing the erroneous digital word segmentation can be located in the standard text, and the text segment formed by all digital word segmentations within this digital region is taken as the differential text segment. The text segment formed by all digital word segmentations can be understood as the original writing of the digital region in the standard text, or in other words, all digital word segmentations are the set of all digit-related word segmentations contained in the digital region of the standard text, which may include symbols such as decimal points and percent signs. For example, the standard text includes the digital word segmentation "2.05", which is misread as "er dian wu" (two point five), that is, it appears as "er dian wu" in the initial text. Since text comparison compares differences in text characters, the to-be-processed text segment is "0". After adjusting the word segmentation coverage range in the aforementioned manner, the complete digital region of the erroneous digital word segmentation can be determined as "2.05", that is, "2.05" or "er dian ling wu" (two point zero five) can be used as the differential text segment.

[0114] Further, when comparing the standard text and the initial text, there are polyphonic characters that have different pronunciations but identical characters. Such characters cannot be identified for differences through text comparison, so polyphonic characters can be manually identified, and the actual pronunciation of the polyphonic characters can be marked by combining phonemes and tones, that is, the phonemes and tones corresponding to the polyphonic characters can be obtained. Then, the phonemes and tones of the polyphonic character are taken as an independent text segment, the polyphonic character region in the initial text is taken as a differential text segment, and the audio segment corresponding to the polyphonic character in the initial audio is taken as a differential audio segment. After that, audio can be separately synthesized for the independent text segment corresponding to the polyphonic character, and spliced into the target audio in chronological order, or in other words, the differential audio segment corresponding to the polyphonic character in the target audio is replaced, so as to ensure that the polyphonic character can be pronounced according to its marked phonemes and tones.

[0115] In the above manner, the word segmentation coverage range of the differential text that requires audio correction can be adjusted according to the text difference type, so that a differential text segment with more reasonable boundaries can be obtained. In this case, the differential text segment can contain sufficient context information, avoiding the problem of inaccurate audio correction caused by an excessively short differential text segment, and improving the accuracy and naturalness of the subsequently generated corrected audio segment.

[0116] In one or more embodiments of this specification, in the above step 202, determining the differential audio segment corresponding to each differential text segment according to the time interval of each differential text segment in the initial audio includes: Acquiring the first time interval of each word segmentation in the standard text in the initial audio; For each different text segment, the first time intervals of each word segment contained in the different text segment are merged to obtain the second time interval of the different text segment in the initial audio, and the audio segment corresponding to the second time interval in the initial audio is taken as the different audio segment corresponding to the different text segment.

[0117] Corresponding to the foregoing, a forced alignment algorithm may be used to determine the first time interval of each word segment in the standard text in the initial audio, and the first time interval may be understood as the start and end time range or the pronunciation time range corresponding to each word segment in the standard text in the initial audio. Therefore, when determining the different audio segment corresponding to each different text segment, for each different text segment, the first time interval of each word segment contained in the different text segment may be acquired, and then the first time intervals of the respective word segments contained in the different text segment may be merged, specifically, the first time intervals of the respective word segments contained in the different text segment may be spliced together in chronological order, so as to obtain a second time interval, that is, the pronunciation time range of the different text segment in the initial audio can be obtained, and the audio segment in the initial audio corresponding to the second time interval is the different audio segment corresponding to the different text segment.

[0118] It can be understood that, since the different text segment is a text composed of consecutive characters, the corresponding first time intervals are consecutive in time, so that the splicing can be performed in chronological order. For example, the different text segment is "take a walk", the first time interval corresponding to the word "take" is [2.5 seconds, 2.8 seconds], and the first time interval corresponding to the word "walk" is [2.9 seconds, 3 seconds], then the second time interval can be determined as [2.5 seconds, 3 seconds].

[0119] It should be noted that when the forced alignment algorithm is used to determine the first time interval of each word segment in the standard text in the initial audio, the specific granularity of the word segments may not be limited, for example, it may be one character or multiple characters, and when determining the second time interval corresponding to the different text segment, the word segments contained therein may be determined based on the corresponding granularity, and the first time interval corresponding to each word segment may be determined.

[0120] In addition, in practical applications, after the second time interval is obtained, the second time interval may be extended, specifically, the front end of the second time interval may be extended forward by a preset duration and / or the back end of the second time interval may be extended backward by a preset duration, so as to cover transition phonemes that may exist at the boundaries of the word segments, and improve the smoothness when the subsequent corrected audio segment is spliced with the original audio segment.

[0121] Understandably, in the case of missing word segments, that is, when some word segments in the different text segments do not have corresponding time intervals in the initial audio, the time intervals of adjacent word segments can be used for interpolation estimation, or the time interval of the missing word segment can be directly set to the range between the end time of the preceding word segment and the start time of the following word segment to obtain the first time interval corresponding to the missing word segment.

[0122] The above-mentioned method of merging time intervals based on word segmentation to determine the corresponding audio segments of the differing text segments can utilize the fine time alignment information provided by the alignment algorithm to ensure that the time interval of the differing audio segments is neither too small, which would lead to the omission of some content, nor too large, which would lead to the unnecessary inclusion of correct parts. This improves the accuracy and rationality of the differing audio segments and provides accurate time interval positioning for subsequent correction of the differing audio segments, ensuring the accuracy and effectiveness of audio correction.

[0123] In one or more embodiments of this specification, in step 204 above, determining the mask word sequence corresponding to each differing audio segment based on the differing word subsequence corresponding to each differing audio segment includes: The lexical units in the subsequence of lexical units corresponding to each lexical unit of the lexical unit are masked to obtain the masked lexical unit sequence corresponding to each lexical unit of the lexical unit.

[0124] Specifically, for each differing audio segment, when generating a masked word sequence based on the differing word subsequence generator corresponding to that audio segment, each word in the differing word subsequence can be masked. Masking can be understood as replacing each word in the differing word subsequence with a preset mask marker. For example, replacing the encoded value of each word in the differing word subsequence with a preset mask marker yields the masked word sequence for that audio segment. The mask marker can be a predefined special symbol in the model vocabulary of the target audio processing model, used to indicate that the target audio processing model needs to predict and generate the content at the mask marker positions.

[0125] By replacing each lexical in the differential lexical subsequence corresponding to the differential audio segment with a mask marker, the target audio processing model can explicitly identify the lexical positions that need to be regenerated.

[0126] In one or more embodiments of this specification, determining the masked word sequence corresponding to each differing audio segment based on the differing word subsequence corresponding to each differing audio segment includes: For each differing audio segment, obtain a mask marker with the same number of tokens as the differing token subsequence corresponding to that differing audio segment, and use each obtained mask marker as the mask token sequence corresponding to that differing audio segment.

[0127] In practical applications, when generating masked lexical sequences, an independent masking tag sequence can also be used to construct the masked lexical sequence. This method does not rely on the original lexical values ​​in the differencing lexical subsequence, but instead constructs a masking tag sequence of the same length as the differencing lexical subsequence, which has high flexibility and versatility.

[0128] Specifically, mask markers can be understood as preset markers for lexical encoding values ​​independent of the differing lexical subsequence, and can be set based on actual application scenarios and needs. Unlike the above method of replacing lexical encoding values ​​in the differing lexical subsequence with mask markers, here we directly construct an equal number of mask markers based on the number of lexical units contained in the differing lexical subsequence, thus forming a mask marker sequence corresponding to the differing lexical subsequence. For example, if the differing lexical subsequence contains fifteen lexical units, then fifteen preset mask markers can be obtained. These fifteen mask markers, arranged in the order of acquisition, constitute the masked lexical sequence corresponding to the differing audio segment.

[0129] The above method constructs a masked word sequence by directly obtaining a masked token of equal quantity to the difference word subsequence, thereby decoupling the construction process of the masked word sequence from the specific content of the difference word subsequence, simplifying the construction process of the masked word sequence, and ensuring that the length of the masked word sequence is consistent with the length of the difference word subsequence, thus enabling accurate control of the duration of the corrected audio segment corresponding to the difference audio segment.

[0130] This manual also provides the specific structure of the audio processing model, such as Figure 3 As shown, Figure 3 This is a schematic diagram of an audio processing model provided in one embodiment of this specification. The audio processing model may include a prediction network and a decoding network. The prediction network is used to predict the predicted words corresponding to each masked word position in the masked word sequence based on the word sequence to be processed, thereby outputting the predicted word sequence corresponding to each masked word sequence. The decoding network is used to restore each predicted word sequence to an audio signal.

[0131] In one or more embodiments of this specification, the sequence of words to be processed is input into a target audio processing model to obtain corrected audio segments corresponding to each differing audio segment output by the target audio processing model, including: The word sequence to be processed is input into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed output by the prediction network. Each predicted word sequence is input into the decoding network of the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the decoding network.

[0132] In one or more embodiments of this specification, the prediction network can be understood as a network module in the audio processing model responsible for parallel content prediction at mask positions. For example, it can be a bidirectional sequence modeling network based on a transformer architecture, used to predict the word that should be filled at the mask position based on contextual information provided by non-masked words before and after the mask position. The predicted word sequence refers to the word sequence output by the prediction network after predicting the words corresponding to each mask position in the mask word sequence. The decoding network can be understood as a network module in the audio processing model that converts the predicted word sequence into an audio waveform. For example, it can be a vocoder network based on deconvolution or subband filtering, used to convert discrete word encoding values ​​into continuous audio waveforms.

[0133] In one or more embodiments of this specification, the prediction network may employ a non-autoregressive architecture, meaning that the prediction network can perform parallel predictions on all mask positions in the input word sequence, rather than generating them sequentially position by position.

[0134] Therefore, after the word sequence to be processed is input into the target audio processing model, the prediction network can use each non-masked word sequence in the word sequence to be processed as context information to generate the predicted word sequence corresponding to each masked word sequence in the word sequence to be processed in parallel. Then, each predicted word sequence can be input into the decoding network, and the decoding network can generate the audio waveform corresponding to each predicted word sequence, thereby obtaining the corrected audio segment corresponding to each difference audio segment.

[0135] It should be noted that when training the initial audio processing model to obtain the target audio processing model, the prediction network and the decoding network can be trained separately using different training strategies, or they can be trained jointly end-to-end. The appropriate training scheme can be selected based on actual needs and the actual scenario.

[0136] By designing the audio processing model using the above method, a prediction network and a decoding network are used. The prediction network is used to predict mask positions in parallel, and the decoding network is used to convert the predicted word sequence into an audio waveform. This decouples word semantic prediction and audio waveform generation, improving the performance and correction quality of the audio processing model. Furthermore, the prediction network can predict multiple mask word sequences in parallel, thereby improving audio correction efficiency.

[0137] It should be noted that the structure of the above audio processing model is only an example. Since current audio decoders are relatively mature and can be directly used to decode the word sequence corresponding to the audio, the decoding network in the audio processing model in one or more embodiments of this specification is not a necessary network structure. That is to say, the audio processing model in one or more embodiments of this specification may only include a prediction network to perform prediction reconstruction of words at the mask position.

[0138] In another alternative embodiment, the audio processing model in one or more embodiments of this specification may further include a preset audio encoder, a prediction network, and a decoding network, thereby using the audio processing model to perform an end-to-end audio processing flow.

[0139] Furthermore, in the prediction process of non-autoregressive models, although a single forward propagation can predict the content of all mask positions in parallel, the prediction quality at some positions is poor due to the lack of sequence dependencies between mask positions. Especially when the mask word sequence is long or there are multiple mask word sequences, the predicted words at some mask positions may be predicted inaccurately. In one or more embodiments of this specification, a parallel prediction plus iterative backfilling approach can be adopted to improve prediction quality while ensuring prediction efficiency.

[0140] In one or more embodiments of this specification, the sequence of lexical units to be processed is input into a prediction network in a target audio processing model to obtain the predicted lexical sequences corresponding to each masked lexical sequence in the sequence of lexical units to be processed, as output by the prediction network, including: The word sequence to be processed is input into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed, which is output in parallel by the prediction network. If all predicted word sequences meet the preset conditions, the predicted word sequences corresponding to each mask word sequence are obtained. If any predicted word sequence does not meet the preset conditions, the predicted word sequence that meets the preset conditions is backfilled into the word sequence to be processed to obtain the backfilled word sequence. The backfilled word sequence is then used as the word sequence to be processed. The word sequence to be processed is then input into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed, which is output in parallel by the prediction network.

[0141] Parallel output can be understood as the prediction network simultaneously outputting the predicted word sequences corresponding to all masked word sequences in the word sequence to be processed through a single forward propagation, rather than the serial processing of masked word sequences one by one based on autoregression. For example, if the word sequence to be processed contains three masked word sequences, the prediction network will output three predicted word sequences simultaneously in a single forward propagation, and the generation process of each predicted word sequence is independent of each other.

[0142] The preset conditions can be understood as the criteria used to determine whether the predicted word sequence meets the preset requirements. These criteria can be determined based on the actual scenario and needs. For example, they can be determined based on the confidence level of each predicted word sequence and a preset confidence threshold. Specifically, for each predicted word sequence, the confidence level of each word in the predicted word sequence output by the target processing model can be obtained, and it can be determined whether the confidence level of each word in the predicted word sequence is greater than the preset confidence threshold. If so, the predicted word sequence meets the preset conditions, indicating that it is a high-quality word sequence that meets the preset requirements and can be backfilled into the word sequence to be processed. If not, the predicted word sequence does not meet the preset conditions, indicating that it is a low-quality word sequence that does not meet the preset requirements. After judging all the predicted word sequences, the backfilled word sequence can be obtained.

[0143] Then, the backfilled lexical sequence can be used again as the lexical sequence to be processed, and the step of inputting the lexical sequence to be processed into the prediction network in the target audio processing model is re-executed to obtain the predicted lexical sequence corresponding to each mask lexical sequence in the lexical sequence to be processed in parallel output by the prediction network. The confidence threshold comparison is continued until all predicted lexical sequences meet the preset conditions, and the backfilled lexical sequence in which all predicted lexical sequences meet the preset conditions is obtained. At this time, each predicted lexical sequence in the backfilled lexical sequence in which all predicted lexical sequences meet the preset conditions can be determined as the final predicted lexical sequence corresponding to each mask lexical sequence.

[0144] In one or more embodiments of this specification, after re-executing the step of inputting the word sequence to be processed into the prediction network of the target audio processing model to obtain the predicted word sequences corresponding to each mask word sequence in the word sequence to be processed output in parallel by the prediction network, and continuing to compare confidence thresholds, the termination condition for this loop step, except for until all predicted word sequences meet a preset condition, is, in some optional embodiments, the termination condition is that the number of loops reaches a preset loop count threshold. It can be understood that the number of loops can be the number of times the confidence threshold comparison is repeated, or it can be the number of times the step of inputting the word sequence to be processed into the prediction network of the target audio processing model is re-executed, etc. When it is determined that the number of loops has reached the preset loop count threshold, each current mask word sequence in the current word sequence to be processed can be obtained. For each current mask word sequence, each historical predicted word sequence obtained in the historical loop corresponding to the current mask word sequence can be obtained. Then, the historical predicted word sequence with the highest confidence can be selected as the final predicted word sequence corresponding to the current mask word sequence. Thus, the predicted word sequences corresponding to each mask word sequence can be obtained.

[0145] By using the parallel prediction plus iterative backfilling method described above, in each step of the parallel prediction iteration, the predicted word sequence with higher confidence is backfilled into the word sequence to be processed, and it becomes the known context for the prediction of subsequent word sequences. This gradually reduces the number of remaining mask positions and improves the prediction quality of the words corresponding to the remaining mask positions. It can improve the prediction quality while ensuring prediction efficiency.

[0146] Furthermore, after obtaining the target audio, a quality audit can be performed on the target audio to ensure that the corrected target audio meets the requirements. In one or more embodiments of this specification, after updating the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio, the method further includes: If the target audio does not meet the review conditions, the initial time interval of each difference audio segment in the initial audio is determined and each initial time interval is expanded to obtain the adjusted audio segment of each difference audio segment in the initial audio. The target audio is regenerated based on each adjusted audio segment until the regenerated target audio meets the review criteria.

[0147] The review criteria can be understood as multi-dimensional standards used to determine whether the quality of the target audio meets the standards. These criteria can be set based on actual needs. For example, they may include text accuracy criteria, i.e., whether the consistency between the text corresponding to the target audio and the standard text reaches a threshold; numerical accuracy criteria, i.e., whether the target audio correctly reads the numerical regions in the standard text; and duration consistency criteria, i.e., whether the duration difference between the target audio and the initial audio is within a preset duration range, etc.

[0148] In one or more embodiments of this specification, after obtaining the target audio, speech recognition can be performed on the target audio to transcribe it into text, obtaining the recognized text corresponding to the target audio. Then, the recognized text and the standard text can be reviewed based on the aforementioned three-dimensional judgment criteria.

[0149] In one or more embodiments of the present specification, the character similarity between the recognized text and the standard text may be calculated in a normalization space, and when the character similarity is greater than a preset character similarity threshold, for example, 0.99, it is determined that the target audio satisfies the text accuracy condition. In particular, when there are different characters between the recognized text and the standard text, the recognized text and the standard text may be respectively converted into phoneme texts, that is, a recognized phoneme text and a standard phoneme text can be obtained, then the character similarity between the recognized phoneme text and the standard phoneme text may be calculated in the normalization space, and when the character similarity is greater than a preset character similarity threshold, for example, 0.99, it is determined that the target audio satisfies the text accuracy condition. It can be understood that the maximum value of the aforementioned two character similarities may be taken as the final character similarity to be compared with the preset character similarity threshold.

[0150] In such cases, differences caused by selecting different Chinese characters with the same pronunciation can be avoided, thereby improving the accuracy of reviewing the target audio. For example, the Chinese characters "she" (meaning "set") and "she" (meaning "shoot") are homophones, and speech recognition may select either one, but the pronunciation is actually correct, so the target speech actually meets the standard.

[0151] In one or more embodiments of the present specification, numbers in the recognized text and the standard text may be extracted to compare their corresponding numerical values one by one. Specifically, each first number text in the recognized text and each second number text corresponding to each first number text in the standard text are extracted, then it may be determined whether the numerical value corresponding to each first number text is equal to the numerical value corresponding to its corresponding second number text. When it is determined that the numerical value corresponding to each first number text is equal to the numerical value corresponding to its corresponding second number text, it can be determined that the target audio satisfies the number accuracy condition. In this way, missed detection of number differences by the text accuracy condition can be avoided, the review accuracy is improved, and the correctness of the target audio is enhanced. For example, the first number text in the recognized text is "three hundred thirty-three point two two", which is 333.22, and the second number text corresponding to this first number text in the standard text is "three hundred thirty-three point three two", which is 333.32. There is only one character difference at the text dimension, which may be judged as satisfying in the aforementioned review of the text accuracy condition, that is, the target audio meets the requirements, but it should actually be judged as not meeting the requirements, so a review targeting the numbers is necessary.

[0152] In one or more embodiments of the present specification, a duration difference between the duration of the target audio and the duration of the initial audio may be calculated, and when the duration difference is less than a preset duration difference threshold, it is determined that the target audio satisfies the duration consistency condition.

[0153] When the target audio simultaneously satisfies the above three dimension determination criteria, it can be determined that the target audio satisfies the review condition, that is, it can be determined that the current audio modification is successful, and the current target audio is a high-quality audio that meets the requirements.

[0154] If the target audio does not meet any of the above criteria, it can be determined that the target audio does not meet the review conditions, and the audio modification is deemed unsuccessful, meaning the target audio is not a compliant audio.

[0155] In one or more embodiments of this specification, when it is determined that the target audio does not meet the review conditions, the initial time interval of each differing audio segment in the initial audio can be obtained and expanded. The initial time interval is the aforementioned second time interval of each differing audio segment in the initial audio. Then, the second time interval of each differing audio segment in the initial audio can be expanded. Expanding the initial time interval can be understood as extending the time range of the original initial time interval forward and / or backward by a preset specified duration, or expanding the duration coverage of the initial time interval by a preset specified duration. Then, the adjusted audio segment of each differing audio segment in the initial audio can be obtained. The adjusted audio segment is the audio segment corresponding to the expanded initial time interval in the initial audio. Here, the initial time interval is the time range or pronunciation time corresponding to the differing audio segment in the initial audio, and its specific acquisition process is the same as the aforementioned method, which will not be repeated here. Furthermore, the preset specified duration can be set according to actual needs, such as 0.1 seconds, 0.3 seconds, etc.

[0156] For example, if the initial time interval of the difference audio segment A is [2 seconds, 3 seconds], and the initial time interval is extended forward by 0.2 seconds, the extended second time interval is [1.8 seconds, 3 seconds]. Then the adjusted audio segment is the audio segment corresponding to [1.8 seconds, 3 seconds] in the initial audio.

[0157] Then, the target audio can be regenerated based on each adjusted audio segment until the regenerated target audio meets the review conditions. Specifically, each adjusted audio segment can be used as a difference audio segment to re-execute the audio processing method corresponding to steps 202 to 208, that is, to re-correct the audio based on the target audio processing model to obtain the target audio again and conduct the review until the target audio meets the aforementioned review conditions.

[0158] Understandably, if the target audio meets the review criteria, indicating that the target audio meets user needs or satisfies preset quality requirements, there is no need to regenerate it.

[0159] In one or more embodiments of this specification, if the generated target audio cannot meet the review conditions, the number of times the time interval is expanded can be obtained. If it is determined that the number of times the time interval is expanded reaches a preset expansion threshold, the currently generated target audio, i.e., the audio obtained after the last expansion of the time interval, can be used as the final target audio.

[0160] The above methods can be used to review the quality of the target audio, ensuring the quality and accuracy of the final target audio.

[0161] Furthermore, it should be noted that the audio processing methods provided in one or more embodiments of this specification can be applied to various practical scenarios. For example, in the aforementioned video dubbing scenario, the narration of short dramas or marketing materials in practical applications is usually generated by a TTS model. When individual words are mispronounced in the synthesized audio, it is not necessary to re-synthesize the entire audio segment. The aforementioned audio processing method can be used to locate the discrepancy audio segments and correct only those segments, saving computational resources, maintaining overall speech style consistency, and ensuring the quality of the corrected audio. Similarly, in the audiobook scenario, the recorded audio corresponding to the original text of the book may contain errors such as slips of the tongue or omissions. The aforementioned audio processing method can be used to locate the discrepancy audio segments and correct only those segments, avoiding the need to re-record the entire content of the original text. This saves computational resources, maintains overall speech style consistency, and ensures the quality of the corrected audio.

[0162] like Figure 4 As shown, Figure 4 This document provides a flowchart of an audio processing method for a cloud-based device, which includes the following steps, as part of one embodiment of the present specification.

[0163] Step 402: Obtain an audio processing request sent by the end device, wherein the audio processing request carries an initial audio containing at least one differential audio segment, the differential audio segment representing an audio segment whose text corresponding to the initial audio differs from the standard text.

[0164] In one or more embodiments of this specification, the cloud-side device can receive an audio processing request sent by the end-side device via a network. The initial audio carried in the audio processing request is audio synthesized by a TTS model deployed or invoked by the end-side device. The audio processing request may also carry standard text, audio processing parameters (such as correction accuracy requirements, duration constraints, etc.), and the identification information of the end-side device. After receiving the audio processing request, the cloud-side device can parse the audio processing request, extract the initial audio and related parameters carried in the audio processing request, and prepare for subsequent audio correction processing.

[0165] It should be noted that, in some optional embodiments, the cloud-side device can obtain the standard text corresponding to the initial audio based on the audio processing request sent by the end-side device. In other optional embodiments, the cloud-side device pre-stores the standard text, and the initial audio in the audio processing request sent by the end-side device is itself generated based on the standard text obtained from the cloud-side device. Therefore, in this case, the cloud-side device can directly obtain the standard text.

[0166] Step 404: Construct the word sequence to be processed based on the word sequences corresponding to each differing audio segment and at least one non-dissimilar audio segment.

[0167] In one or more embodiments of this specification, the cloud-side device processes the initial audio, determines the differential audio segments, and constructs a masked word sequence and a non-masked word sequence based on the differential audio segments, which are then concatenated to form the word sequence to be processed. The specific implementation of this step can be found in the detailed description of the differential audio segment determination and word sequence construction in step 204 above, and will not be repeated here.

[0168] Step 406: Input the word sequence to be processed into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model.

[0169] In one or more embodiments of this specification, the cloud-side device can input the constructed sequence of words to be processed into a target audio processing model deployed in the cloud-side device. The target audio processing model may include the aforementioned prediction network and decoding network. The prediction network is used to perform parallel prediction of each masked word sequence in the sequence of words to be processed, and the decoding network is used to decode each predicted word sequence into an audio waveform to obtain the waveform of the corrected audio segment.

[0170] Step 408: Update the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio.

[0171] In one or more embodiments of this specification, the cloud-side device can replace the corresponding difference audio segments in the initial audio with each corrected audio segment, and perform smooth transition processing at the splicing boundary to obtain the target audio. The specific implementation of this step can be found in the relevant description in step 208 above, and will not be repeated here.

[0172] Step 410: Send the target audio to the end-side device.

[0173] In the aforementioned audio processing method applied to cloud-side devices, the cloud-side devices can correct the differing audio segments in the initial audio, avoiding the need for indiscriminate regeneration of the entire audio, thus improving generation efficiency and saving computational resources. By constructing a sequence of words to be processed based on the word sequences corresponding to each differing and non-dissimilar audio segment, the target audio processing model can utilize the contextual information of the non-dissimilar audio segments when correcting the audio, thereby providing a reference for correcting the differing audio segments. This improves the accuracy of the correction while controlling the duration of the generated corrected audio segments, avoiding audio-visual asynchrony caused by duration changes in practical applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-dissimilar audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-dissimilar audio segments in terms of timbre and rhythm, improving the overall naturalness and coherence of the target audio.

[0174] The above is an illustrative scheme of an audio processing method applied to a cloud-side device according to this embodiment. It should be noted that the technical solution of this audio processing method applied to a cloud-side device belongs to the same concept as the technical solution of the audio processing method described above. For details not described in detail in the technical solution of the audio processing method applied to a cloud-side device, please refer to the description of the technical solution of the audio processing method described above.

[0175] This manual also provides training methods for the audio processing model, such as... Figure 5 As shown, Figure 5 This is a flowchart illustrating a training method for an audio processing model provided in one embodiment of this specification, which specifically includes the following steps.

[0176] Step 502: Obtain the sample word sequence, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio.

[0177] In one or more embodiments of this specification, training data needs to be constructed first during the training process. Sample audio is audio data used to train the initial audio processing model. In an optional embodiment, the sample audio is audio without any differing audio segments, that is, the sample audio is consistent with its corresponding sample standard text. In this case, after the sample audio is processed by word encoding using a preset audio encoder to obtain the sample global word sequence corresponding to the sample audio, at least some of the words can be masked to obtain the sample word sequence. Then, the masked words obtained after masking can be divided into at least one sample masked word subsequence. For example, a specified number of consecutive masked words can be used as a sample masked word subsequence. Specifically, each masked word can be used as a sample masked word subsequence, or every five consecutive masked words can be used as a sample masked word subsequence, etc. This specification does not impose specific limitations. Thus, a sample word sequence containing at least one sample masked word subsequence and a sample non-masked word subsequence can be obtained. Understandably, the subsequences in the global lexical sequence of the samples, excluding the masked lexical subsequences of each sample, are the sample non-masked lexical subsequences. The lexical subsequences corresponding to the masked lexical subsequences in the global lexical sequence of the samples are the standard lexical subsequences corresponding to the masked lexical subsequences of each sample.

[0178] In another optional embodiment, the sample audio can be audio with differing audio segments, that is, the sample audio contains both manually annotated or automatically detected differing audio segments and non-disagreeing audio segments. In this case, the global lexical sequence is the complete lexical sequence obtained after lexical encoding of the sample audio. The sample masked lexical subsequence is the result of masking the lexical segments at the corresponding positions in the global lexical sequence of the differing audio segments, and the sample non-masked lexical subsequence is the original lexical segment at the corresponding position in the global lexical sequence of the non-disagreeing audio segments. Then, the label audio segments corresponding to each differing audio segment can be obtained, i.e., the corresponding correct audio segments. Then, the lexical sequence of the label audio segments corresponding to each differing audio segment can be obtained through a preset audio encoder, which can be used as the standard lexical subsequence corresponding to each differing audio segment. That is, the standard lexical subsequence can be understood as the lexical sequence corresponding to the correct audio that each differing audio segment should be corrected to, i.e., the label audio segment, i.e., the standard lexical subsequence. Here, the label audio segment is the standard audio segment corresponding to the differing audio segment, which can be obtained through manual annotation or other methods. In one specific example, a human can listen to the audio segment showing sample differences and then re-record the correct pronunciation corresponding to that segment, which will serve as the label audio segment for that segment. In another specific example, the corresponding text segment of the audio segment showing sample differences in the standard text can be input into another pre-trained speech synthesis model. This model will then generate the correct audio for the corresponding text segment in the standard text, which will serve as the label audio segment for that segment. This specification does not limit the specific method for obtaining the label audio segment.

[0179] Step 504: Input the sample word sequence into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model. Each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence.

[0180] In one or more embodiments of this specification, the constructed sample word sequence is input into the initial audio processing model. The initial audio processing model is an untrained original model or a pre-trained model, and its predictive ability does not yet meet the requirements of practical applications.

[0181] In one or more embodiments of this specification, after inputting the sample word sequence into the initial audio processing model, the initial audio processing model can use the sample non-masked word subsequences in the sample word sequence as context information to predict the sample predicted word sequence corresponding to each sample masked word subsequence in parallel, and obtain the predicted audio segment corresponding to each sample difference audio segment based on each sample predicted word sequence. As mentioned above, the audio processing model may include a prediction network and a decoding network. The prediction network in the initial audio processing model can first perform parallel prediction on each sample masked word subsequence in the sample word sequence to obtain the sample predicted word sequence corresponding to each sample masked word subsequence, and then decode each sample predicted word sequence into each predicted audio segment through the decoding network. Since the initial audio processing model has not been fully trained, the predicted audio segment it outputs may deviate from the labeled audio segment, and the predicted sample predicted word sequence may deviate from its corresponding standard word subsequence. Therefore, in subsequent steps, the initial audio processing model can be trained based on these data.

[0182] It should be noted that, in one or more embodiments of this specification, the initial audio processing model can be understood as a pre-trained audio model capable of reconstructing masked words. During its training phase, its masked word reconstruction capability can be enhanced through single-pass parallel self-supervised learning and / or supervised learning. During the inference phase, the reconstructed words corresponding to each masked word can be obtained through single-pass parallel prediction, or through parallel prediction combined with confidence levels to perform multiple parallel predictions.

[0183] Step 506: Train the initial audio processing model based on the masked word subsequences of each sample and the standard word subsequences of each sample masked word subsequence in the global word sequence of the samples, and obtain the target audio processing model.

[0184] In one or more embodiments of this specification, training can be performed based on the difference between the predicted word sequence of each sample and the standard word subsequence corresponding to the masked word subsequence of each sample. Specifically, the predicted word sequence of each sample and the standard word subsequence corresponding to the masked word subsequence of each sample can be input into a first loss function to obtain a first loss value. Then, the model parameters of the initial audio processing model can be adjusted according to the first loss value until the model training conditions are met, thereby obtaining the target audio processing model. The first loss function can be set based on actual needs; for example, the first loss function can be any combination of one or more of the mean squared error loss function and the cross-entropy loss function. This specification does not impose specific limitations on this.

[0185] In one or more embodiments of this specification, the loss value can also be determined from the audio dimension. In an optional embodiment, when the sample audio is correct audio without any discrepancies, the audio segments of the masked words corresponding to each sample masked word subsequence in the sample audio can be used as the label audio segments. Then, the predicted audio segments corresponding to each sample masked word subsequence and the label audio segments can be input into the second loss function to obtain the second loss value. The model parameters of the initial audio processing model can then be adjusted according to the second loss value until the model training conditions are met, thus obtaining the target audio processing model. The second loss function can be set based on actual needs. For example, the second loss function can be any one or more combinations of the mean squared error loss function, cross-entropy loss function, contrastive loss function, and perceptual loss function. This specification does not impose specific limitations on this.

[0186] In another optional embodiment, when the sample audio contains differential audio segments, the corresponding label audio segments for each differential audio segment can be obtained, as described above. The predicted audio segments and label audio segments corresponding to each differential audio segment can then be input into a second loss function to obtain a second loss value. The model parameters of the initial audio processing model can then be adjusted based on the second loss value until the model training conditions are met, thus obtaining the target audio processing model.

[0187] In one or more embodiments of this specification, the model parameters of the initial audio processing model can be adjusted based on either the first loss value or the second loss value, or based on the sum or weighted sum of the two losses. This specification does not impose specific limitations. Preferably, the model parameters of the initial audio processing model are adjusted using the first loss value.

[0188] It should be noted that, in one or more embodiments of this specification, when adjusting the model parameters of the initial audio processing model, the parameters of the decoding network can be frozen, and only the parameters of the prediction network can be adjusted. Since the decoding network can usually be a well-trained mature network in practical applications, freezing its parameters when training the prediction network can reduce training overhead while ensuring that the prediction network focuses on learning the ability to predict tokens at mask positions.

[0189] Of course, to further improve the decoding network's ability to decode audio in the current scenario, the parameters of the decoding network can be adjusted simultaneously, thereby training it to generate higher-quality audio that is more adapted to the current scenario. For example, in audio restoration scenarios targeting specific speakers, specific languages, or specific recording environments, a general decoding network may have domain shifts. In this case, adjusting the parameters of the prediction network while simultaneously fine-tuning the decoding network can improve the naturalness and accuracy of the generated audio.

[0190] Among them, the model training conditions can be that the first loss value or the second loss value is less than the preset loss threshold, the number of iterations can reach the preset number of iterations, and the amount of training data used can reach the preset amount. This manual does not impose specific restrictions.

[0191] The training method for the aforementioned audio processing model constructs a training data format consistent with practical applications. It trains the initial audio processing model using a sample word sequence containing both masked and unmasked word sequences. This allows the model to learn to predict and correct masked regions in parallel while keeping unmasked regions unchanged. The trained target audio processing model can execute the corresponding steps in the aforementioned audio processing method. It constructs a word sequence to be processed based on the masked word sequences corresponding to each differing audio segment and the unmasked word sequences corresponding to non-dissimilar audio segments. This distinguishes between differing audio segments requiring correction and non-dissimilar audio segments that do not require correction at the word level. The target audio processing model can utilize the contextual information of non-dissimilar audio segments when correcting audio, thus providing a reference for correcting differing audio segments. This improves the accuracy of correction and allows control over the duration of the generated corrected audio segments, avoiding audio-visual desynchronization caused by duration variations in practical applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-differentiated audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-differentiated audio segments in terms of timbre and rhythm, thereby improving the overall naturalness and coherence of the target audio.

[0192] Corresponding to the training methods for the aforementioned audio processing models, training methods for audio processing models applied to cloud-side devices are also provided. For example... Figure 6 As shown, Figure 6 The flowchart of a training method for an audio processing model applied to a cloud-side device, provided as an embodiment of this specification, specifically includes the following steps.

[0193] Step 602: Obtain the model training request sent by the end device, wherein the model training request carries the model identifier of the initial audio processing model.

[0194] In one or more embodiments of this specification, the cloud-side device can parse the model training request to obtain the model identifier carried therein, and then obtain the corresponding initial audio processing model and training data from the cloud-side model repository based on the model identifier. That is, the cloud-side device is configured with a cloud-side model repository, which contains various audio processing models and their corresponding model identifiers, and may also contain training data corresponding to each audio processing model. It can then obtain the corresponding training data and the corresponding audio processing model based on the model training request and perform training.

[0195] Step 604: Obtain the sample word sequence corresponding to the initial audio processing model, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio.

[0196] Step 606: Input the sample word sequence into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model. Each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence.

[0197] Step 608: Train the initial audio processing model based on the masked word subsequences of each sample and the standard word subsequences of each sample masked word subsequence in the global word sequence of the samples, and obtain the target audio processing model and the model parameters of the target audio processing model.

[0198] Step 610: Send the model parameters of the target audio processing model to the end device.

[0199] In an optional embodiment, when sending the model parameters of the target audio processing model to the end device, model parameters that differ from the model parameters of the initial audio processing model can be sent to the end device to reduce data transmission volume and improve transmission efficiency.

[0200] In another optional embodiment, the full set of model parameters of the target audio processing model can be sent to the edge device to ensure the integrity and deployment accuracy of the model parameters of the target audio processing model.

[0201] The training method for the audio processing model applied to cloud-side devices described above constructs a training data format consistent with actual applications. It trains the initial audio processing model using a sample word sequence containing both masked and unmasked word sequences. This allows the model to learn to predict and correct masked regions in parallel while keeping unmasked regions unchanged. The trained target audio processing model can execute the corresponding steps in the above audio processing method. It constructs a word sequence to be processed based on the masked word sequences corresponding to each differing audio segment and the unmasked word sequences corresponding to non-dissimilar audio segments. This distinguishes between differing audio segments requiring correction and non-dissimilar audio segments that do not require correction at the word level. This allows the target audio processing model to utilize the contextual information of non-dissimilar audio segments when correcting audio, providing a reference for correcting differing audio segments. This improves the accuracy of correction and allows control over the duration of the generated corrected audio segments, avoiding audio-visual desynchronization caused by duration variations in actual applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-differentiated audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-differentiated audio segments in terms of timbre and rhythm, thereby improving the overall naturalness and coherence of the target audio.

[0202] The above is an illustrative scheme of a training method for an audio processing model applied to a cloud-side device according to this embodiment. It should be noted that the technical solution of this training method for an audio processing model applied to a cloud-side device belongs to the same concept as the technical solution of the audio processing model training method described above. For details not described in detail in the technical solution of the audio processing model training method applied to a cloud-side device, please refer to the description of the technical solution of the audio processing model training method described above.

[0203] Corresponding to the above method embodiments, this specification also provides embodiments of audio processing apparatus. Figure 7 A schematic diagram of an audio processing apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The first acquisition module 702 is configured to acquire an initial audio containing at least one differential audio segment, wherein the differential audio segment represents an audio segment whose text corresponds to the initial audio and differs from the standard text. The first construction module 704 is configured to construct a sequence of words to be processed based on the word sequences corresponding to each differential audio segment and at least one non-differential audio segment. The first input module 706 is configured to input the word sequence to be processed into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model. The first update module 708 is configured to update the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio.

[0204] Optionally, the first acquisition module 702 is further configured to acquire the initial text corresponding to the initial audio; compare the initial text and the standard text to obtain at least one difference text segment in the standard text; and determine the difference audio segment corresponding to each difference text segment based on the time interval of each difference text segment in the initial audio.

[0205] Optionally, the first acquisition module 702 is further configured to compare the initial text and the standard text, so as to take the text segments that differ between the standard text and the initial text as text segments to be processed; and to adjust the word segmentation coverage of each text segment in the standard text according to the text difference type corresponding to each text segment to be processed, so as to obtain the difference text segments corresponding to each text segment to be processed.

[0206] Optionally, the first acquisition module 702 is further configured to, when determining that the text difference type corresponding to the target text segment to be processed is any one of missing segmentation, multiple-read segmentation, or misread segmentation, acquire a text segment composed of related segments in the standard text that are associated with the missing segmentation, and take the text segment composed of the related segments and the target text segment to be processed as the difference text segment corresponding to the target text segment to be processed; or, when determining that the text difference type corresponding to the target text segment to be processed is a numeric segmentation error, take the text segment composed of all numeric segments corresponding to the numeric region containing the erroneous numeric segmentation in the standard text as the difference text segment corresponding to the target text segment to be processed; wherein, the target text segment to be processed is any one of the text segments to be processed.

[0207] Optionally, the first acquisition module 702 is further configured to acquire the first time interval of each word in the standard text in the initial audio; for each difference text segment, merge the first time intervals of each word contained in the difference text segment to obtain the second time interval of the difference text segment in the initial audio, and take the audio segment in the initial audio corresponding to the second time interval as the difference audio segment corresponding to the difference text segment.

[0208] Optionally, the first construction module 704 is further configured to obtain the masked word sequence corresponding to each differing audio segment and the non-masked word sequence corresponding to at least one non-disagreeable audio segment; and to construct the word sequence to be processed based on each masked word sequence and at least one non-masked word sequence.

[0209] Optionally, the first construction module 704 is further configured to: obtain an initial word sequence corresponding to the initial audio; determine a differential word subsequence corresponding to each differential audio segment in the initial word sequence, and determine a non-differential word subsequence corresponding to each non-differential audio segment in the initial word sequence; determine a mask word sequence corresponding to each differential audio segment based on the differential word subsequence corresponding to each differential audio segment, and determine a non-masked word sequence corresponding to each non-differential audio segment based on the non-differential word subsequence corresponding to each non-differential audio segment.

[0210] Optionally, the first construction module 704 is further configured to perform masking processing on the lexical units in the differential lexical subsequences corresponding to each differential audio segment to obtain the masked lexical sequence corresponding to each differential audio segment.

[0211] Optionally, the first construction module 704 is further configured to, for each differential audio segment, obtain a mask marker with the same number of lexical units in the differential lexical subsequence corresponding to the differential audio segment, and use each obtained mask marker as the mask lexical sequence corresponding to the differential audio segment.

[0212] Optionally, the first construction module 704 is further configured to concatenate the masked word sequence corresponding to each differing audio segment and the non-masked word sequence corresponding to at least one non-disagreeing audio segment in the time order of each differing audio segment and at least one non-disagreeing audio segment in the initial audio to obtain the word sequence to be processed.

[0213] Optionally, the first input module 706 is further configured to input the word sequence to be processed into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed output by the prediction network; and input each predicted word sequence into the decoding network in the target audio processing model to obtain the corrected audio segment corresponding to each difference audio segment output by the decoding network.

[0214] Optionally, the first input module 706 is further configured to input the word sequence to be processed into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed output in parallel by the prediction network; if each predicted word sequence satisfies a preset condition, obtain the predicted word sequence corresponding to each mask word sequence; if any predicted word sequence does not satisfy the preset condition, backfill the predicted word sequence that satisfies the preset condition into the word sequence to be processed to obtain the backfilled word sequence, and use the backfilled word sequence as the word sequence to be processed, and execute the input of the word sequence to be processed into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed output in parallel by the prediction network.

[0215] Optionally, the first update module 708 is further configured to replace each difference audio segment in the initial audio with the corrected audio segment corresponding to each difference audio segment, so as to obtain the target audio corresponding to the initial audio.

[0216] Optionally, the first update module 708 is further configured to, when it is determined that the target audio does not meet the review conditions, determine the initial time interval of each different audio segment in the initial audio and expand each initial time interval to obtain the adjusted audio segment of each different audio segment in the initial audio; and regenerate the target audio based on each adjusted audio segment until the regenerated target audio meets the review conditions.

[0217] The aforementioned audio processing device can correct discrepancies in the initial audio, avoiding the need for indiscriminate regeneration of the entire audio, thus improving generation efficiency and saving computational resources. By constructing a sequence of words to be processed based on the word sequences corresponding to each discrepancy and non-discrepancy audio segment, the target audio processing model can utilize the contextual information of the non-discrepancy audio segments when correcting the audio, providing a reference for correcting the discrepancy audio segments. This improves the accuracy of the correction while controlling the duration of the generated corrected audio segments, avoiding audio-visual asynchrony caused by duration changes in practical applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-discrepancy audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-discrepancy audio segments in terms of timbre and rhythm, improving the overall naturalness and coherence of the target audio.

[0218] The above is an illustrative embodiment of an audio processing apparatus according to this invention. It should be noted that the technical solution of this audio processing apparatus and the technical solution of the aforementioned audio processing method belong to the same concept. Details not described in detail in the technical solution of the audio processing apparatus can be found in the description of the technical solution of the aforementioned audio processing method.

[0219] Corresponding to the above method embodiments, this specification also provides embodiments of an audio processing device located on a cloud-side device. Figure 8 A schematic diagram of an audio processing apparatus located on a cloud-side device according to one embodiment of this specification is shown. Figure 8 As shown, the device includes: The second acquisition module 802 is configured to acquire an audio processing request sent by the end device, wherein the audio processing request carries an initial audio containing at least one differential audio segment, the differential audio segment representing an audio segment whose text corresponding to the initial audio differs from the standard text. The second construction module 804 is configured to construct a sequence of words to be processed based on the word sequences corresponding to each differential audio segment and at least one non-differential audio segment. The second input module 806 is configured to input the word sequence to be processed into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model. The second update module 808 is configured to update the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio. The first transmitting module 810 is configured to transmit the target audio to the end-side device.

[0220] The aforementioned audio processing device located on the cloud side can correct discrepancies in the initial audio, avoiding the need for indiscriminate regeneration of the entire audio, thus improving generation efficiency and saving computational resources. By constructing a sequence of words to be processed based on the word sequences corresponding to each discrepancy and non-discrepancy audio segment, the target audio processing model can utilize the contextual information of the non-discrepancy audio segments when correcting the audio, providing a reference for correcting the discrepancy audio segments. This improves the accuracy of the correction while controlling the duration of the generated corrected audio segments, avoiding audio-visual asynchrony caused by duration changes in practical applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-discrepancy audio segments when generating corrected audio segments, the generated corrected audio segments can maintain consistency with the non-discrepancy audio segments in terms of timbre and rhythm, improving the overall naturalness and coherence of the target audio.

[0221] The above is an illustrative scheme of an audio processing apparatus located on a cloud-side device according to this embodiment. It should be noted that the technical solution of this audio processing apparatus located on a cloud-side device belongs to the same concept as the technical solution of the audio processing method located on a cloud-side device described above. For details not described in detail in the technical solution of the audio processing apparatus located on a cloud-side device, please refer to the description of the technical solution of the audio processing method located on a cloud-side device described above.

[0222] Corresponding to the above method embodiments, this specification also provides embodiments of an audio processing model training device. Figure 9 A schematic diagram of a training apparatus for an audio processing model according to one embodiment of this specification is shown. Figure 9 As shown, the device includes: The third acquisition module 902 is configured to acquire a sample word sequence, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio. The third input module 904 is configured to input the sample word sequence into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model, wherein each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence; The first training module 906 is configured to train the initial audio processing model based on the masked word subsequences of each sample and the standard word subsequences of each sample masked word subsequence in the global word sequence of the samples, so as to obtain the target audio processing model.

[0223] The training device for the aforementioned audio processing model constructs a training data format consistent with practical applications. It trains the initial audio processing model using a sample word sequence containing both masked and unmasked word sequences. This enables the model to learn to predict and correct masked regions in parallel while keeping unmasked regions unchanged. The trained target audio processing model can execute the corresponding steps in the aforementioned audio processing method. It constructs a word sequence to be processed based on the masked word sequences corresponding to each differing audio segment and the unmasked word sequences corresponding to non-dissimilar audio segments. This distinguishes between differing audio segments requiring correction and non-dissimilar audio segments that do not require correction at the word level. The target audio processing model can utilize the contextual information of non-dissimilar audio segments when correcting audio, thus providing a reference for correcting differing audio segments. This improves the accuracy of correction and allows control over the duration of the generated corrected audio segments, avoiding audio-visual asynchrony caused by duration variations in practical applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-differentiated audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-differentiated audio segments in terms of timbre and rhythm, thereby improving the overall naturalness and coherence of the target audio.

[0224] The above is a schematic scheme of an audio processing model training device according to this embodiment. It should be noted that the technical solution of this audio processing model training device and the technical solution of the audio processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the audio processing model training device, please refer to the description of the technical solution of the audio processing model training method described above.

[0225] Corresponding to the above method embodiments, this specification also provides an embodiment of a training device for an audio processing model located on a cloud-side device. Figure 10 This specification illustrates a schematic diagram of a training apparatus for an audio processing model located on a cloud-side device, according to one embodiment of this specification. Figure 10 As shown, the device includes: The fourth acquisition module 1002 is configured to acquire a model training request sent by the end device, wherein the model training request carries a model identifier of the initial audio processing model. The fifth acquisition module 1004 is configured to acquire the sample word sequence corresponding to the initial audio processing model, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio. The fourth input module 1006 is configured to input the sample word sequence into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model, wherein each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence; The second training module 1008 is configured to train the initial audio processing model based on the masked word subsequences of each sample and the standard word subsequences of each sample masked word subsequence in the global word sequence of the sample, and to obtain the target audio processing model and the model parameters of the target audio processing model. The second sending module 1010 is configured to send the model parameters of the target audio processing model to the end-side device.

[0226] The training device for the audio processing model located on the cloud-side device described above, by constructing a training data format consistent with actual applications, trains the initial audio processing model using a sample word sequence containing sample masked word subsequences and sample unmasked word subsequences. This enables the model to learn to perform parallel prediction and correction of masked regions while keeping unmasked regions unchanged. The trained target audio processing model can execute the corresponding steps in the aforementioned audio processing method. It constructs a word sequence to be processed based on the masked word sequence corresponding to each differing audio segment and the unmasked word sequence corresponding to each non-dissimilar audio segment. This distinguishes the differing audio segments requiring correction from the non-dissimilar audio segments that do not require correction at the word level. This allows the target audio processing model to utilize the contextual information of the non-dissimilar audio segments when correcting audio, thus providing a reference for correcting the differing audio segments. This improves the accuracy of the correction while controlling the duration of the generated corrected audio segments, avoiding audio-visual asynchrony caused by duration variations in actual applications, specifically the misalignment of the target audio with its corresponding video frame. Furthermore, since the target audio processing model can utilize the contextual information provided by the non-differentiated audio segments when generating the corrected audio segments, the generated corrected audio segments can maintain consistency with the non-differentiated audio segments in terms of timbre and rhythm, thereby improving the overall naturalness and coherence of the target audio.

[0227] The above is an illustrative scheme of a training device for an audio processing model located on a cloud-side device according to this embodiment. It should be noted that the technical solution of this training device for an audio processing model located on a cloud-side device belongs to the same concept as the technical solution of the above-described training method for an audio processing model applied to a cloud-side device. For details not described in detail in the technical solution of the training device for an audio processing model located on a cloud-side device, please refer to the description of the technical solution of the above-described training method for an audio processing model applied to a cloud-side device.

[0228] Figure 11 A structural block diagram of a computing device 1100 provided according to one embodiment of this specification is shown.

[0229] The computing device 1100 includes: Memory 1110 and processor 1120; The memory 1110 is used to store computer programs / instructions, and the processor 1120 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1120, they implement the steps of the audio processing method and the training method of the audio processing model.

[0230] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0231] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, audio processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0232] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0233] Figure 12 A structural block diagram of an electronic device 1200 provided according to one embodiment of this specification is shown.

[0234] The memory 1210 and the processor 1220 are connected via a bus 1230; The memory 1210 is used to store computer programs / instructions, and the processor 1220 is used to execute the computer programs / instructions, which, when executed by the processor 1220, implement the steps of the method.

[0235] Specifically, the components of the electronic device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0236] Electronic device 1200 also includes access device 1240, which enables electronic device 1200 to communicate via one or more networks 1260. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 1240 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0237] In one embodiment of this specification, the above-described components of the electronic device 1200 and Figure 12 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 12The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0238] Electronic device 1200 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 1200 can also be a mobile or stationary server.

[0239] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the technical solutions of the audio processing method and the audio processing model training method described above. For details not described in detail in the technical solution of the electronic device, please refer to the description of the technical solutions of the audio processing method and the audio processing model training method described above.

[0240] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method and the training method for the audio processing model.

[0241] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the audio processing method and the audio processing model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the audio processing method and the audio processing model training method described above.

[0242] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described audio processing method and the training method for the audio processing model.

[0243] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-mentioned audio processing method and audio processing model training method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-mentioned audio processing method and audio processing model training method.

[0244] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0245] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0246] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0247] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0248] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An audio processing method, comprising: Obtain initial audio containing at least one differential audio segment, wherein the differential audio segment represents an audio segment whose text corresponds to the initial audio and differs from the standard text; Based on the word sequences corresponding to each differential audio segment and at least one non-differential audio segment, construct the word sequence to be processed; The sequence of words to be processed is input into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model; The initial audio is updated based on each corrected audio segment to obtain the target audio corresponding to the initial audio.

2. The method of claim 1, wherein the difference audio segments are determined in the following manner: Obtain the initial text corresponding to the initial audio; By comparing the initial text and the standard text, at least one differing text segment in the standard text is obtained; Based on the time interval of each differing text segment in the initial audio, determine the differing audio segment corresponding to each differing text segment.

3. The method of claim 2, wherein comparing the initial text and the standard text to obtain at least one differing text segment in the standard text, comprises: The initial text and the standard text are compared to identify text segments that differ from the initial text and are to be processed. Based on the text difference type corresponding to each text segment to be processed, the word segmentation coverage of each text segment to be processed in the standard text is adjusted to obtain the difference text segment corresponding to each text segment to be processed.

4. The method as described in claim 3, wherein the word segmentation coverage of each text segment to be processed in the standard text is adjusted according to the text difference type corresponding to each text segment to be processed, to obtain the difference text segment corresponding to each text segment to be processed, includes: If the text difference type corresponding to the target text segment is determined to be any one of missing segmentation, multiple-read segmentation, or misread segmentation, then the text segment composed of related segments in the standard text that are related to the missing segmentation is obtained, and the text segment composed of the related segments and the target text segment is taken as the difference text segment corresponding to the target text segment; or, If the text difference type corresponding to the target text segment to be processed is determined to be a numeric word segmentation error, the text segment consisting of the full numeric word segmentation corresponding to the numeric region containing the erroneous numeric word segmentation in the standard text is taken as the difference text segment corresponding to the target text segment to be processed. The target text segment to be processed is any one of the text segments to be processed.

5. The method as described in claim 2, wherein determining the corresponding difference audio segment based on the time interval of each difference text segment in the initial audio, includes: Obtain the first time interval of each word segment in the standard text within the initial audio; For each differing text segment, the first time intervals of each word segment contained in the differing text segment are merged to obtain the second time interval of the differing text segment in the initial audio, and the audio segment in the initial audio that corresponds to the second time interval is taken as the differing audio segment corresponding to the differing text segment.

6. The method as described in claim 1, comprising constructing a word sequence to be processed based on the word sequences corresponding to each differing audio segment and at least one non-differential audio segment, comprising: Obtain the masked word sequence corresponding to each differential audio segment and the non-masked word sequence corresponding to at least one non-differential audio segment; Based on each masked word sequence and at least one non-masked word sequence, construct the word sequence to be processed.

7. The method as described in claim 6, wherein the masked word sequence corresponding to the differing audio segment and the non-masked word sequence corresponding to the non-disagreeing audio segment are obtained in the following manner: Obtain the initial word sequence corresponding to the initial audio; In the initial lexical sequence, determine the differential lexical subsequences corresponding to each differential audio segment, and in the initial lexical sequence, determine the non-differential lexical subsequences corresponding to each non-differential audio segment; Based on the differential word subsequences corresponding to each differential audio segment, determine the mask word sequence corresponding to each differential audio segment; and based on the non-differential word subsequences corresponding to each non-differential audio segment, determine the non-mask word sequence corresponding to each non-differential audio segment.

8. The method as described in claim 7, wherein determining the masked word sequence corresponding to each differing audio segment based on the differing word subsequence corresponding to each differing audio segment includes: The lexical units in the subsequence of lexical units corresponding to each lexical unit of the lexical unit are masked to obtain the masked lexical unit sequence corresponding to each lexical unit of the lexical unit.

9. The method of claim 7, wherein determining the masked word sequence corresponding to each differing audio segment based on the differing word subsequence corresponding to each differing audio segment includes: For each differing audio segment, obtain a mask marker with the same number of tokens as the differing token subsequence corresponding to that differing audio segment, and use each obtained mask marker as the mask token sequence corresponding to that differing audio segment.

10. The method of claim 6, wherein a sequence of words to be processed is constructed based on each masked word sequence and at least one unmasked word sequence, comprising: According to the temporal order of each differing audio segment and at least one non-discrete audio segment in the initial audio, the masked word sequence corresponding to each differing audio segment and the non-masked word sequence corresponding to at least one non-discrete audio segment are concatenated to obtain the word sequence to be processed.

11. The method of claim 6, wherein the sequence of words to be processed is input into a target audio processing model to obtain corrected audio segments corresponding to each differing audio segment output by the target audio processing model, comprising: The word sequence to be processed is input into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed output by the prediction network. Each predicted word sequence is input into the decoding network of the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the decoding network.

12. The method of claim 11, wherein the word sequence to be processed is input into a prediction network in a target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed output by the prediction network, comprising: The word sequence to be processed is input into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed, which is output in parallel by the prediction network. If all predicted word sequences meet the preset conditions, the predicted word sequences corresponding to each mask word sequence are obtained. If any predicted word sequence does not meet the preset conditions, the predicted word sequence that meets the preset conditions is backfilled into the word sequence to be processed to obtain the backfilled word sequence. The backfilled word sequence is then used as the word sequence to be processed. The word sequence to be processed is then input into the prediction network in the target audio processing model to obtain the predicted word sequence corresponding to each mask word sequence in the word sequence to be processed, which is output in parallel by the prediction network.

13. The method according to any one of claims 1 to 12, wherein the initial audio is updated according to each corrected audio segment to obtain the target audio corresponding to the initial audio, comprising: Replace each differing audio segment in the initial audio with the corresponding corrected audio segment to obtain the target audio corresponding to the initial audio.

14. The method according to any one of claims 1 to 12, after updating the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio, the method further includes: If the target audio does not meet the review conditions, the initial time interval of each difference audio segment in the initial audio is determined and each initial time interval is expanded to obtain the adjusted audio segment of each difference audio segment in the initial audio. The target audio is regenerated based on each adjusted audio segment until the regenerated target audio meets the review criteria.

15. An audio processing method applied to a cloud-side device, comprising: Obtain an audio processing request sent by the end device, wherein the audio processing request carries an initial audio containing at least one differential audio segment, the differential audio segment representing an audio segment whose text corresponding to the initial audio differs from the standard text; Based on the word sequences corresponding to each differential audio segment and each non-differential audio segment, construct the word sequence to be processed; The sequence of words to be processed is input into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model; The initial audio is updated according to each corrected audio segment to obtain the target audio corresponding to the initial audio. The target audio is sent to the end-side device.

16. A training method for an audio processing model, comprising: Obtain a sample word sequence, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio. The sample word sequence is input into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model. Each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence. The initial audio processing model is trained based on the masked lexical subsequences of each sample and the standard lexical subsequences of each sample masked lexical subsequence in the global lexical sequence of the sample, thereby obtaining the target audio processing model.

17. A training method for an audio processing model, applied to a cloud-based device, comprising: Obtain the model training request sent by the end device, wherein the model training request carries the model identifier of the initial audio processing model; Obtain the sample word sequence corresponding to the initial audio processing model, wherein the sample word sequence consists of at least one sample masked word subsequence and a sample non-masked word subsequence, and the sample word sequence is obtained by masking the sample global word sequence, wherein the sample global word sequence is the word sequence corresponding to the sample audio. The sample word sequence is input into the initial audio processing model to obtain the predicted audio segments corresponding to each sample mask word subsequence output by the initial audio processing model. Each predicted audio segment is obtained by the initial audio processing model decoding each sample predicted word sequence, and each sample predicted word sequence is obtained by the initial audio processing model performing parallel prediction on each sample mask word subsequence. The initial audio processing model is trained based on the masked word subsequences of each sample and the standard word subsequences of each sample masked word subsequence in the global word sequence of the sample, thereby obtaining the target audio processing model and the model parameters of the target audio processing model. The model parameters of the target audio processing model are sent to the end-side device.

18. An audio processing apparatus, comprising: The first acquisition module is configured to acquire initial audio containing at least one differential audio segment, wherein the differential audio segment represents an audio segment whose text corresponds to the initial audio and differs from the standard text. The first construction module is configured to construct the word sequence to be processed based on the word sequence corresponding to each differential audio segment and at least one non-differential audio segment. The first input module is configured to input the word sequence to be processed into the target audio processing model to obtain the corrected audio segments corresponding to each difference audio segment output by the target audio processing model. The first update module is configured to update the initial audio according to each corrected audio segment to obtain the target audio corresponding to the initial audio.

19. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 17.

20. An electronic device, comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 17.

21. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 17.

22. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 17.

Images