Audio processing method, device and equipment
By analyzing the prosodic features of the input speech stream in real time, the electronic device extracts speech segments for translation at the segmentation time that meets the semantic integrity condition, solving the problem of long delay in electronic device speech translation and achieving faster and more accurate translation.
Patent Information
- Application Number
- CN202511777411.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-27
AI Technical Summary
Existing technologies for electronic devices suffer from significant delays in speech translation, especially since translation begins only after the speaker has finished speaking, resulting in substantial delays.
By analyzing the prosodic features of the input speech stream in real time, the segmentation time that meets the semantic integrity condition is determined, and speech segments are extracted for translation, avoiding waiting for the entire sentence to end.
It effectively shortens the delay in speech translation, ensures the accuracy and semantic integrity of the translation results, and reduces translation latency.
Smart Images
Figure CN121583239A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of audio processing technology, and in particular to an audio processing method, apparatus and device. Background Technology
[0002] Some electronic devices have translation capabilities, enabling them to receive input speech, translate it into another language, and output the result. However, a significant delay exists in the translation process using these electronic devices. Summary of the Invention
[0003] Therefore, this application discloses the following technical solution:
[0004] The first aspect of this application provides an audio processing method, including:
[0005] Obtain the input speech stream;
[0006] When the duration of the input speech stream reaches the segmentation threshold, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream.
[0007] Extract the input speech stream obtained up to the segmentation time as a speech segment;
[0008] For each obtained speech segment, the speech segment is translated to obtain a processed speech segment. The language of the processed speech segment is different from the language of the original speech segment.
[0009] Optionally, when the duration of the input speech stream reaches a segmentation threshold, determining the segmentation time that satisfies the semantic integrity condition based on the prosodic features of the input speech stream in real time includes:
[0010] When the duration of the input speech stream reaches the segmentation threshold, silence detection is performed on the input speech stream to detect the pause time of the input speech stream;
[0011] In the absence of detected pauses in the input speech stream, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream.
[0012] Also includes:
[0013] If a pause in the input speech stream is detected, the input speech stream obtained up to the pause time is extracted as a speech segment.
[0014] Optionally, determining the segmentation time that satisfies the semantic integrity condition based on the prosodic features of the input speech stream in real time includes:
[0015] The pitch features of the input speech stream are extracted in real time to obtain the prosodic features at the current moment based on the pitch features. The prosodic features at the current moment represent the pitch change trend of the input speech stream within a certain time period up to the current moment.
[0016] If the prosodic features at the current moment match the reference prosodic features, the current moment is determined to be a segmentation moment that satisfies the semantic integrity condition. The reference prosodic features include prosodic features obtained from sample speech segments, and the sample speech segments are pre-annotated speech segments with complete semantics.
[0017] Optionally, the translation processing of the speech segment to obtain the processed speech segment includes:
[0018] The first text data of the corresponding first language of the speech segment is obtained based on speech recognition;
[0019] Process the first text data to obtain the second text data in the corresponding second language;
[0020] Processed speech segments are generated based on the second text data.
[0021] Optionally, processing the first text data to obtain second text data in the corresponding second language includes:
[0022] Perform text repair on the first text data to obtain the repaired first text data;
[0023] Based on the repaired first text data, a language conversion is performed to obtain second text data in the corresponding second language.
[0024] Optionally, the step of performing text repair on the first text data to obtain repaired first text data includes:
[0025] Predicted text data is determined based on the first text data and historical text data, wherein the predicted text data represents the text data corresponding to the next speech segment to be obtained in the future.
[0026] Based on the historical text data and the predicted text data, the first text data is repaired to obtain the repaired first text data.
[0027] Optionally, the first text data includes a first text encoding sequence, the first text encoding sequence includes a plurality of first text codes, and the first text codes represent the lexical units of the first language contained in the speech segment;
[0028] The second text data includes a second text encoding sequence, which includes multiple second text codes, and the second text codes represent lexical units of a second language.
[0029] Optionally, generating the processed speech segment based on the second text data includes:
[0030] Based on any one or more of the timbre, pitch, spectrum, and volume of the speech segment, the audio features of the speech segment are obtained;
[0031] Based on the second text data and the audio features of the speech segment, a processed speech segment is generated; wherein the speech content of the processed speech segment matches the second text data, and the audio features of the processed speech segment match the audio features of the speech segment.
[0032] A second aspect of this application provides an audio processing apparatus, comprising:
[0033] The acquisition unit is used to acquire the input speech stream;
[0034] The determining unit is used to determine, in real time, the segmentation time that satisfies the semantic integrity condition based on the prosodic features of the input speech stream when the duration of the input speech stream reaches the segmentation threshold.
[0035] The segmentation unit is used to extract the input speech stream obtained up to the segmentation time as a speech segment;
[0036] The translation unit is used to translate each acquired speech segment to obtain a processed speech segment, wherein the language of the processed speech segment is different from the language of the original speech segment.
[0037] A third aspect of this application provides an electronic device, comprising: a acquisition module for acquiring an input voice stream;
[0038] Memory, used to store computer programs;
[0039] A processor for executing the computer program to perform:
[0040] When the duration of the input speech stream reaches the segmentation threshold, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream.
[0041] Extract the input speech stream obtained up to the segmentation time as a speech segment;
[0042] For each obtained speech segment, the speech segment is translated to obtain a processed speech segment. The language of the processed speech segment is different from the language of the original speech segment. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0044] Figure 1 This is a flowchart of an audio processing method provided in an embodiment of this application;
[0045] Figure 2 This is a flowchart of another audio processing method provided in the embodiments of this application;
[0046] Figure 3 This is a flowchart illustrating how to obtain a processed speech segment, as provided in an embodiment of this application.
[0047] Figure 4 This is an example diagram of an audio processing method provided in an embodiment of this application;
[0048] Figure 5 This is a schematic diagram of an audio processing device provided in an embodiment of this application;
[0049] Figure 6 This is a schematic diagram of a translation unit provided in an embodiment of this application;
[0050] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0052] This application provides an audio processing method. Please refer to [link / reference]. Figure 1 Here is a flowchart of the method, which may include the following steps.
[0053] S101, obtain the input speech stream.
[0054] The audio processing method of this embodiment can be executed by any electronic device with audio processing capabilities, and the type of electronic device is not limited. For example, the audio processing method of this embodiment can be executed by electronic devices such as mobile phones, tablets, dedicated translation devices (e.g., translation pens), and conference machines used for accessing online meetings.
[0055] An input speech stream is a stream of speech data acquired by an electronic device from the speech produced by a user (i.e., the speaker). Optionally, the input speech stream can also be a speech stream received in real time from other devices via a network.
[0056] S102, when the duration of the input speech stream reaches the segmentation threshold, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream.
[0057] S103, extract the input speech stream obtained up to the segmentation time as a speech segment.
[0058] S104. For each obtained speech segment, the speech segment is translated to obtain a processed speech segment. The language of the processed speech segment is different from the language of the original speech segment.
[0059] The beneficial effects of the audio processing method in this embodiment will be explained below.
[0060] In related technologies, electronic devices with translation functions need to obtain a complete sentence before translating it to ensure the accuracy of the translation results. Therefore, when the speaker's sentence is long, the translation by the electronic device will have a significant delay. For example, if the speaker takes 6 seconds to finish speaking a sentence, there will be at least a 6-second delay from the time the speaker begins speaking to the time the electronic device starts outputting the translated speech.
[0061] After applying the processing method of this embodiment, in the first aspect, when the speaker’s utterance is long enough to cause the duration of the input speech stream to exceed the segmentation threshold, the method of this embodiment can obtain the speech segment based on the segmentation time and start translation, without having to wait for the speaker to finish speaking the entire utterance before starting translation. Compared with related technologies that start translation only after the entire utterance is finished, it can obviously effectively shorten the translation delay. In general, the translation delay can be controlled to a level close to the segmentation threshold.
[0062] Secondly, the segmentation time determined by the method in this embodiment is determined based on the prosodic features of the input speech stream and satisfies the semantic integrity condition. In other words, the speech segment obtained based on the segmentation time has relatively complete semantics. This can make the subsequent translation processing of the speech segment more accurate and avoid the situation where the semantics of the processed speech segment after translation is inconsistent with the semantics of the original speech segment due to the incomplete semantics of the speech segment.
[0063] For example, assuming the input voice stream includes the statement "Since the beginning of this year, the group company's import and export trade volume has increased by 10% compared with the same period last year", the method based on this embodiment can determine the segmentation time between "import and export trade" and "business volume", instead of determining the segmentation time within the phrase "import and export trade".
[0064] In summary, the audio processing method of this embodiment can effectively shorten the latency when an electronic device translates a real-time input audio stream while ensuring the accuracy of the translated speech.
[0065] The input speech stream can be acquired by an electronic device through an acquisition module used for audio acquisition. The method in this embodiment can be executed from the moment the acquisition module acquires the input speech stream, and is executed in real time during the acquisition of the input speech stream, ending when the acquisition module stops acquiring the input speech stream, that is, when the input speech stream stops.
[0066] In step S102, the duration of the input voice stream can be the cumulative duration from the start of acquisition by the acquisition module to the present, and can be reset and re-accumulated each time the electronic device captures a voice segment according to the method of this embodiment.
[0067] When executing S103 to extract a speech segment, if it is the first extraction after the start of the current input speech stream, the speech stream from the start of the input speech stream to the segmentation time is extracted as the first speech segment. If it is not the first extraction after the start of the current input speech stream, the speech stream from the end of the previous speech segment to the currently determined segmentation time can be extracted as the speech segment obtained in this extraction.
[0068] Taking a segmentation threshold of 2.5 seconds as an example, the acquisition module starts acquiring the input speech stream at time T0. If the input speech stream has not stopped at T0+2.5 seconds, the electronic device starts determining the segmentation time according to S102, setting T1, which is 2.7 seconds away from T0, as the segmentation time. The speech between T0 and T1 is extracted as a speech segment for subsequent translation processing, and the duration is re-accumulated from T1. If the input speech stream has not stopped at T1+2.5 seconds, another segmentation time T2 is determined according to S102, and the speech segment between T1 and T2 is extracted. The duration is re-accumulated from T2, and so on, until the input speech stream stops. Thus, the electronic device can obtain a translated and processed speech segment approximately every 2.5 seconds (or other segmentation thresholds).
[0069] If the duration of the input speech stream does not reach the segmentation threshold, that is, if the input speech stream stops before the duration reaches the segmentation threshold, then the speech that has not been translated up to the time when the input speech stream stops can be taken as the last speech segment. The last speech segment is translated to obtain the last processed speech segment. At this point, the audio processing method of this embodiment ends.
[0070] The specific value of the segmentation threshold can be set as needed and is not limited. If the shortest possible translation delay is required, a smaller segmentation threshold should be set, such as 1 second. If the power consumption of electronic devices needs to be reduced and the accuracy of the processed speech segments obtained from the translation needs to be improved, a larger segmentation threshold can be set, such as 2.5 seconds.
[0071] A segmentation moment that satisfies the semantic integrity condition refers to a moment in which the speech before this moment has relatively complete semantics. The semantics expressed by the speech before this moment and the semantics expressed by the speech after this moment are relatively independent. If the speech before and after this moment is segmented and translated separately, it will not affect the original semantics. The moment that satisfies the above conditions is the segmentation moment that satisfies the semantic integrity condition in this embodiment.
[0072] Generally, in an input speech stream, the moments corresponding to the end of words, common phrases, and fixed sentence patterns are identified as segmentation moments that satisfy the semantic integrity condition, while the moments in the middle of words, common phrases, and fixed sentence patterns are identified as moments that do not satisfy the semantic integrity condition. As an example, suppose the input speech stream contains the sentence "This is truly a masterpiece of craftsmanship," then the moment between "masterful craftsmanship" and "the masterpiece" might be identified as a segmentation moment that satisfies the semantic integrity condition, while the moment between "masterful craftsmanship" and "masterful craftsmanship" would be identified as a moment that does not satisfy the semantic integrity condition.
[0073] For each acquired speech segment, the electronic device can translate it to obtain a corresponding processed speech segment. The processed speech segment and the original speech segment are in different languages, but their semantics are identical. For example, the input speech stream and the extracted speech segment can be in Chinese, while the processed speech segment can be in English.
[0074] The application of processed audio segments by electronic devices is not limited. For example, upon acquiring a processed audio segment, the electronic device can directly output the processed audio segment based on an output module, such as a speaker, a Bluetooth speaker connected to a communication network, or headphones. The electronic device can play the processed audio segment using the speaker, speaker, or headphones each time it acquires one. And / or, in an online meeting scenario, upon acquiring a processed audio segment, the electronic device can send the processed audio segment to other electronic devices via a communication network, where it can be output by those other electronic devices.
[0075] Optional, please see Figure 2 When the duration of the input speech stream reaches the segmentation threshold, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream, including:
[0076] S202, when the duration of the input speech stream reaches the segmentation threshold, silence detection is performed on the input speech stream to detect the pause time of the input speech stream;
[0077] S203, in the absence of detected pauses in the input speech stream, determine the segmentation time that satisfies the semantic integrity condition in real time based on the prosodic features of the input speech stream;
[0078] Based on silence detection, the method in this embodiment may further include:
[0079] S204, if a pause in the input speech stream is detected, the input speech stream obtained up to the pause time is extracted as a speech segment.
[0080] Silence detection, also known as Voice Activity Detection (VAD), works by detecting the audio energy at each moment in the audio stream in real time. If the detected audio energy at a certain moment is less than or equal to a certain threshold (either a static or dynamic threshold), then that moment is determined to be a silent moment where no sound was detected. In this embodiment, the detected silence moments can be used as the pause moments for short pauses during the speaker's speech.
[0081] One method for detecting the audio energy at each moment is to acquire each audio frame of the audio stream in real time at certain frame intervals (e.g., 100 milliseconds, 50 milliseconds, etc.). For each audio frame, the detection result, based on the signal energy, amplitude, or other indicators of that audio frame using relevant technologies, is used as the audio energy at the corresponding moment. Each audio frame corresponds to a short segment of audio signal with a duration equal to the frame interval.
[0082] The specific implementation method of silence detection is not limited. Silence detection can be performed in real time using a pre-built neural network model capable of performing silence detection, or in real time using algorithms for calculating audio energy in related technologies to achieve silence detection.
[0083] In this embodiment, when the duration of the input speech stream reaches the segmentation threshold, the electronic device first performs silence detection on the input speech stream in real time. If no pause is detected, it continues to detect the segmentation time based on prosodic features. If a pause is detected, it extracts the speech segment based on the pause time according to the scheme in S204.
[0084] The method for extracting speech segments based on pause times is the same as the method for extracting speech segments based on segmentation times, and the duration of the input speech stream is also reset to zero after extracting speech segments based on pause times so that it can be accumulated again.
[0085] The beneficial effect of this embodiment is that the speech segments obtained based on pause times are more in line with the speaker's speaking habits and contain more complete semantics than the speech segments obtained based on segmentation times. Therefore, prioritizing the extraction of speech segments based on pause times is beneficial to improving the accuracy of the processed speech segments obtained from translation and more accurately expressing the original semantics of the input speech stream.
[0086] The prosody of speech can be understood as the change of pitch over time. Therefore, in this embodiment, the pitch characteristics of the input speech stream at different times can be obtained in real time, and the prosodic characteristics of the input speech stream can be determined based on the changing patterns of pitch characteristics.
[0087] Therefore, in some optional embodiments, determining the segmentation time that satisfies the semantic integrity condition based on the prosodic features of the input speech stream in real time may include:
[0088] The pitch features of the input speech stream are extracted in real time to obtain the prosodic features at the current moment based on the pitch features. The prosodic features at the current moment represent the pitch change trend of the input speech stream within a certain time period up to the current moment.
[0089] If the prosodic features at the current moment match the reference prosodic features, the current moment is determined to be a segmentation moment that satisfies the semantic integrity condition. The reference prosodic features include prosodic features obtained from sample speech segments, which are pre-annotated speech segments with complete semantics.
[0090] The electronic device can acquire each audio frame of the input speech stream in real time at the aforementioned frame intervals and extract the pitch features of each audio frame. The pitch features of an audio frame can be understood as the pitch or frequency of that audio frame.
[0091] The trend of pitch change in the input speech stream over a certain period of time up to the current moment is equivalent to the trend of pitch feature changes in multiple consecutive audio frames over a certain period of time up to the current moment. For example, the prosodic features at the current moment can characterize how the pitch of multiple consecutive audio frames changes over a certain period of time, whether it continuously rises, continuously falls, rises then falls, falls then rises, remains basically unchanged, or fluctuates according to a certain period. It can also characterize the peaks, valleys, averages, the number of times peaks occur, and the number of times valleys occur in the pitch of multiple consecutive audio frames over a certain period of time.
[0092] The specific value of this "certain duration" can be pre-configured according to the actual situation and is not limited. If you need to improve the accuracy of the translation results, you can set a larger value; if you need to shorten the delay, you can set a smaller value. As an example, the "certain duration" can be set to 0.5 seconds, and the prosodic features at the current moment can represent the changing trend of the pitch features of multiple audio frames in the past 0.5 seconds up to the current moment.
[0093] After obtaining the prosodic features at the current moment, if the similarity between the prosodic features at the current moment and the reference prosodic features is greater than a preset threshold, then the prosodic features at the current moment and the reference prosodic features are successfully matched. If the similarity between the prosodic features at the current moment and the reference prosodic features is less than or equal to the preset threshold, then the prosodic features at the current moment and the reference prosodic features are not matched.
[0094] There can be multiple reference prosodic features. If the prosodic feature at the current moment matches any of the reference prosodic features, then the current moment is determined to be a segmentation moment that satisfies the semantic integrity condition. If the prosodic feature at the current moment fails to match any of the reference prosodic features, then the current moment is determined to be a segmentation moment that does not satisfy the semantic integrity condition.
[0095] The prosodic features can be obtained by means of prosodic features obtained from sample speech segments, which are pre-annotated speech segments with complete semantics.
[0096] The method described above for determining the segmentation time that satisfies the semantic integrity condition can be implemented using a pre-built and trained prosodic feature detection model, which can be a neural network model of any structure.
[0097] When training the prosodic feature detection model, pre-annotated sample speech segments can be input into the prosodic feature detection model to obtain the recognition result output by the prosodic feature detection model. The recognition result represents the probability that the end of the sample speech segment is a segmentation moment that satisfies the semantic integrity condition. Then, the model loss can be determined based on the difference between the recognition result and the actual annotation result of the sample speech segment. The prosodic feature detection model is trained based on the model loss. Through this iterative training process, the prosodic feature detection model learns how to extract the prosodic features of speech segments and can record the prosodic features of these sample speech segments through the model's own weight parameters.
[0098] Therefore, when it is necessary to determine the segmentation moment that satisfies the semantic integrity condition, the input speech stream can be input into the prosodic feature detection model in real time. The prosodic feature detection model can extract the prosodic features at the current moment in real time, output the similarity between the prosodic features at the current moment and the reference prosodic features, and thus determine whether the current moment is the segmentation moment that satisfies the semantic integrity condition.
[0099] The actual annotation result can be the real result determined by manual annotation. The actual annotation result represents the segmentation moment of the sample speech segment that satisfies the semantic integrity condition.
[0100] Optionally, the speech segment is translated to obtain a processed speech segment, including:
[0101] The first text data in the first language corresponding to the speech segment is obtained based on speech recognition;
[0102] Process the first text data to obtain the second text data in the corresponding second language;
[0103] Processed speech segments are generated based on the second text data.
[0104] In this embodiment, speech recognition technology based on neural network models can be used to identify speech segments and obtain first text data representing the speech content of the speech segments. The first language is the language corresponding to the input speech stream. For example, if the input speech stream is Chinese speech, then the obtained first text data is also text data representing Chinese content.
[0105] Processing the first text data to obtain the second text data can include directly performing language conversion on the first text data to obtain the second text data. The language conversion of the text data can be based on a pre-built neural network model with text translation capabilities, such as a large language model with translation functions or other neural network models. Specific implementation methods can be found in relevant technologies and will not be elaborated here.
[0106] The second language can be any language that the user pre-specifies, different from the first language. For example, if the first language is Chinese, the second language could be English, French, etc.
[0107] After obtaining the second text data, it can be input into a neural network model with speech synthesis capabilities, such as a speech encoder (Vocoder) model or other related models. This model will generate speech content corresponding to the second text data, and the generated speech can then be used as the processed speech segment described above. The training methods and working principles of neural network models with speech synthesis capabilities can be found in related technologies and will not be elaborated upon here.
[0108] In some alternative embodiments, the text data may include strings in the form of natural language. For example, the first text data may be the string "In a faraway place" in Chinese, a first language, and the second text data may be the string "In a faraway place" in English, a second language.
[0109] In some optional embodiments, the first text data includes a first text encoding sequence, which includes a plurality of first text codes, the first text codes representing the lexical units of a first language contained in the speech segment;
[0110] The second text data includes a second text encoding sequence, which includes multiple second text codes, and the second text codes represent the lexical units of the second language.
[0111] Among them, lexical units can include characters, words, or phrases in natural language, while text encoding is a unique numerical encoding determined for each lexical unit in a neural network model in order to represent lexical units.
[0112] Combining the aforementioned example, "in a faraway place" includes the word units "in", "a", "faraway", and "place". The first text codes corresponding to these word units can be "154", "3802", "62", and "9341" respectively. The first text data obtained through speech recognition can be the first text code sequence: "154, 3802, 62, 9341".
[0113] Correspondingly, the second text data may include a second text encoding sequence formed by the second text encoding of the four word elements "In", "a", "faraway", and "place".
[0114] Speech recognition, language conversion, and speech generation are all achieved through neural network models with corresponding functions. However, neural network models cannot recognize strings in natural language form; they can only recognize the corresponding codes, and the output is also a code. Therefore, if strings are used as text data, obtaining text data, as well as performing language conversion and speech generation on the text data, all require conversion between strings and text codes, which increases the time required to obtain the processed speech segments.
[0115] For the opposite, please see Figure 3 By treating the text encoding sequence as text data, the first text encoding sequence output from speech recognition can be directly input into a neural network model with language conversion capabilities, and the second text encoding sequence output from the neural network model with language conversion capabilities can be directly input into the model used to generate speech. This eliminates the time spent on converting between strings and text encoding, further shortening the time to obtain processed speech segments, which in turn reduces the translation delay.
[0116] In some optional embodiments, processing the first text data to obtain second text data corresponding to the second language may include:
[0117] Perform text repair on the first text data to obtain the repaired first text data;
[0118] Based on the repaired first text data, language conversion is performed to obtain the second text data in the corresponding second language.
[0119] In other words, when obtaining the second text data, the first text data is first repaired, and then the language of the repaired first text data is converted, rather than directly converting the language of the first text data.
[0120] Text repair of the first text data can be performed using a pre-trained text correction model with text correction capabilities. The specific working principle and training method of the text correction model can be found in relevant technologies and will not be elaborated here. As an example, in this embodiment, a text correction model can be constructed and trained using Bidirectional and Auto-Regressive Transformers (BART) based on the Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction (MuCGEC) to achieve text repair of the first text data.
[0121] Text repair processing of the first text data may include identifying and correcting any one or more text errors that may occur in the first text data, including but not limited to text duplication, semantic incompleteness, semantic errors, etc.
[0122] The beneficial effects of this embodiment are as follows:
[0123] Due to the influence of extracting speech segments from the input speech stream, speaker factors (such as stuttering or stammering), and network latency, the first text data may contain various text errors. If these text errors are ignored and language conversion is performed directly, the semantics of the obtained second text data may deviate significantly from the semantics of the first text data, resulting in inaccurate translation results.
[0124] The method in this embodiment performs text repair on the first text data before obtaining the second text data, which helps to avoid this situation and ensures that the semantics of the second text data obtained by language conversion is as close as possible to the semantics of the first text data, making the translation result more accurate.
[0125] In some optional embodiments, the first text data can be directly input into the text correction model to obtain the text data output by the text correction model after processing, which can then be used as the corrected first text data.
[0126] In some optional embodiments, text repair is performed on the first text data to obtain repaired first text data, which may also include:
[0127] The predicted text data is determined based on the first text data and the historical text data. The predicted text data represents the text data corresponding to the next speech segment obtained in the future.
[0128] Text repair is performed on the first text data based on historical text data and predicted text data to obtain the repaired first text data.
[0129] The historical text data in this embodiment may include all text data obtained before the first text data, from the moment the current input speech stream is first obtained until the current moment. For example, if the 10th speech segment has been extracted from the input speech stream, the text data obtained by speech recognition from the 10th speech segment is the first text data, and the text data obtained by speech recognition from the previous 1st to 9th speech segments is the historical text data.
[0130] When determining the predicted text data, the initial text data and historical text data can be input into a generative neural network model with text generation capabilities, such as a large language model. Pre-defined prompts are then used to instruct the generative neural network model to continue writing the text based on the input text data; that is, to predict the possible subsequent text data based on the input text data. The text data output by the generative neural network model is then used as the predicted text data. The working principle and training methods of generative neural network models with text generation capabilities can be found in related technologies and will not be elaborated here.
[0131] After obtaining the predicted text data, the historical text data, the first text data, and the predicted text data can be input into the text correction model to obtain the corrected first text data output by the text correction model.
[0132] The beneficial effects of this embodiment are as follows:
[0133] The first text data is obtained from only one speech segment and contains a small amount of information. If text repair is performed based solely on the first text data, the limited information may cause the text correction model to fail to correctly identify possible text errors in the first text data, resulting in inaccurate text repair results.
[0134] In this embodiment, text repair is performed on the first text data based on historical text data and predicted text data. This allows the text correction model to use historical text data and predicted text data as context for the first text data, increasing the amount of information input to the text correction model during text repair. This helps the text correction model to correctly identify possible text errors in the first text data and improves the accuracy of the text repair results.
[0135] Optionally, a processed speech segment is generated based on the second text data, including:
[0136] The audio features of a speech segment are obtained based on any one or more of its timbre, pitch, spectrum, and volume.
[0137] Based on the second text data and the audio features of the speech segment, a processed speech segment is generated; wherein the speech content of the processed speech segment matches the second text data, and the audio features of the processed speech segment match the audio features of the speech segment.
[0138] There are no limitations on the method for obtaining the audio features of a speech segment. Optionally, the speech segment can be divided into multiple audio frames according to the aforementioned frame intervals to obtain timbre data representing the timbre of each audio frame. The vector composed of the timbre data of multiple audio frames can be used as the timbre feature vector of the speech segment. Similarly, pitch feature vectors, spectral feature vectors, and volume feature vectors of the speech segment can be obtained. Finally, these feature vectors can be fused into a single feature vector by adding, weighting, or inputting into a neural network model. This fused feature vector is used as the audio feature of the speech segment.
[0139] When obtaining the processed speech segment, the audio features of the speech segment and the second text data can be input together into a neural network model with speech synthesis capabilities. This model then uses the audio features of the original speech segment as a reference to generate the processed speech segment. The resulting processed speech segment exhibits a high degree of similarity in audio features to the original speech segment extracted from the input speech stream; that is, the audio features of the processed speech segment match those of the original speech segment. For example, the processed speech segment may have similar timbre, pitch, spectrum, and volume to the original speech segment.
[0140] The advantage of obtaining the processed speech segment using the method described in the above embodiments is that:
[0141] The processed speech segments and the original speech segments, apart from the corresponding language conversion, have similar timbre, pitch, spectrum and volume characteristics to the original speech segments. This allows the processed speech segments to simulate the pronunciation characteristics of the speaker corresponding to the input speech stream, thus achieving the effect of simulating the actual speech of the speaker.
[0142] The implementation process of the audio processing method in this embodiment is illustrated below with an example.
[0143] Please see Figure 4 The electronic device receives the input speech stream and, after a continuous segmentation threshold of 2 seconds, begins to determine the segmentation time and extract speech segments according to steps S102 and S103. The first extracted speech segment (i.e. Figure 4 The Chinese audio segment 1) is 2.6 seconds long. After speech recognition, it yields a first text encoding sequence containing 22 first text codes, and its corresponding string is "In a faraway place...".
[0144] The first text encoding sequence corresponding to speech segment 1 is used as the first text data. This data undergoes text repair and language conversion to obtain the second text encoding sequence representing "In a faraway place...". Then, the second text encoding sequence is processed using a model with speech synthesis capabilities to obtain... Figure 4The first processed audio segment from the top center is output through the speaker of the electronic device;
[0145] While obtaining and outputting the processed speech segment in the above manner, the electronic device continues to obtain the input speech stream and extracts a 2.2-second speech segment based on the new segmentation time. Figure 4 Speech segment 2, speech segment 2 is used to obtain a first text encoding sequence containing 20 first text codes through speech recognition, the corresponding string is, for example, "It every day...", the first text encoding sequence is used to obtain a second text encoding sequence representing "Every day..." through text repair and language conversion, the second text encoding sequence is used to obtain a second processed speech segment through speech synthesis and output through a speaker;
[0146] After capturing speech segment 2, the input speech stream stops after 2.8 seconds. The speech stream after speech segment 2 for the next 2.8 seconds is captured as speech segment 3. Speech segment 3 is processed by speech recognition to obtain a first text encoding sequence containing 24 first text codes. The corresponding string is, for example, "one day, ...". The first text encoding sequence is processed by text repair and language conversion to obtain a second text encoding sequence representing "One day, ...". The second text encoding sequence is processed by speech synthesis to obtain a third processed speech segment and output through a speaker.
[0147] As can be seen from the above example, by applying the method of this embodiment, from the moment the input speech stream is collected (i.e., the moment the speaker begins to speak), the electronic device can output the first processed speech segment after translation after 2.6 seconds, and then continuously output the second and third processed speech segments until the input speech stream stops and the speaker stops speaking. In contrast, if a solution using related technologies is applied, it may take 7.6 seconds for the electronic device to start outputting the translated speech after the entire sentence has been spoken. It is evident that the method of this embodiment shortens the delay of real-time speech translation to about 2.6 seconds, effectively reducing the delay during real-time speech translation.
[0148] This application also provides an audio processing device; please refer to [link to relevant documentation]. Figure 5 The device may include the following units.
[0149] The acquisition unit 501 is used to acquire the input speech stream;
[0150] The determining unit 502 is used to determine the segmentation time that satisfies the semantic integrity condition in real time based on the prosodic features of the input speech stream when the duration of the input speech stream reaches the segmentation threshold.
[0151] The segmentation unit 503 is used to segment the input speech stream obtained up to the segmentation time as a speech segment;
[0152] The translation unit 504 is used to translate each acquired speech segment to obtain a processed speech segment. The language of the processed speech segment is different from the language of the original speech segment.
[0153] Please see below. Figure 6 The translation unit 504 may specifically include a speech recognition unit 601, a text repair unit 602, a machine translation unit 603, and a speech synthesis unit 604.
[0154] The speech recognition unit 601 is used to perform speech recognition on a obtained speech segment to obtain the first text data;
[0155] The text repair unit 602 is used to repair the first text data to obtain the repaired first text data;
[0156] Machine translation unit 603 is used to perform language conversion on the repaired first text data to obtain second text data;
[0157] The speech synthesis unit 604 is used to generate processed speech segments based on the second text data.
[0158] The specific working principle of the above audio processing device can be found in the relevant steps of the audio processing method in the foregoing embodiments, and will not be repeated here.
[0159] This application embodiment may include providing an electronic device; please refer to [link / reference]. Figure 7 The electronic device may include a acquisition module 701 for acquiring the input voice stream;
[0160] Memory 702 is used to store computer programs;
[0161] Processor 703 is used to execute computer programs to perform:
[0162] When the duration of the input speech stream reaches the segmentation threshold, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream.
[0163] Extract the input speech stream obtained up to the segmentation time as a speech segment;
[0164] For each obtained speech segment, the speech segment is translated to obtain a processed speech segment. The language of the processed speech segment is different from that of the original speech segment.
[0165] The acquisition module 701 can be the microphone of the electronic device itself, or it can be another microphone device that communicates with the electronic device, used to acquire the voice of the current user speaking in real time, and the acquired voice is used as the input voice stream.
[0166] The acquisition module 701 can also be a network module of an electronic device, used to receive voice streams acquired and sent by other electronic devices via a network as input voice streams.
[0167] Optionally, the electronic device in this embodiment may also include a communication module and an audio output module. The communication module is used to send the processed voice segments to other electronic devices that can output voice in real time, and the audio output module (e.g., a speaker) is used to output the obtained processed voice segments in real time.
[0168] The working principle of the electronic device in this embodiment can be found in the relevant steps of the audio processing method in any of the foregoing embodiments, and will not be repeated here.
[0169] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0170] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.
[0171] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0172] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0173] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An audio processing method, comprising: Obtain the input speech stream; When the duration of the input speech stream reaches the segmentation threshold, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream. Extract the input speech stream obtained up to the segmentation time as a speech segment; For each obtained speech segment, the speech segment is translated to obtain a processed speech segment. The language of the processed speech segment is different from the language of the original speech segment.
2. The method according to claim 1, wherein when the duration of the input speech stream reaches a segmentation threshold, determining in real time the segmentation time that satisfies the semantic integrity condition based on the prosodic features of the input speech stream includes: When the duration of the input speech stream reaches the segmentation threshold, silence detection is performed on the input speech stream to detect the pause time of the input speech stream; In the absence of detected pauses in the input speech stream, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream. Also includes: If a pause in the input speech stream is detected, the input speech stream obtained up to the pause time is extracted as a speech segment.
3. The method according to claim 1, wherein determining the segmentation time that satisfies the semantic integrity condition based on the prosodic features of the input speech stream in real time includes: The pitch features of the input speech stream are extracted in real time to obtain the prosodic features at the current moment based on the pitch features. The prosodic features at the current moment represent the pitch change trend of the input speech stream within a certain time period up to the current moment. If the prosodic features at the current moment match the reference prosodic features, the current moment is determined to be a segmentation moment that satisfies the semantic integrity condition. The reference prosodic features include prosodic features obtained from sample speech segments, and the sample speech segments are pre-annotated speech segments with complete semantics.
4. The method according to claim 1, wherein translating the speech segment to obtain a processed speech segment includes: The first text data of the corresponding first language of the speech segment is obtained based on speech recognition; Process the first text data to obtain the second text data in the corresponding second language; Processed speech segments are generated based on the second text data.
5. The method according to claim 4, wherein processing the first text data to obtain second text data corresponding to the second language includes: Perform text repair on the first text data to obtain the repaired first text data; Based on the repaired first text data, a language conversion is performed to obtain second text data in the corresponding second language.
6. The method according to claim 5, wherein performing text repair on the first text data to obtain repaired first text data includes: Predicted text data is determined based on the first text data and historical text data, wherein the predicted text data represents the text data corresponding to the next speech segment to be obtained in the future. Based on the historical text data and the predicted text data, the first text data is repaired to obtain the repaired first text data.
7. The method according to claim 4, wherein the first text data includes a first text encoding sequence, the first text encoding sequence includes a plurality of first text codes, and the first text codes represent the lexical units of the first language contained in the speech segment; The second text data includes a second text encoding sequence, which includes multiple second text codes, and the second text codes represent lexical units of a second language.
8. The method according to claim 4, wherein generating the processed speech segment based on the second text data comprises: Based on any one or more of the timbre, pitch, spectrum, and volume of the speech segment, the audio features of the speech segment are obtained; Based on the second text data and the audio features of the speech segment, a processed speech segment is generated; wherein the speech content of the processed speech segment matches the second text data, and the audio features of the processed speech segment match the audio features of the speech segment.
9. An audio processing apparatus, comprising: The acquisition unit is used to acquire the input speech stream; The determining unit is used to determine, in real time, the segmentation time that satisfies the semantic integrity condition based on the prosodic features of the input speech stream when the duration of the input speech stream reaches the segmentation threshold. The segmentation unit is used to extract the input speech stream obtained up to the segmentation time as a speech segment; The translation unit is used to translate each acquired speech segment to obtain a processed speech segment, wherein the language of the processed speech segment is different from the language of the original speech segment.
10. An electronic device, comprising: Acquisition module, used to obtain input speech stream; Memory, used to store computer programs; A processor for executing the computer program to perform: When the duration of the input speech stream reaches the segmentation threshold, the segmentation time that satisfies the semantic integrity condition is determined in real time based on the prosodic features of the input speech stream. Extract the input speech stream obtained up to the segmentation time as a speech segment; For each obtained speech segment, the speech segment is translated to obtain a processed speech segment. The language of the processed speech segment is different from the language of the original speech segment.